pax_global_header 0000666 0000000 0000000 00000000064 13741174344 0014522 g ustar 00root root 0000000 0000000 52 comment=5d1c1d03f8703c2e81478d9a30e9afa2d3e4bd8a
appengine-1.6.7/ 0000775 0000000 0000000 00000000000 13741174344 0013503 5 ustar 00root root 0000000 0000000 appengine-1.6.7/.travis.yml 0000664 0000000 0000000 00000000407 13741174344 0015615 0 ustar 00root root 0000000 0000000 language: go
go_import_path: google.golang.org/appengine
install:
- ./travis_install.sh
script:
- ./travis_test.sh
matrix:
include:
- go: 1.9.x
env: GOAPP=true
- go: 1.10.x
env: GOAPP=false
- go: 1.11.x
env: GO111MODULE=on
appengine-1.6.7/CONTRIBUTING.md 0000664 0000000 0000000 00000007103 13741174344 0015735 0 ustar 00root root 0000000 0000000 # Contributing
1. Sign one of the contributor license agreements below.
1. Get the package:
`go get -d google.golang.org/appengine`
1. Change into the checked out source:
`cd $GOPATH/src/google.golang.org/appengine`
1. Fork the repo.
1. Set your fork as a remote:
`git remote add fork git@github.com:GITHUB_USERNAME/appengine.git`
1. Make changes, commit to your fork.
1. Send a pull request with your changes.
The first line of your commit message is conventionally a one-line summary of the change, prefixed by the primary affected package, and is used as the title of your pull request.
# Testing
## Running system tests
Download and install the [Go App Engine SDK](https://cloud.google.com/appengine/docs/go/download). Make sure the `go_appengine` dir is in your `PATH`.
Set the `APPENGINE_DEV_APPSERVER` environment variable to `/path/to/go_appengine/dev_appserver.py`.
Run tests with `goapp test`:
```
goapp test -v google.golang.org/appengine/...
```
## Contributor License Agreements
Before we can accept your pull requests you'll need to sign a Contributor
License Agreement (CLA):
- **If you are an individual writing original source code** and **you own the
intellectual property**, then you'll need to sign an [individual CLA][indvcla].
- **If you work for a company that wants to allow you to contribute your work**,
then you'll need to sign a [corporate CLA][corpcla].
You can sign these electronically (just scroll to the bottom). After that,
we'll be able to accept your pull requests.
## Contributor Code of Conduct
As contributors and maintainers of this project,
and in the interest of fostering an open and welcoming community,
we pledge to respect all people who contribute through reporting issues,
posting feature requests, updating documentation,
submitting pull requests or patches, and other activities.
We are committed to making participation in this project
a harassment-free experience for everyone,
regardless of level of experience, gender, gender identity and expression,
sexual orientation, disability, personal appearance,
body size, race, ethnicity, age, religion, or nationality.
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery
* Personal attacks
* Trolling or insulting/derogatory comments
* Public or private harassment
* Publishing other's private information,
such as physical or electronic
addresses, without explicit permission
* Other unethical or unprofessional conduct.
Project maintainers have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct.
By adopting this Code of Conduct,
project maintainers commit themselves to fairly and consistently
applying these principles to every aspect of managing this project.
Project maintainers who do not follow or enforce the Code of Conduct
may be permanently removed from the project team.
This code of conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community.
Instances of abusive, harassing, or otherwise unacceptable behavior
may be reported by opening an issue
or contacting one or more of the project maintainers.
This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.2.0,
available at [http://contributor-covenant.org/version/1/2/0/](http://contributor-covenant.org/version/1/2/0/)
[indvcla]: https://developers.google.com/open-source/cla/individual
[corpcla]: https://developers.google.com/open-source/cla/corporate
appengine-1.6.7/LICENSE 0000664 0000000 0000000 00000026136 13741174344 0014520 0 ustar 00root root 0000000 0000000
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
appengine-1.6.7/README.md 0000664 0000000 0000000 00000011345 13741174344 0014766 0 ustar 00root root 0000000 0000000 # Go App Engine packages
[](https://travis-ci.org/golang/appengine)
This repository supports the Go runtime on *App Engine standard*.
It provides APIs for interacting with App Engine services.
Its canonical import path is `google.golang.org/appengine`.
See https://cloud.google.com/appengine/docs/go/
for more information.
File issue reports and feature requests on the [GitHub's issue
tracker](https://github.com/golang/appengine/issues).
## Upgrading an App Engine app to the flexible environment
This package does not work on *App Engine flexible*.
There are many differences between the App Engine standard environment and
the flexible environment.
See the [documentation on upgrading to the flexible environment](https://cloud.google.com/appengine/docs/flexible/go/upgrading).
## Directory structure
The top level directory of this repository is the `appengine` package. It
contains the
basic APIs (e.g. `appengine.NewContext`) that apply across APIs. Specific API
packages are in subdirectories (e.g. `datastore`).
There is an `internal` subdirectory that contains service protocol buffers,
plus packages required for connectivity to make API calls. App Engine apps
should not directly import any package under `internal`.
## Updating from legacy (`import "appengine"`) packages
If you're currently using the bare `appengine` packages
(that is, not these ones, imported via `google.golang.org/appengine`),
then you can use the `aefix` tool to help automate an upgrade to these packages.
Run `go get google.golang.org/appengine/cmd/aefix` to install it.
### 1. Update import paths
The import paths for App Engine packages are now fully qualified, based at `google.golang.org/appengine`.
You will need to update your code to use import paths starting with that; for instance,
code importing `appengine/datastore` will now need to import `google.golang.org/appengine/datastore`.
### 2. Update code using deprecated, removed or modified APIs
Most App Engine services are available with exactly the same API.
A few APIs were cleaned up, and there are some differences:
* `appengine.Context` has been replaced with the `Context` type from `golang.org/x/net/context`.
* Logging methods that were on `appengine.Context` are now functions in `google.golang.org/appengine/log`.
* `appengine.Timeout` has been removed. Use `context.WithTimeout` instead.
* `appengine.Datacenter` now takes a `context.Context` argument.
* `datastore.PropertyLoadSaver` has been simplified to use slices in place of channels.
* `delay.Call` now returns an error.
* `search.FieldLoadSaver` now handles document metadata.
* `urlfetch.Transport` no longer has a Deadline field; set a deadline on the
`context.Context` instead.
* `aetest` no longer declares its own Context type, and uses the standard one instead.
* `taskqueue.QueueStats` no longer takes a maxTasks argument. That argument has been
deprecated and unused for a long time.
* `appengine.BackendHostname` and `appengine.BackendInstance` were for the deprecated backends feature.
Use `appengine.ModuleHostname`and `appengine.ModuleName` instead.
* Most of `appengine/file` and parts of `appengine/blobstore` are deprecated.
Use [Google Cloud Storage](https://godoc.org/cloud.google.com/go/storage) if the
feature you require is not present in the new
[blobstore package](https://google.golang.org/appengine/blobstore).
* `appengine/socket` is not required on App Engine flexible environment / Managed VMs.
Use the standard `net` package instead.
## Key Encode/Decode compatibiltiy to help with datastore library migrations
Key compatibility updates have been added to help customers transition from google.golang.org/appengine/datastore to cloud.google.com/go/datastore.
The `EnableKeyConversion` enables automatic conversion from a key encoded with cloud.google.com/go/datastore to google.golang.org/appengine/datastore key type.
### Enabling key conversion
Enable key conversion by calling `EnableKeyConversion(ctx)` in the `/_ah/start` handler for basic and manual scaling or any handler in automatic scaling.
#### 1. Basic or manual scaling
This start handler will enable key conversion for all handlers in the service.
```
http.HandleFunc("/_ah/start", func(w http.ResponseWriter, r *http.Request) {
datastore.EnableKeyConversion(appengine.NewContext(r))
})
```
#### 2. Automatic scaling
`/_ah/start` is not supported for automatic scaling and `/_ah/warmup` is not guaranteed to run, so you must call `datastore.EnableKeyConversion(appengine.NewContext(r))`
before you use code that needs key conversion.
You may want to add this to each of your handlers, or introduce middleware where it's called.
`EnableKeyConversion` is safe for concurrent use. Any call to it after the first is ignored. appengine-1.6.7/aetest/ 0000775 0000000 0000000 00000000000 13741174344 0014770 5 ustar 00root root 0000000 0000000 appengine-1.6.7/aetest/doc.go 0000664 0000000 0000000 00000001631 13741174344 0016065 0 ustar 00root root 0000000 0000000 /*
Package aetest provides an API for running dev_appserver for use in tests.
An example test file:
package foo_test
import (
"testing"
"google.golang.org/appengine/memcache"
"google.golang.org/appengine/aetest"
)
func TestFoo(t *testing.T) {
ctx, done, err := aetest.NewContext()
if err != nil {
t.Fatal(err)
}
defer done()
it := &memcache.Item{
Key: "some-key",
Value: []byte("some-value"),
}
err = memcache.Set(ctx, it)
if err != nil {
t.Fatalf("Set err: %v", err)
}
it, err = memcache.Get(ctx, "some-key")
if err != nil {
t.Fatalf("Get err: %v; want no error", err)
}
if g, w := string(it.Value), "some-value" ; g != w {
t.Errorf("retrieved Item.Value = %q, want %q", g, w)
}
}
The environment variable APPENGINE_DEV_APPSERVER specifies the location of the
dev_appserver.py executable to use. If unset, the system PATH is consulted.
*/
package aetest
appengine-1.6.7/aetest/instance.go 0000664 0000000 0000000 00000003736 13741174344 0017134 0 ustar 00root root 0000000 0000000 package aetest
import (
"io"
"net/http"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine"
)
// Instance represents a running instance of the development API Server.
type Instance interface {
// Close kills the child api_server.py process, releasing its resources.
io.Closer
// NewRequest returns an *http.Request associated with this instance.
NewRequest(method, urlStr string, body io.Reader) (*http.Request, error)
}
// Options is used to specify options when creating an Instance.
type Options struct {
// AppID specifies the App ID to use during tests.
// By default, "testapp".
AppID string
// StronglyConsistentDatastore is whether the local datastore should be
// strongly consistent. This will diverge from production behaviour.
StronglyConsistentDatastore bool
// SupportDatastoreEmulator is whether use Cloud Datastore Emulator or
// use old SQLite based Datastore backend or use default settings.
SupportDatastoreEmulator *bool
// SuppressDevAppServerLog is whether the dev_appserver running in tests
// should output logs.
SuppressDevAppServerLog bool
// StartupTimeout is a duration to wait for instance startup.
// By default, 15 seconds.
StartupTimeout time.Duration
}
// NewContext starts an instance of the development API server, and returns
// a context that will route all API calls to that server, as well as a
// closure that must be called when the Context is no longer required.
func NewContext() (context.Context, func(), error) {
inst, err := NewInstance(nil)
if err != nil {
return nil, nil, err
}
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
inst.Close()
return nil, nil, err
}
ctx := appengine.NewContext(req)
return ctx, func() {
inst.Close()
}, nil
}
// PrepareDevAppserver is a hook which, if set, will be called before the
// dev_appserver.py is started, each time it is started. If aetest.NewContext
// is invoked from the goapp test tool, this hook is unnecessary.
var PrepareDevAppserver func() error
appengine-1.6.7/aetest/instance_classic.go 0000664 0000000 0000000 00000001113 13741174344 0020620 0 ustar 00root root 0000000 0000000 // +build appengine
package aetest
import "appengine/aetest"
// NewInstance launches a running instance of api_server.py which can be used
// for multiple test Contexts that delegate all App Engine API calls to that
// instance.
// If opts is nil the default values are used.
func NewInstance(opts *Options) (Instance, error) {
aetest.PrepareDevAppserver = PrepareDevAppserver
var aeOpts *aetest.Options
if opts != nil {
aeOpts = &aetest.Options{
AppID: opts.AppID,
StronglyConsistentDatastore: opts.StronglyConsistentDatastore,
}
}
return aetest.NewInstance(aeOpts)
}
appengine-1.6.7/aetest/instance_test.go 0000664 0000000 0000000 00000005442 13741174344 0020167 0 ustar 00root root 0000000 0000000 package aetest
import (
"os"
"testing"
"google.golang.org/appengine"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/internal"
"google.golang.org/appengine/memcache"
"google.golang.org/appengine/user"
)
func TestBasicAPICalls(t *testing.T) {
// Only run the test if APPENGINE_DEV_APPSERVER is explicitly set.
if os.Getenv("APPENGINE_DEV_APPSERVER") == "" {
t.Skip("APPENGINE_DEV_APPSERVER not set")
}
resetEnv := internal.SetTestEnv()
defer resetEnv()
inst, err := NewInstance(nil)
if err != nil {
t.Fatalf("NewInstance: %v", err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "http://example.com/page", nil)
if err != nil {
t.Fatalf("NewRequest: %v", err)
}
ctx := appengine.NewContext(req)
it := &memcache.Item{
Key: "some-key",
Value: []byte("some-value"),
}
err = memcache.Set(ctx, it)
if err != nil {
t.Fatalf("Set err: %v", err)
}
it, err = memcache.Get(ctx, "some-key")
if err != nil {
t.Fatalf("Get err: %v; want no error", err)
}
if g, w := string(it.Value), "some-value"; g != w {
t.Errorf("retrieved Item.Value = %q, want %q", g, w)
}
type Entity struct{ Value string }
e := &Entity{Value: "foo"}
k := datastore.NewIncompleteKey(ctx, "Entity", nil)
k, err = datastore.Put(ctx, k, e)
if err != nil {
t.Fatalf("datastore.Put: %v", err)
}
e = new(Entity)
if err := datastore.Get(ctx, k, e); err != nil {
t.Fatalf("datastore.Get: %v", err)
}
if g, w := e.Value, "foo"; g != w {
t.Errorf("retrieved Entity.Value = %q, want %q", g, w)
}
}
func TestContext(t *testing.T) {
// Only run the test if APPENGINE_DEV_APPSERVER is explicitly set.
if os.Getenv("APPENGINE_DEV_APPSERVER") == "" {
t.Skip("APPENGINE_DEV_APPSERVER not set")
}
// Check that the context methods work.
_, done, err := NewContext()
if err != nil {
t.Fatalf("NewContext: %v", err)
}
done()
}
func TestUsers(t *testing.T) {
// Only run the test if APPENGINE_DEV_APPSERVER is explicitly set.
if os.Getenv("APPENGINE_DEV_APPSERVER") == "" {
t.Skip("APPENGINE_DEV_APPSERVER not set")
}
inst, err := NewInstance(nil)
if err != nil {
t.Fatalf("NewInstance: %v", err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "http://example.com/page", nil)
if err != nil {
t.Fatalf("NewRequest: %v", err)
}
ctx := appengine.NewContext(req)
if user := user.Current(ctx); user != nil {
t.Errorf("user.Current initially %v, want nil", user)
}
u := &user.User{
Email: "gopher@example.com",
Admin: true,
}
Login(u, req)
if got := user.Current(ctx); got.Email != u.Email {
t.Errorf("user.Current: %v, want %v", got, u)
}
if admin := user.IsAdmin(ctx); !admin {
t.Errorf("user.IsAdmin: %t, want true", admin)
}
Logout(req)
if user := user.Current(ctx); user != nil {
t.Errorf("user.Current after logout %v, want nil", user)
}
}
appengine-1.6.7/aetest/instance_vm.go 0000664 0000000 0000000 00000015670 13741174344 0017636 0 ustar 00root root 0000000 0000000 // +build !appengine
package aetest
import (
"bufio"
"crypto/rand"
"errors"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"os"
"os/exec"
"path/filepath"
"regexp"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
)
// NewInstance launches a running instance of api_server.py which can be used
// for multiple test Contexts that delegate all App Engine API calls to that
// instance.
// If opts is nil the default values are used.
func NewInstance(opts *Options) (Instance, error) {
i := &instance{
opts: opts,
appID: "testapp",
startupTimeout: 15 * time.Second,
}
if opts != nil {
if opts.AppID != "" {
i.appID = opts.AppID
}
if opts.StartupTimeout > 0 {
i.startupTimeout = opts.StartupTimeout
}
}
if err := i.startChild(); err != nil {
return nil, err
}
return i, nil
}
func newSessionID() string {
var buf [16]byte
io.ReadFull(rand.Reader, buf[:])
return fmt.Sprintf("%x", buf[:])
}
// instance implements the Instance interface.
type instance struct {
opts *Options
child *exec.Cmd
apiURL *url.URL // base URL of API HTTP server
adminURL string // base URL of admin HTTP server
appDir string
appID string
startupTimeout time.Duration
relFuncs []func() // funcs to release any associated contexts
}
// NewRequest returns an *http.Request associated with this instance.
func (i *instance) NewRequest(method, urlStr string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequest(method, urlStr, body)
if err != nil {
return nil, err
}
// Associate this request.
req, release := internal.RegisterTestRequest(req, i.apiURL, func(ctx context.Context) context.Context {
ctx = internal.WithAppIDOverride(ctx, "dev~"+i.appID)
return ctx
})
i.relFuncs = append(i.relFuncs, release)
return req, nil
}
// Close kills the child api_server.py process, releasing its resources.
func (i *instance) Close() (err error) {
for _, rel := range i.relFuncs {
rel()
}
i.relFuncs = nil
child := i.child
if child == nil {
return nil
}
defer func() {
i.child = nil
err1 := os.RemoveAll(i.appDir)
if err == nil {
err = err1
}
}()
if p := child.Process; p != nil {
errc := make(chan error, 1)
go func() {
errc <- child.Wait()
}()
// Call the quit handler on the admin server.
res, err := http.Get(i.adminURL + "/quit")
if err != nil {
p.Kill()
return fmt.Errorf("unable to call /quit handler: %v", err)
}
res.Body.Close()
select {
case <-time.After(15 * time.Second):
p.Kill()
return errors.New("timeout killing child process")
case err = <-errc:
// Do nothing.
}
}
return
}
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}
func findPython() (path string, err error) {
for _, name := range []string{"python2.7", "python"} {
path, err = exec.LookPath(name)
if err == nil {
return
}
}
return
}
func findDevAppserver() (string, error) {
if p := os.Getenv("APPENGINE_DEV_APPSERVER"); p != "" {
if fileExists(p) {
return p, nil
}
return "", fmt.Errorf("invalid APPENGINE_DEV_APPSERVER environment variable; path %q doesn't exist", p)
}
return exec.LookPath("dev_appserver.py")
}
var apiServerAddrRE = regexp.MustCompile(`Starting API server at: (\S+)`)
var adminServerAddrRE = regexp.MustCompile(`Starting admin server at: (\S+)`)
func (i *instance) startChild() (err error) {
if PrepareDevAppserver != nil {
if err := PrepareDevAppserver(); err != nil {
return err
}
}
executable := os.Getenv("APPENGINE_DEV_APPSERVER_BINARY")
var appserverArgs []string
if len(executable) == 0 {
executable, err = findPython()
if err != nil {
return fmt.Errorf("Could not find python interpreter: %v", err)
}
devAppserver, err := findDevAppserver()
if err != nil {
return fmt.Errorf("Could not find dev_appserver.py: %v", err)
}
appserverArgs = append(appserverArgs, devAppserver)
}
i.appDir, err = ioutil.TempDir("", "appengine-aetest")
if err != nil {
return err
}
defer func() {
if err != nil {
os.RemoveAll(i.appDir)
}
}()
err = os.Mkdir(filepath.Join(i.appDir, "app"), 0755)
if err != nil {
return err
}
err = ioutil.WriteFile(filepath.Join(i.appDir, "app", "app.yaml"), []byte(i.appYAML()), 0644)
if err != nil {
return err
}
err = ioutil.WriteFile(filepath.Join(i.appDir, "app", "stubapp.go"), []byte(appSource), 0644)
if err != nil {
return err
}
appserverArgs = append(appserverArgs,
"--port=0",
"--api_port=0",
"--admin_port=0",
"--automatic_restart=false",
"--skip_sdk_update_check=true",
"--clear_datastore=true",
"--clear_search_indexes=true",
"--datastore_path", filepath.Join(i.appDir, "datastore"),
)
if i.opts != nil && i.opts.StronglyConsistentDatastore {
appserverArgs = append(appserverArgs, "--datastore_consistency_policy=consistent")
}
if i.opts != nil && i.opts.SupportDatastoreEmulator != nil {
appserverArgs = append(appserverArgs, fmt.Sprintf("--support_datastore_emulator=%t", *i.opts.SupportDatastoreEmulator))
}
appserverArgs = append(appserverArgs, filepath.Join(i.appDir, "app"))
i.child = exec.Command(executable, appserverArgs...)
i.child.Stdout = os.Stdout
var stderr io.Reader
stderr, err = i.child.StderrPipe()
if err != nil {
return err
}
if err = i.child.Start(); err != nil {
return err
}
// Read stderr until we have read the URLs of the API server and admin interface.
errc := make(chan error, 1)
go func() {
s := bufio.NewScanner(stderr)
for s.Scan() {
// Pass stderr along as we go so the user can see it.
if !(i.opts != nil && i.opts.SuppressDevAppServerLog) {
fmt.Fprintln(os.Stderr, s.Text())
}
if match := apiServerAddrRE.FindStringSubmatch(s.Text()); match != nil {
u, err := url.Parse(match[1])
if err != nil {
errc <- fmt.Errorf("failed to parse API URL %q: %v", match[1], err)
return
}
i.apiURL = u
}
if match := adminServerAddrRE.FindStringSubmatch(s.Text()); match != nil {
i.adminURL = match[1]
}
if i.adminURL != "" && i.apiURL != nil {
// Pass along stderr to the user after we're done with it.
if !(i.opts != nil && i.opts.SuppressDevAppServerLog) {
go io.Copy(os.Stderr, stderr)
}
break
}
}
errc <- s.Err()
}()
select {
case <-time.After(i.startupTimeout):
if p := i.child.Process; p != nil {
p.Kill()
}
return errors.New("timeout starting child process")
case err := <-errc:
if err != nil {
return fmt.Errorf("error reading child process stderr: %v", err)
}
}
if i.adminURL == "" {
return errors.New("unable to find admin server URL")
}
if i.apiURL == nil {
return errors.New("unable to find API server URL")
}
return nil
}
func (i *instance) appYAML() string {
return fmt.Sprintf(appYAMLTemplate, i.appID)
}
const appYAMLTemplate = `
application: %s
version: 1
runtime: go111
handlers:
- url: /.*
script: _go_app
`
const appSource = `
package main
import "google.golang.org/appengine"
func main() { appengine.Main() }
`
appengine-1.6.7/aetest/user.go 0000664 0000000 0000000 00000002713 13741174344 0016300 0 ustar 00root root 0000000 0000000 package aetest
import (
"hash/crc32"
"net/http"
"strconv"
"google.golang.org/appengine/user"
)
// Login causes the provided Request to act as though issued by the given user.
func Login(u *user.User, req *http.Request) {
req.Header.Set("X-AppEngine-User-Email", u.Email)
id := u.ID
if id == "" {
id = strconv.Itoa(int(crc32.Checksum([]byte(u.Email), crc32.IEEETable)))
}
req.Header.Set("X-AppEngine-User-Id", id)
req.Header.Set("X-AppEngine-Federated-Identity", u.FederatedIdentity)
req.Header.Set("X-AppEngine-Federated-Provider", u.FederatedProvider)
// NOTE: the following two headers are wrong, but are preserved to not break legacy tests.
req.Header.Set("X-AppEngine-User-Federated-Identity", u.Email)
req.Header.Set("X-AppEngine-User-Federated-Provider", u.FederatedProvider)
if u.Admin {
req.Header.Set("X-AppEngine-User-Is-Admin", "1")
} else {
req.Header.Set("X-AppEngine-User-Is-Admin", "0")
}
}
// Logout causes the provided Request to act as though issued by a logged-out
// user.
func Logout(req *http.Request) {
req.Header.Del("X-AppEngine-User-Email")
req.Header.Del("X-AppEngine-User-Id")
req.Header.Del("X-AppEngine-User-Is-Admin")
req.Header.Del("X-AppEngine-Federated-Identity")
req.Header.Del("X-AppEngine-Federated-Provider")
// NOTE: the following two headers are wrong, but are preserved to not break legacy tests.
req.Header.Del("X-AppEngine-User-Federated-Identity")
req.Header.Del("X-AppEngine-User-Federated-Provider")
}
appengine-1.6.7/appengine.go 0000664 0000000 0000000 00000010371 13741174344 0016002 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package appengine provides basic functionality for Google App Engine.
//
// For more information on how to write Go apps for Google App Engine, see:
// https://cloud.google.com/appengine/docs/go/
package appengine // import "google.golang.org/appengine"
import (
"net/http"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
)
// The gophers party all night; the rabbits provide the beats.
// Main is the principal entry point for an app running in App Engine.
//
// On App Engine Flexible it installs a trivial health checker if one isn't
// already registered, and starts listening on port 8080 (overridden by the
// $PORT environment variable).
//
// See https://cloud.google.com/appengine/docs/flexible/custom-runtimes#health_check_requests
// for details on how to do your own health checking.
//
// On App Engine Standard it ensures the server has started and is prepared to
// receive requests.
//
// Main never returns.
//
// Main is designed so that the app's main package looks like this:
//
// package main
//
// import (
// "google.golang.org/appengine"
//
// _ "myapp/package0"
// _ "myapp/package1"
// )
//
// func main() {
// appengine.Main()
// }
//
// The "myapp/packageX" packages are expected to register HTTP handlers
// in their init functions.
func Main() {
internal.Main()
}
// IsDevAppServer reports whether the App Engine app is running in the
// development App Server.
func IsDevAppServer() bool {
return internal.IsDevAppServer()
}
// IsStandard reports whether the App Engine app is running in the standard
// environment. This includes both the first generation runtimes (<= Go 1.9)
// and the second generation runtimes (>= Go 1.11).
func IsStandard() bool {
return internal.IsStandard()
}
// IsFlex reports whether the App Engine app is running in the flexible environment.
func IsFlex() bool {
return internal.IsFlex()
}
// IsAppEngine reports whether the App Engine app is running on App Engine, in either
// the standard or flexible environment.
func IsAppEngine() bool {
return internal.IsAppEngine()
}
// IsSecondGen reports whether the App Engine app is running on the second generation
// runtimes (>= Go 1.11).
func IsSecondGen() bool {
return internal.IsSecondGen()
}
// NewContext returns a context for an in-flight HTTP request.
// This function is cheap.
func NewContext(req *http.Request) context.Context {
return internal.ReqContext(req)
}
// WithContext returns a copy of the parent context
// and associates it with an in-flight HTTP request.
// This function is cheap.
func WithContext(parent context.Context, req *http.Request) context.Context {
return internal.WithContext(parent, req)
}
// BlobKey is a key for a blobstore blob.
//
// Conceptually, this type belongs in the blobstore package, but it lives in
// the appengine package to avoid a circular dependency: blobstore depends on
// datastore, and datastore needs to refer to the BlobKey type.
type BlobKey string
// GeoPoint represents a location as latitude/longitude in degrees.
type GeoPoint struct {
Lat, Lng float64
}
// Valid returns whether a GeoPoint is within [-90, 90] latitude and [-180, 180] longitude.
func (g GeoPoint) Valid() bool {
return -90 <= g.Lat && g.Lat <= 90 && -180 <= g.Lng && g.Lng <= 180
}
// APICallFunc defines a function type for handling an API call.
// See WithCallOverride.
type APICallFunc func(ctx context.Context, service, method string, in, out proto.Message) error
// WithAPICallFunc returns a copy of the parent context
// that will cause API calls to invoke f instead of their normal operation.
//
// This is intended for advanced users only.
func WithAPICallFunc(ctx context.Context, f APICallFunc) context.Context {
return internal.WithCallOverride(ctx, internal.CallOverrideFunc(f))
}
// APICall performs an API call.
//
// This is not intended for general use; it is exported for use in conjunction
// with WithAPICallFunc.
func APICall(ctx context.Context, service, method string, in, out proto.Message) error {
return internal.Call(ctx, service, method, in, out)
}
appengine-1.6.7/appengine_test.go 0000664 0000000 0000000 00000001363 13741174344 0017042 0 ustar 00root root 0000000 0000000 // Copyright 2014 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package appengine
import (
"testing"
)
func TestValidGeoPoint(t *testing.T) {
testCases := []struct {
desc string
pt GeoPoint
want bool
}{
{
"valid",
GeoPoint{67.21, 13.37},
true,
},
{
"high lat",
GeoPoint{-90.01, 13.37},
false,
},
{
"low lat",
GeoPoint{90.01, 13.37},
false,
},
{
"high lng",
GeoPoint{67.21, 182},
false,
},
{
"low lng",
GeoPoint{67.21, -181},
false,
},
}
for _, tc := range testCases {
if got := tc.pt.Valid(); got != tc.want {
t.Errorf("%s: got %v, want %v", tc.desc, got, tc.want)
}
}
}
appengine-1.6.7/appengine_vm.go 0000664 0000000 0000000 00000001044 13741174344 0016501 0 ustar 00root root 0000000 0000000 // Copyright 2015 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// +build !appengine
package appengine
import (
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
)
// BackgroundContext returns a context not associated with a request.
// This should only be used when not servicing a request.
// This only works in App Engine "flexible environment".
func BackgroundContext() context.Context {
return internal.BackgroundContext()
}
appengine-1.6.7/blobstore/ 0000775 0000000 0000000 00000000000 13741174344 0015476 5 ustar 00root root 0000000 0000000 appengine-1.6.7/blobstore/blobstore.go 0000664 0000000 0000000 00000023117 13741174344 0020024 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package blobstore provides a client for App Engine's persistent blob
// storage service.
package blobstore // import "google.golang.org/appengine/blobstore"
import (
"bufio"
"bytes"
"encoding/base64"
"fmt"
"io"
"io/ioutil"
"mime"
"mime/multipart"
"net/http"
"net/textproto"
"net/url"
"strconv"
"strings"
"time"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"golang.org/x/text/encoding/htmlindex"
"google.golang.org/appengine"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/internal"
basepb "google.golang.org/appengine/internal/base"
blobpb "google.golang.org/appengine/internal/blobstore"
)
const (
blobInfoKind = "__BlobInfo__"
blobFileIndexKind = "__BlobFileIndex__"
zeroKey = appengine.BlobKey("")
)
// BlobInfo is the blob metadata that is stored in the datastore.
// Filename may be empty.
type BlobInfo struct {
BlobKey appengine.BlobKey
ContentType string `datastore:"content_type"`
CreationTime time.Time `datastore:"creation"`
Filename string `datastore:"filename"`
Size int64 `datastore:"size"`
MD5 string `datastore:"md5_hash"`
// ObjectName is the Google Cloud Storage name for this blob.
ObjectName string `datastore:"gs_object_name"`
}
// isErrFieldMismatch returns whether err is a datastore.ErrFieldMismatch.
//
// The blobstore stores blob metadata in the datastore. When loading that
// metadata, it may contain fields that we don't care about. datastore.Get will
// return datastore.ErrFieldMismatch in that case, so we ignore that specific
// error.
func isErrFieldMismatch(err error) bool {
_, ok := err.(*datastore.ErrFieldMismatch)
return ok
}
// Stat returns the BlobInfo for a provided blobKey. If no blob was found for
// that key, Stat returns datastore.ErrNoSuchEntity.
func Stat(c context.Context, blobKey appengine.BlobKey) (*BlobInfo, error) {
c, _ = appengine.Namespace(c, "") // Blobstore is always in the empty string namespace
dskey := datastore.NewKey(c, blobInfoKind, string(blobKey), 0, nil)
bi := &BlobInfo{
BlobKey: blobKey,
}
if err := datastore.Get(c, dskey, bi); err != nil && !isErrFieldMismatch(err) {
return nil, err
}
return bi, nil
}
// Send sets the headers on response to instruct App Engine to send a blob as
// the response body. This is more efficient than reading and writing it out
// manually and isn't subject to normal response size limits.
func Send(response http.ResponseWriter, blobKey appengine.BlobKey) {
hdr := response.Header()
hdr.Set("X-AppEngine-BlobKey", string(blobKey))
if hdr.Get("Content-Type") == "" {
// This value is known to dev_appserver to mean automatic.
// In production this is remapped to the empty value which
// means automatic.
hdr.Set("Content-Type", "application/vnd.google.appengine.auto")
}
}
// UploadURL creates an upload URL for the form that the user will
// fill out, passing the application path to load when the POST of the
// form is completed. These URLs expire and should not be reused. The
// opts parameter may be nil.
func UploadURL(c context.Context, successPath string, opts *UploadURLOptions) (*url.URL, error) {
req := &blobpb.CreateUploadURLRequest{
SuccessPath: proto.String(successPath),
}
if opts != nil {
if n := opts.MaxUploadBytes; n != 0 {
req.MaxUploadSizeBytes = &n
}
if n := opts.MaxUploadBytesPerBlob; n != 0 {
req.MaxUploadSizePerBlobBytes = &n
}
if s := opts.StorageBucket; s != "" {
req.GsBucketName = &s
}
}
res := &blobpb.CreateUploadURLResponse{}
if err := internal.Call(c, "blobstore", "CreateUploadURL", req, res); err != nil {
return nil, err
}
return url.Parse(*res.Url)
}
// UploadURLOptions are the options to create an upload URL.
type UploadURLOptions struct {
MaxUploadBytes int64 // optional
MaxUploadBytesPerBlob int64 // optional
// StorageBucket specifies the Google Cloud Storage bucket in which
// to store the blob.
// This is required if you use Cloud Storage instead of Blobstore.
// Your application must have permission to write to the bucket.
// You may optionally specify a bucket name and path in the format
// "bucket_name/path", in which case the included path will be the
// prefix of the uploaded object's name.
StorageBucket string
}
// Delete deletes a blob.
func Delete(c context.Context, blobKey appengine.BlobKey) error {
return DeleteMulti(c, []appengine.BlobKey{blobKey})
}
// DeleteMulti deletes multiple blobs.
func DeleteMulti(c context.Context, blobKey []appengine.BlobKey) error {
s := make([]string, len(blobKey))
for i, b := range blobKey {
s[i] = string(b)
}
req := &blobpb.DeleteBlobRequest{
BlobKey: s,
}
res := &basepb.VoidProto{}
if err := internal.Call(c, "blobstore", "DeleteBlob", req, res); err != nil {
return err
}
return nil
}
func errorf(format string, args ...interface{}) error {
return fmt.Errorf("blobstore: "+format, args...)
}
// ParseUpload parses the synthetic POST request that your app gets from
// App Engine after a user's successful upload of blobs. Given the request,
// ParseUpload returns a map of the blobs received (keyed by HTML form
// element name) and other non-blob POST parameters.
func ParseUpload(req *http.Request) (blobs map[string][]*BlobInfo, other url.Values, err error) {
_, params, err := mime.ParseMediaType(req.Header.Get("Content-Type"))
if err != nil {
return nil, nil, err
}
boundary := params["boundary"]
if boundary == "" {
return nil, nil, errorf("did not find MIME multipart boundary")
}
blobs = make(map[string][]*BlobInfo)
other = make(url.Values)
mreader := multipart.NewReader(io.MultiReader(req.Body, strings.NewReader("\r\n\r\n")), boundary)
for {
part, perr := mreader.NextPart()
if perr == io.EOF {
break
}
if perr != nil {
return nil, nil, errorf("error reading next mime part with boundary %q (len=%d): %v",
boundary, len(boundary), perr)
}
bi := &BlobInfo{}
ctype, params, err := mime.ParseMediaType(part.Header.Get("Content-Disposition"))
if err != nil {
return nil, nil, err
}
bi.Filename = params["filename"]
formKey := params["name"]
ctype, params, err = mime.ParseMediaType(part.Header.Get("Content-Type"))
if err != nil {
return nil, nil, err
}
bi.BlobKey = appengine.BlobKey(params["blob-key"])
charset := params["charset"]
if ctype != "message/external-body" || bi.BlobKey == "" {
if formKey != "" {
slurp, serr := ioutil.ReadAll(part)
if serr != nil {
return nil, nil, errorf("error reading %q MIME part", formKey)
}
// Handle base64 content transfer encoding. multipart.Part transparently
// handles quoted-printable, and no special handling is required for
// 7bit, 8bit, or binary.
ctype, params, err = mime.ParseMediaType(part.Header.Get("Content-Transfer-Encoding"))
if err == nil && ctype == "base64" {
slurp, serr = ioutil.ReadAll(base64.NewDecoder(
base64.StdEncoding, bytes.NewReader(slurp)))
if serr != nil {
return nil, nil, errorf("error %s decoding %q MIME part", ctype, formKey)
}
}
// Handle charset
if charset != "" {
encoding, err := htmlindex.Get(charset)
if err != nil {
return nil, nil, errorf("error getting decoder for charset %q", charset)
}
slurp, err = encoding.NewDecoder().Bytes(slurp)
if err != nil {
return nil, nil, errorf("error decoding from charset %q", charset)
}
}
other[formKey] = append(other[formKey], string(slurp))
}
continue
}
// App Engine sends a MIME header as the body of each MIME part.
tp := textproto.NewReader(bufio.NewReader(part))
header, mimeerr := tp.ReadMIMEHeader()
if mimeerr != nil {
return nil, nil, mimeerr
}
bi.Size, err = strconv.ParseInt(header.Get("Content-Length"), 10, 64)
if err != nil {
return nil, nil, err
}
bi.ContentType = header.Get("Content-Type")
// Parse the time from the MIME header like:
// X-AppEngine-Upload-Creation: 2011-03-15 21:38:34.712136
createDate := header.Get("X-AppEngine-Upload-Creation")
if createDate == "" {
return nil, nil, errorf("expected to find an X-AppEngine-Upload-Creation header")
}
bi.CreationTime, err = time.Parse("2006-01-02 15:04:05.000000", createDate)
if err != nil {
return nil, nil, errorf("error parsing X-AppEngine-Upload-Creation: %s", err)
}
if hdr := header.Get("Content-MD5"); hdr != "" {
md5, err := base64.URLEncoding.DecodeString(hdr)
if err != nil {
return nil, nil, errorf("bad Content-MD5 %q: %v", hdr, err)
}
bi.MD5 = string(md5)
}
// If the GCS object name was provided, record it.
bi.ObjectName = header.Get("X-AppEngine-Cloud-Storage-Object")
blobs[formKey] = append(blobs[formKey], bi)
}
return
}
// Reader is a blob reader.
type Reader interface {
io.Reader
io.ReaderAt
io.Seeker
}
// NewReader returns a reader for a blob. It always succeeds; if the blob does
// not exist then an error will be reported upon first read.
func NewReader(c context.Context, blobKey appengine.BlobKey) Reader {
return openBlob(c, blobKey)
}
// BlobKeyForFile returns a BlobKey for a Google Storage file.
// The filename should be of the form "/gs/bucket_name/object_name".
func BlobKeyForFile(c context.Context, filename string) (appengine.BlobKey, error) {
req := &blobpb.CreateEncodedGoogleStorageKeyRequest{
Filename: &filename,
}
res := &blobpb.CreateEncodedGoogleStorageKeyResponse{}
if err := internal.Call(c, "blobstore", "CreateEncodedGoogleStorageKey", req, res); err != nil {
return "", err
}
return appengine.BlobKey(*res.BlobKey), nil
}
appengine-1.6.7/blobstore/blobstore_test.go 0000664 0000000 0000000 00000020667 13741174344 0021072 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package blobstore
import (
"bytes"
"encoding/base64"
"fmt"
"io"
"mime/multipart"
"mime/quotedprintable"
"net/http"
"net/textproto"
"os"
"strconv"
"strings"
"testing"
"golang.org/x/text/encoding/htmlindex"
"google.golang.org/appengine"
"google.golang.org/appengine/internal/aetesting"
pb "google.golang.org/appengine/internal/blobstore"
)
const rbs = readBufferSize
const charsetUTF8 = "utf-8"
const charsetISO2022JP = "iso-2022-jp"
const nonASCIIStr = "Hello, 世界"
func min(x, y int) int {
if x < y {
return x
}
return y
}
func fakeFetchData(req *pb.FetchDataRequest, res *pb.FetchDataResponse) error {
i0 := int(*req.StartIndex)
i1 := int(*req.EndIndex + 1) // Blobstore's end-indices are inclusive; Go's are exclusive.
bk := *req.BlobKey
if i := strings.Index(bk, "."); i != -1 {
// Strip everything past the ".".
bk = bk[:i]
}
switch bk {
case "a14p":
const s = "abcdefghijklmnop"
i0 := min(len(s), i0)
i1 := min(len(s), i1)
res.Data = []byte(s[i0:i1])
case "longBlob":
res.Data = make([]byte, i1-i0)
for i := range res.Data {
res.Data[i] = 'A' + uint8(i0/rbs)
i0++
}
}
return nil
}
// step is one step of a readerTest.
// It consists of a Reader method to call, the method arguments
// (lenp, offset, whence) and the expected results.
type step struct {
method string
lenp int
offset int64
whence int
want string
wantErr error
}
var readerTest = []struct {
blobKey string
step []step
}{
{"noSuchBlobKey", []step{
{"Read", 8, 0, 0, "", io.EOF},
}},
{"a14p.0", []step{
// Test basic reads.
{"Read", 1, 0, 0, "a", nil},
{"Read", 3, 0, 0, "bcd", nil},
{"Read", 1, 0, 0, "e", nil},
{"Read", 2, 0, 0, "fg", nil},
// Test Seek.
{"Seek", 0, 2, os.SEEK_SET, "2", nil},
{"Read", 5, 0, 0, "cdefg", nil},
{"Seek", 0, 2, os.SEEK_CUR, "9", nil},
{"Read", 1, 0, 0, "j", nil},
// Test reads up to and past EOF.
{"Read", 5, 0, 0, "klmno", nil},
{"Read", 5, 0, 0, "p", nil},
{"Read", 5, 0, 0, "", io.EOF},
// Test ReadAt.
{"ReadAt", 4, 0, 0, "abcd", nil},
{"ReadAt", 4, 3, 0, "defg", nil},
{"ReadAt", 4, 12, 0, "mnop", nil},
{"ReadAt", 4, 13, 0, "nop", io.EOF},
{"ReadAt", 4, 99, 0, "", io.EOF},
}},
{"a14p.1", []step{
// Test Seek before any reads.
{"Seek", 0, 2, os.SEEK_SET, "2", nil},
{"Read", 1, 0, 0, "c", nil},
// Test that ReadAt doesn't affect the Read offset.
{"ReadAt", 3, 9, 0, "jkl", nil},
{"Read", 3, 0, 0, "def", nil},
}},
{"a14p.2", []step{
// Test ReadAt before any reads or seeks.
{"ReadAt", 2, 14, 0, "op", nil},
}},
{"longBlob.0", []step{
// Test basic read.
{"Read", 1, 0, 0, "A", nil},
// Test that Read returns early when the buffer is exhausted.
{"Seek", 0, rbs - 2, os.SEEK_SET, strconv.Itoa(rbs - 2), nil},
{"Read", 5, 0, 0, "AA", nil},
{"Read", 3, 0, 0, "BBB", nil},
// Test that what we just read is still in the buffer.
{"Seek", 0, rbs - 2, os.SEEK_SET, strconv.Itoa(rbs - 2), nil},
{"Read", 5, 0, 0, "AABBB", nil},
// Test ReadAt.
{"ReadAt", 3, rbs - 4, 0, "AAA", nil},
{"ReadAt", 6, rbs - 4, 0, "AAAABB", nil},
{"ReadAt", 8, rbs - 4, 0, "AAAABBBB", nil},
{"ReadAt", 5, rbs - 4, 0, "AAAAB", nil},
{"ReadAt", 2, rbs - 4, 0, "AA", nil},
// Test seeking backwards from the Read offset.
{"Seek", 0, 2*rbs - 8, os.SEEK_SET, strconv.Itoa(2*rbs - 8), nil},
{"Read", 1, 0, 0, "B", nil},
{"Read", 1, 0, 0, "B", nil},
{"Read", 1, 0, 0, "B", nil},
{"Read", 1, 0, 0, "B", nil},
{"Read", 8, 0, 0, "BBBBCCCC", nil},
}},
{"longBlob.1", []step{
// Test ReadAt with a slice larger than the buffer size.
{"LargeReadAt", 2*rbs - 2, 0, 0, strconv.Itoa(2*rbs - 2), nil},
{"LargeReadAt", 2*rbs - 1, 0, 0, strconv.Itoa(2*rbs - 1), nil},
{"LargeReadAt", 2*rbs + 0, 0, 0, strconv.Itoa(2*rbs + 0), nil},
{"LargeReadAt", 2*rbs + 1, 0, 0, strconv.Itoa(2*rbs + 1), nil},
{"LargeReadAt", 2*rbs + 2, 0, 0, strconv.Itoa(2*rbs + 2), nil},
{"LargeReadAt", 2*rbs - 2, 1, 0, strconv.Itoa(2*rbs - 2), nil},
{"LargeReadAt", 2*rbs - 1, 1, 0, strconv.Itoa(2*rbs - 1), nil},
{"LargeReadAt", 2*rbs + 0, 1, 0, strconv.Itoa(2*rbs + 0), nil},
{"LargeReadAt", 2*rbs + 1, 1, 0, strconv.Itoa(2*rbs + 1), nil},
{"LargeReadAt", 2*rbs + 2, 1, 0, strconv.Itoa(2*rbs + 2), nil},
}},
}
func TestReader(t *testing.T) {
for _, rt := range readerTest {
c := aetesting.FakeSingleContext(t, "blobstore", "FetchData", fakeFetchData)
r := NewReader(c, appengine.BlobKey(rt.blobKey))
for i, step := range rt.step {
var (
got string
gotErr error
n int
offset int64
)
switch step.method {
case "LargeReadAt":
p := make([]byte, step.lenp)
n, gotErr = r.ReadAt(p, step.offset)
got = strconv.Itoa(n)
case "Read":
p := make([]byte, step.lenp)
n, gotErr = r.Read(p)
got = string(p[:n])
case "ReadAt":
p := make([]byte, step.lenp)
n, gotErr = r.ReadAt(p, step.offset)
got = string(p[:n])
case "Seek":
offset, gotErr = r.Seek(step.offset, step.whence)
got = strconv.FormatInt(offset, 10)
default:
t.Fatalf("unknown method: %s", step.method)
}
if gotErr != step.wantErr {
t.Fatalf("%s step %d: got error %v want %v", rt.blobKey, i, gotErr, step.wantErr)
}
if got != step.want {
t.Fatalf("%s step %d: got %q want %q", rt.blobKey, i, got, step.want)
}
}
}
}
// doPlainTextParseUploadTest tests ParseUpload's decoding of non-file form fields.
// It ensures that MIME multipart parts with Content-Type not equal to
// "message/external-body" (i.e. form fields that are not file uploads) are decoded
// correctly according to the value of their Content-Transfer-Encoding header field.
// If charset is not the empty string it will be set in the request's Content-Type
// header field, and if encoding is not the empty string then the Content-Transfer-Encoding
// header field will be set.
func doPlainTextParseUploadTest(t *testing.T, charset string, encoding string,
rawContent string, encodedContent string) {
bodyBuf := &bytes.Buffer{}
w := multipart.NewWriter(bodyBuf)
fieldName := "foo"
hdr := textproto.MIMEHeader{}
hdr.Set("Content-Disposition", fmt.Sprintf("form-data; name=%q", fieldName))
if charset != "" {
hdr.Set("Content-Type", fmt.Sprintf("text/plain; charset=%q", charset))
} else {
hdr.Set("Content-Type", "text/plain")
}
if encoding != "" {
hdr.Set("Content-Transfer-Encoding", encoding)
}
pw, err := w.CreatePart(hdr)
if err != nil {
t.Fatalf("error creating part: %v", err)
}
pw.Write([]byte(encodedContent))
if err := w.Close(); err != nil {
t.Fatalf("error closing multipart writer: %v\n", err)
}
req, err := http.NewRequest("POST", "/upload", bodyBuf)
if err != nil {
t.Fatalf("error creating request: %v", err)
}
req.Header.Set("Content-Type", w.FormDataContentType())
_, other, err := ParseUpload(req)
if err != nil {
t.Fatalf("error parsing upload: %v", err)
}
if other[fieldName][0] != rawContent {
t.Errorf("got %q expected %q", other[fieldName][0], rawContent)
}
}
func TestParseUploadUTF8Base64Encoding(t *testing.T) {
encoded := base64.StdEncoding.EncodeToString([]byte(nonASCIIStr))
doPlainTextParseUploadTest(t, charsetUTF8, "base64", nonASCIIStr, encoded)
}
func TestParseUploadUTF8Base64EncodingMultiline(t *testing.T) {
testStr := "words words words words words words words words words words words words"
encoded := "d29yZHMgd29yZHMgd29yZHMgd29yZHMgd29yZHMgd29yZHMgd29yZHMgd29yZHMgd29yZHMgd29y\r\nZHMgd29yZHMgd29yZHM="
doPlainTextParseUploadTest(t, charsetUTF8, "base64", testStr, encoded)
}
func TestParseUploadUTF8QuotedPrintableEncoding(t *testing.T) {
var encoded bytes.Buffer
writer := quotedprintable.NewWriter(&encoded)
writer.Write([]byte(nonASCIIStr))
writer.Close()
doPlainTextParseUploadTest(t, charsetUTF8, "quoted-printable", nonASCIIStr,
encoded.String())
}
func TestParseUploadISO2022JPBase64Encoding(t *testing.T) {
testStr := "こんにちは"
encoding, err := htmlindex.Get(charsetISO2022JP)
if err != nil {
t.Fatalf("error getting encoding: %v", err)
}
charsetEncoded, err := encoding.NewEncoder().String(testStr)
if err != nil {
t.Fatalf("error encoding string: %v", err)
}
base64Encoded := base64.StdEncoding.EncodeToString([]byte(charsetEncoded))
doPlainTextParseUploadTest(t, charsetISO2022JP, "base64", testStr, base64Encoded)
}
func TestParseUploadNoEncoding(t *testing.T) {
doPlainTextParseUploadTest(t, "", "", "Hello", "Hello")
}
appengine-1.6.7/blobstore/read.go 0000664 0000000 0000000 00000007630 13741174344 0016746 0 ustar 00root root 0000000 0000000 // Copyright 2012 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package blobstore
import (
"errors"
"fmt"
"io"
"os"
"sync"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/internal"
blobpb "google.golang.org/appengine/internal/blobstore"
)
// openBlob returns a reader for a blob. It always succeeds; if the blob does
// not exist then an error will be reported upon first read.
func openBlob(c context.Context, blobKey appengine.BlobKey) Reader {
return &reader{
c: c,
blobKey: blobKey,
}
}
const readBufferSize = 256 * 1024
// reader is a blob reader. It implements the Reader interface.
type reader struct {
c context.Context
// Either blobKey or filename is set:
blobKey appengine.BlobKey
filename string
closeFunc func() // is nil if unavailable or already closed.
// buf is the read buffer. r is how much of buf has been read.
// off is the offset of buf[0] relative to the start of the blob.
// An invariant is 0 <= r && r <= len(buf).
// Reads that don't require an RPC call will increment r but not off.
// Seeks may modify r without discarding the buffer, but only if the
// invariant can be maintained.
mu sync.Mutex
buf []byte
r int
off int64
}
func (r *reader) Close() error {
if f := r.closeFunc; f != nil {
f()
}
r.closeFunc = nil
return nil
}
func (r *reader) Read(p []byte) (int, error) {
if len(p) == 0 {
return 0, nil
}
r.mu.Lock()
defer r.mu.Unlock()
if r.r == len(r.buf) {
if err := r.fetch(r.off + int64(r.r)); err != nil {
return 0, err
}
}
n := copy(p, r.buf[r.r:])
r.r += n
return n, nil
}
func (r *reader) ReadAt(p []byte, off int64) (int, error) {
if len(p) == 0 {
return 0, nil
}
r.mu.Lock()
defer r.mu.Unlock()
// Convert relative offsets to absolute offsets.
ab0 := r.off + int64(r.r)
ab1 := r.off + int64(len(r.buf))
ap0 := off
ap1 := off + int64(len(p))
// Check if we can satisfy the read entirely out of the existing buffer.
if r.off <= ap0 && ap1 <= ab1 {
// Convert off from an absolute offset to a relative offset.
rp0 := int(ap0 - r.off)
return copy(p, r.buf[rp0:]), nil
}
// Restore the original Read/Seek offset after ReadAt completes.
defer r.seek(ab0)
// Repeatedly fetch and copy until we have filled p.
n := 0
for len(p) > 0 {
if err := r.fetch(off + int64(n)); err != nil {
return n, err
}
r.r = copy(p, r.buf)
n += r.r
p = p[r.r:]
}
return n, nil
}
func (r *reader) Seek(offset int64, whence int) (ret int64, err error) {
r.mu.Lock()
defer r.mu.Unlock()
switch whence {
case os.SEEK_SET:
ret = offset
case os.SEEK_CUR:
ret = r.off + int64(r.r) + offset
case os.SEEK_END:
return 0, errors.New("seeking relative to the end of a blob isn't supported")
default:
return 0, fmt.Errorf("invalid Seek whence value: %d", whence)
}
if ret < 0 {
return 0, errors.New("negative Seek offset")
}
return r.seek(ret)
}
// fetch fetches readBufferSize bytes starting at the given offset. On success,
// the data is saved as r.buf.
func (r *reader) fetch(off int64) error {
req := &blobpb.FetchDataRequest{
BlobKey: proto.String(string(r.blobKey)),
StartIndex: proto.Int64(off),
EndIndex: proto.Int64(off + readBufferSize - 1), // EndIndex is inclusive.
}
res := &blobpb.FetchDataResponse{}
if err := internal.Call(r.c, "blobstore", "FetchData", req, res); err != nil {
return err
}
if len(res.Data) == 0 {
return io.EOF
}
r.buf, r.r, r.off = res.Data, 0, off
return nil
}
// seek seeks to the given offset with an effective whence equal to SEEK_SET.
// It discards the read buffer if the invariant cannot be maintained.
func (r *reader) seek(off int64) (int64, error) {
delta := off - r.off
if delta >= 0 && delta < int64(len(r.buf)) {
r.r = int(delta)
return off, nil
}
r.buf, r.r, r.off = nil, 0, off
return off, nil
}
appengine-1.6.7/capability/ 0000775 0000000 0000000 00000000000 13741174344 0015624 5 ustar 00root root 0000000 0000000 appengine-1.6.7/capability/capability.go 0000664 0000000 0000000 00000002740 13741174344 0020277 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
/*
Package capability exposes information about outages and scheduled downtime
for specific API capabilities.
This package does not work in App Engine "flexible environment".
Example:
if !capability.Enabled(c, "datastore_v3", "write") {
// show user a different page
}
*/
package capability // import "google.golang.org/appengine/capability"
import (
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
"google.golang.org/appengine/log"
pb "google.golang.org/appengine/internal/capability"
)
// Enabled returns whether an API's capabilities are enabled.
// The wildcard "*" capability matches every capability of an API.
// If the underlying RPC fails (if the package is unknown, for example),
// false is returned and information is written to the application log.
func Enabled(ctx context.Context, api, capability string) bool {
// For non datastore*/write requests always return ENABLED
if !(api == "datastore_v3" && capability == "write") {
return true
}
req := &pb.IsEnabledRequest{
Package: &api,
Capability: []string{capability},
}
res := &pb.IsEnabledResponse{}
if err := internal.Call(ctx, "capability_service", "IsEnabled", req, res); err != nil {
log.Warningf(ctx, "capability.Enabled: RPC failed: %v", err)
return false
}
return *res.SummaryStatus == pb.IsEnabledResponse_ENABLED
}
appengine-1.6.7/channel/ 0000775 0000000 0000000 00000000000 13741174344 0015113 5 ustar 00root root 0000000 0000000 appengine-1.6.7/channel/channel.go 0000664 0000000 0000000 00000005113 13741174344 0017052 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
/*
Package channel implements the server side of App Engine's Channel API.
Create creates a new channel associated with the given clientID,
which must be unique to the client that will use the returned token.
token, err := channel.Create(c, "player1")
if err != nil {
// handle error
}
// return token to the client in an HTTP response
Send sends a message to the client over the channel identified by clientID.
channel.Send(c, "player1", "Game over!")
Deprecated: The Channel API feature has been deprecated and is going to be removed. See the Channel API Turndown document for details and timetable.
https://cloud.google.com/appengine/docs/deprecations/channel
*/
package channel // import "google.golang.org/appengine/channel"
import (
"encoding/json"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/internal"
basepb "google.golang.org/appengine/internal/base"
pb "google.golang.org/appengine/internal/channel"
)
// Create creates a channel and returns a token for use by the client.
// The clientID is an application-provided string used to identify the client.
func Create(c context.Context, clientID string) (token string, err error) {
req := &pb.CreateChannelRequest{
ApplicationKey: &clientID,
}
resp := &pb.CreateChannelResponse{}
err = internal.Call(c, service, "CreateChannel", req, resp)
token = resp.GetToken()
return token, remapError(err)
}
// Send sends a message on the channel associated with clientID.
func Send(c context.Context, clientID, message string) error {
req := &pb.SendMessageRequest{
ApplicationKey: &clientID,
Message: &message,
}
resp := &basepb.VoidProto{}
return remapError(internal.Call(c, service, "SendChannelMessage", req, resp))
}
// SendJSON is a helper function that sends a JSON-encoded value
// on the channel associated with clientID.
func SendJSON(c context.Context, clientID string, value interface{}) error {
m, err := json.Marshal(value)
if err != nil {
return err
}
return Send(c, clientID, string(m))
}
// remapError fixes any APIError referencing "xmpp" into one referencing "channel".
func remapError(err error) error {
if e, ok := err.(*internal.APIError); ok {
if e.Service == "xmpp" {
e.Service = "channel"
}
}
return err
}
var service = "xmpp" // prod
func init() {
if appengine.IsDevAppServer() {
service = "channel" // dev
}
internal.RegisterErrorCodeMap("channel", pb.ChannelServiceError_ErrorCode_name)
}
appengine-1.6.7/channel/channel_test.go 0000664 0000000 0000000 00000000726 13741174344 0020116 0 ustar 00root root 0000000 0000000 // Copyright 2015 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package channel
import (
"testing"
"google.golang.org/appengine/internal"
)
func TestRemapError(t *testing.T) {
err := &internal.APIError{
Service: "xmpp",
}
err = remapError(err).(*internal.APIError)
if err.Service != "channel" {
t.Errorf("err.Service = %q, want %q", err.Service, "channel")
}
}
appengine-1.6.7/cloudsql/ 0000775 0000000 0000000 00000000000 13741174344 0015331 5 ustar 00root root 0000000 0000000 appengine-1.6.7/cloudsql/cloudsql.go 0000664 0000000 0000000 00000003452 13741174344 0017512 0 ustar 00root root 0000000 0000000 // Copyright 2013 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
/*
Package cloudsql exposes access to Google Cloud SQL databases.
This package does not work in App Engine "flexible environment".
This package is intended for MySQL drivers to make App Engine-specific
connections. Applications should use this package through database/sql:
Select a pure Go MySQL driver that supports this package, and use sql.Open
with protocol "cloudsql" and an address of the Cloud SQL instance.
A Go MySQL driver that has been tested to work well with Cloud SQL
is the go-sql-driver:
import "database/sql"
import _ "github.com/go-sql-driver/mysql"
db, err := sql.Open("mysql", "user@cloudsql(project-id:instance-name)/dbname")
Another driver that works well with Cloud SQL is the mymysql driver:
import "database/sql"
import _ "github.com/ziutek/mymysql/godrv"
db, err := sql.Open("mymysql", "cloudsql:instance-name*dbname/user/password")
Using either of these drivers, you can perform a standard SQL query.
This example assumes there is a table named 'users' with
columns 'first_name' and 'last_name':
rows, err := db.Query("SELECT first_name, last_name FROM users")
if err != nil {
log.Errorf(ctx, "db.Query: %v", err)
}
defer rows.Close()
for rows.Next() {
var firstName string
var lastName string
if err := rows.Scan(&firstName, &lastName); err != nil {
log.Errorf(ctx, "rows.Scan: %v", err)
continue
}
log.Infof(ctx, "First: %v - Last: %v", firstName, lastName)
}
if err := rows.Err(); err != nil {
log.Errorf(ctx, "Row error: %v", err)
}
*/
package cloudsql
import (
"net"
)
// Dial connects to the named Cloud SQL instance.
func Dial(instance string) (net.Conn, error) {
return connect(instance)
}
appengine-1.6.7/cloudsql/cloudsql_classic.go 0000664 0000000 0000000 00000000504 13741174344 0021206 0 ustar 00root root 0000000 0000000 // Copyright 2013 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// +build appengine
package cloudsql
import (
"net"
"appengine/cloudsql"
)
func connect(instance string) (net.Conn, error) {
return cloudsql.Dial(instance)
}
appengine-1.6.7/cloudsql/cloudsql_vm.go 0000664 0000000 0000000 00000000560 13741174344 0020211 0 ustar 00root root 0000000 0000000 // Copyright 2013 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// +build !appengine
package cloudsql
import (
"errors"
"net"
)
func connect(instance string) (net.Conn, error) {
return nil, errors.New(`cloudsql: not supported in App Engine "flexible environment"`)
}
appengine-1.6.7/cmd/ 0000775 0000000 0000000 00000000000 13741174344 0014246 5 ustar 00root root 0000000 0000000 appengine-1.6.7/cmd/aebundler/ 0000775 0000000 0000000 00000000000 13741174344 0016207 5 ustar 00root root 0000000 0000000 appengine-1.6.7/cmd/aebundler/aebundler.go 0000664 0000000 0000000 00000021607 13741174344 0020505 0 ustar 00root root 0000000 0000000 // Copyright 2015 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Program aebundler turns a Go app into a fully self-contained tar file.
// The app and its subdirectories (if any) are placed under "."
// and the dependencies from $GOPATH are placed under ./_gopath/src.
// A main func is synthesized if one does not exist.
//
// A sample Dockerfile to be used with this bundler could look like this:
// FROM gcr.io/google-appengine/go-compat
// ADD . /app
// RUN GOPATH=/app/_gopath go build -tags appenginevm -o /app/_ah/exe
package main
import (
"archive/tar"
"flag"
"fmt"
"go/ast"
"go/build"
"go/parser"
"go/token"
"io"
"io/ioutil"
"os"
"path/filepath"
"strings"
)
var (
output = flag.String("o", "", "name of output tar file or '-' for stdout")
rootDir = flag.String("root", ".", "directory name of application root")
vm = flag.Bool("vm", true, `bundle an app for App Engine "flexible environment"`)
skipFiles = map[string]bool{
".git": true,
".gitconfig": true,
".hg": true,
".travis.yml": true,
}
)
const (
newMain = `package main
import "google.golang.org/appengine"
func main() {
appengine.Main()
}
`
)
func usage() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, "\t%s -o \tBundle app to named tar file or stdout\n", os.Args[0])
fmt.Fprintf(os.Stderr, "\noptional arguments:\n")
flag.PrintDefaults()
}
func main() {
flag.Usage = usage
flag.Parse()
var tags []string
if *vm {
tags = append(tags, "appenginevm")
} else {
tags = append(tags, "appengine")
}
tarFile := *output
if tarFile == "" {
usage()
errorf("Required -o flag not specified.")
}
app, err := analyze(tags)
if err != nil {
errorf("Error analyzing app: %v", err)
}
if err := app.bundle(tarFile); err != nil {
errorf("Unable to bundle app: %v", err)
}
}
// errorf prints the error message and exits.
func errorf(format string, a ...interface{}) {
fmt.Fprintf(os.Stderr, "aebundler: "+format+"\n", a...)
os.Exit(1)
}
type app struct {
hasMain bool
appFiles []string
imports map[string]string
}
// analyze checks the app for building with the given build tags and returns hasMain,
// app files, and a map of full directory import names to original import names.
func analyze(tags []string) (*app, error) {
ctxt := buildContext(tags)
hasMain, appFiles, err := checkMain(ctxt)
if err != nil {
return nil, err
}
gopath := filepath.SplitList(ctxt.GOPATH)
im, err := imports(ctxt, *rootDir, gopath)
return &app{
hasMain: hasMain,
appFiles: appFiles,
imports: im,
}, err
}
// buildContext returns the context for building the source.
func buildContext(tags []string) *build.Context {
return &build.Context{
GOARCH: build.Default.GOARCH,
GOOS: build.Default.GOOS,
GOROOT: build.Default.GOROOT,
GOPATH: build.Default.GOPATH,
Compiler: build.Default.Compiler,
BuildTags: append(build.Default.BuildTags, tags...),
}
}
// bundle bundles the app into the named tarFile ("-"==stdout).
func (s *app) bundle(tarFile string) (err error) {
var out io.Writer
if tarFile == "-" {
out = os.Stdout
} else {
f, err := os.Create(tarFile)
if err != nil {
return err
}
defer func() {
if cerr := f.Close(); err == nil {
err = cerr
}
}()
out = f
}
tw := tar.NewWriter(out)
for srcDir, importName := range s.imports {
dstDir := "_gopath/src/" + importName
if err = copyTree(tw, dstDir, srcDir); err != nil {
return fmt.Errorf("unable to copy directory %v to %v: %v", srcDir, dstDir, err)
}
}
if err := copyTree(tw, ".", *rootDir); err != nil {
return fmt.Errorf("unable to copy root directory to /app: %v", err)
}
if !s.hasMain {
if err := synthesizeMain(tw, s.appFiles); err != nil {
return fmt.Errorf("unable to synthesize new main func: %v", err)
}
}
if err := tw.Close(); err != nil {
return fmt.Errorf("unable to close tar file %v: %v", tarFile, err)
}
return nil
}
// synthesizeMain generates a new main func and writes it to the tarball.
func synthesizeMain(tw *tar.Writer, appFiles []string) error {
appMap := make(map[string]bool)
for _, f := range appFiles {
appMap[f] = true
}
var f string
for i := 0; i < 100; i++ {
f = fmt.Sprintf("app_main%d.go", i)
if !appMap[filepath.Join(*rootDir, f)] {
break
}
}
if appMap[filepath.Join(*rootDir, f)] {
return fmt.Errorf("unable to find unique name for %v", f)
}
hdr := &tar.Header{
Name: f,
Mode: 0644,
Size: int64(len(newMain)),
}
if err := tw.WriteHeader(hdr); err != nil {
return fmt.Errorf("unable to write header for %v: %v", f, err)
}
if _, err := tw.Write([]byte(newMain)); err != nil {
return fmt.Errorf("unable to write %v to tar file: %v", f, err)
}
return nil
}
// imports returns a map of all import directories (recursively) used by the app.
// The return value maps full directory names to original import names.
func imports(ctxt *build.Context, srcDir string, gopath []string) (map[string]string, error) {
pkg, err := ctxt.ImportDir(srcDir, 0)
if err != nil {
return nil, fmt.Errorf("unable to analyze source: %v", err)
}
// Resolve all non-standard-library imports
result := make(map[string]string)
for _, v := range pkg.Imports {
if !strings.Contains(v, ".") {
continue
}
src, err := findInGopath(v, gopath)
if err != nil {
return nil, fmt.Errorf("unable to find import %v in gopath %v: %v", v, gopath, err)
}
result[src] = v
im, err := imports(ctxt, src, gopath)
if err != nil {
return nil, fmt.Errorf("unable to parse package %v: %v", src, err)
}
for k, v := range im {
result[k] = v
}
}
return result, nil
}
// findInGopath searches the gopath for the named import directory.
func findInGopath(dir string, gopath []string) (string, error) {
for _, v := range gopath {
dst := filepath.Join(v, "src", dir)
if _, err := os.Stat(dst); err == nil {
return dst, nil
}
}
return "", fmt.Errorf("unable to find package %v in gopath %v", dir, gopath)
}
// copyTree copies srcDir to tar file dstDir, ignoring skipFiles.
func copyTree(tw *tar.Writer, dstDir, srcDir string) error {
entries, err := ioutil.ReadDir(srcDir)
if err != nil {
return fmt.Errorf("unable to read dir %v: %v", srcDir, err)
}
for _, entry := range entries {
n := entry.Name()
if skipFiles[n] {
continue
}
s := filepath.Join(srcDir, n)
d := filepath.Join(dstDir, n)
if entry.IsDir() {
if err := copyTree(tw, d, s); err != nil {
return fmt.Errorf("unable to copy dir %v to %v: %v", s, d, err)
}
continue
}
if err := copyFile(tw, d, s); err != nil {
return fmt.Errorf("unable to copy dir %v to %v: %v", s, d, err)
}
}
return nil
}
// copyFile copies src to tar file dst.
func copyFile(tw *tar.Writer, dst, src string) error {
s, err := os.Open(src)
if err != nil {
return fmt.Errorf("unable to open %v: %v", src, err)
}
defer s.Close()
fi, err := s.Stat()
if err != nil {
return fmt.Errorf("unable to stat %v: %v", src, err)
}
hdr, err := tar.FileInfoHeader(fi, dst)
if err != nil {
return fmt.Errorf("unable to create tar header for %v: %v", dst, err)
}
hdr.Name = dst
if err := tw.WriteHeader(hdr); err != nil {
return fmt.Errorf("unable to write header for %v: %v", dst, err)
}
_, err = io.Copy(tw, s)
if err != nil {
return fmt.Errorf("unable to copy %v to %v: %v", src, dst, err)
}
return nil
}
// checkMain verifies that there is a single "main" function.
// It also returns a list of all Go source files in the app.
func checkMain(ctxt *build.Context) (bool, []string, error) {
pkg, err := ctxt.ImportDir(*rootDir, 0)
if err != nil {
return false, nil, fmt.Errorf("unable to analyze source: %v", err)
}
if !pkg.IsCommand() {
errorf("Your app's package needs to be changed from %q to \"main\".\n", pkg.Name)
}
// Search for a "func main"
var hasMain bool
var appFiles []string
for _, f := range pkg.GoFiles {
n := filepath.Join(*rootDir, f)
appFiles = append(appFiles, n)
if hasMain, err = readFile(n); err != nil {
return false, nil, fmt.Errorf("error parsing %q: %v", n, err)
}
}
return hasMain, appFiles, nil
}
// isMain returns whether the given function declaration is a main function.
// Such a function must be called "main", not have a receiver, and have no arguments or return types.
func isMain(f *ast.FuncDecl) bool {
ft := f.Type
return f.Name.Name == "main" && f.Recv == nil && ft.Params.NumFields() == 0 && ft.Results.NumFields() == 0
}
// readFile reads and parses the Go source code file and returns whether it has a main function.
func readFile(filename string) (hasMain bool, err error) {
var src []byte
src, err = ioutil.ReadFile(filename)
if err != nil {
return
}
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, filename, src, 0)
for _, decl := range file.Decls {
funcDecl, ok := decl.(*ast.FuncDecl)
if !ok {
continue
}
if !isMain(funcDecl) {
continue
}
hasMain = true
break
}
return
}
appengine-1.6.7/cmd/aedeploy/ 0000775 0000000 0000000 00000000000 13741174344 0016050 5 ustar 00root root 0000000 0000000 appengine-1.6.7/cmd/aedeploy/aedeploy.go 0000664 0000000 0000000 00000003762 13741174344 0020211 0 ustar 00root root 0000000 0000000 // Copyright 2015 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Program aedeploy assists with deploying App Engine "flexible environment" Go apps to production.
// A temporary directory is created; the app, its subdirectories, and all its
// dependencies from $GOPATH are copied into the directory; then the app
// is deployed to production with the provided command.
//
// The app must be in "package main".
//
// This command must be issued from within the root directory of the app
// (where the app.yaml file is located).
package main
import (
"flag"
"fmt"
"log"
"os"
"os/exec"
"strings"
)
func usage() {
fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0])
fmt.Fprintf(os.Stderr, "\t%s gcloud --verbosity debug app deploy --version myversion ./app.yaml\tDeploy app to production\n", os.Args[0])
}
var verbose bool
// vlogf logs to stderr if the "-v" flag is provided.
func vlogf(f string, v ...interface{}) {
if !verbose {
return
}
log.Printf("[aedeploy] "+f, v...)
}
func main() {
flag.BoolVar(&verbose, "v", false, "Verbose logging.")
flag.Usage = usage
flag.Parse()
if flag.NArg() < 1 {
usage()
os.Exit(1)
}
notice := func() {
fmt.Fprintln(os.Stderr, `NOTICE: aedeploy is deprecated. Just use "gcloud app deploy".`)
}
notice()
if err := deploy(); err != nil {
fmt.Fprintf(os.Stderr, os.Args[0]+": Error: %v\n", err)
notice()
fmt.Fprintln(os.Stderr, `You might need to update gcloud. Run "gcloud components update".`)
os.Exit(1)
}
notice() // Make sure they see it at the end.
}
// deploy calls the provided command to deploy the app from the temporary directory.
func deploy() error {
vlogf("Running command %v", flag.Args())
cmd := exec.Command(flag.Arg(0), flag.Args()[1:]...)
cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
if err := cmd.Run(); err != nil {
return fmt.Errorf("unable to run %q: %v", strings.Join(flag.Args(), " "), err)
}
return nil
}
appengine-1.6.7/cmd/aefix/ 0000775 0000000 0000000 00000000000 13741174344 0015342 5 ustar 00root root 0000000 0000000 appengine-1.6.7/cmd/aefix/ae.go 0000664 0000000 0000000 00000011044 13741174344 0016256 0 ustar 00root root 0000000 0000000 // Copyright 2016 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package main
import (
"go/ast"
"path"
"strconv"
"strings"
)
const (
ctxPackage = "golang.org/x/net/context"
newPackageBase = "google.golang.org/"
stutterPackage = false
)
func init() {
register(fix{
"ae",
"2016-04-15",
aeFn,
`Update old App Engine APIs to new App Engine APIs`,
})
}
// logMethod is the set of methods on appengine.Context used for logging.
var logMethod = map[string]bool{
"Debugf": true,
"Infof": true,
"Warningf": true,
"Errorf": true,
"Criticalf": true,
}
// mapPackage turns "appengine" into "google.golang.org/appengine", etc.
func mapPackage(s string) string {
if stutterPackage {
s += "/" + path.Base(s)
}
return newPackageBase + s
}
func aeFn(f *ast.File) bool {
// During the walk, we track the last thing seen that looks like
// an appengine.Context, and reset it once the walk leaves a func.
var lastContext *ast.Ident
fixed := false
// Update imports.
mainImp := "appengine"
for _, imp := range f.Imports {
pth, _ := strconv.Unquote(imp.Path.Value)
if pth == "appengine" || strings.HasPrefix(pth, "appengine/") {
newPth := mapPackage(pth)
imp.Path.Value = strconv.Quote(newPth)
fixed = true
if pth == "appengine" {
mainImp = newPth
}
}
}
// Update any API changes.
walk(f, func(n interface{}) {
if ft, ok := n.(*ast.FuncType); ok && ft.Params != nil {
// See if this func has an `appengine.Context arg`.
// If so, remember its identifier.
for _, param := range ft.Params.List {
if !isPkgDot(param.Type, "appengine", "Context") {
continue
}
if len(param.Names) == 1 {
lastContext = param.Names[0]
break
}
}
return
}
if as, ok := n.(*ast.AssignStmt); ok {
if len(as.Lhs) == 1 && len(as.Rhs) == 1 {
// If this node is an assignment from an appengine.NewContext invocation,
// remember the identifier on the LHS.
if isCall(as.Rhs[0], "appengine", "NewContext") {
if ident, ok := as.Lhs[0].(*ast.Ident); ok {
lastContext = ident
return
}
}
// x (=|:=) appengine.Timeout(y, z)
// should become
// x, _ (=|:=) context.WithTimeout(y, z)
if isCall(as.Rhs[0], "appengine", "Timeout") {
addImport(f, ctxPackage)
as.Lhs = append(as.Lhs, ast.NewIdent("_"))
// isCall already did the type checking.
sel := as.Rhs[0].(*ast.CallExpr).Fun.(*ast.SelectorExpr)
sel.X = ast.NewIdent("context")
sel.Sel = ast.NewIdent("WithTimeout")
fixed = true
return
}
}
return
}
// If this node is a FuncDecl, we've finished the function, so reset lastContext.
if _, ok := n.(*ast.FuncDecl); ok {
lastContext = nil
return
}
if call, ok := n.(*ast.CallExpr); ok {
if isPkgDot(call.Fun, "appengine", "Datacenter") && len(call.Args) == 0 {
insertContext(f, call, lastContext)
fixed = true
return
}
if isPkgDot(call.Fun, "taskqueue", "QueueStats") && len(call.Args) == 3 {
call.Args = call.Args[:2] // drop last arg
fixed = true
return
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return
}
if lastContext != nil && refersTo(sel.X, lastContext) && logMethod[sel.Sel.Name] {
// c.Errorf(...)
// should become
// log.Errorf(c, ...)
addImport(f, mapPackage("appengine/log"))
sel.X = &ast.Ident{ // ast.NewIdent doesn't preserve the position.
NamePos: sel.X.Pos(),
Name: "log",
}
insertContext(f, call, lastContext)
fixed = true
return
}
}
})
// Change any `appengine.Context` to `context.Context`.
// Do this in a separate walk because the previous walk
// wants to identify "appengine.Context".
walk(f, func(n interface{}) {
expr, ok := n.(ast.Expr)
if ok && isPkgDot(expr, "appengine", "Context") {
addImport(f, ctxPackage)
// isPkgDot did the type checking.
n.(*ast.SelectorExpr).X.(*ast.Ident).Name = "context"
fixed = true
return
}
})
// The changes above might remove the need to import "appengine".
// Check if it's used, and drop it if it isn't.
if fixed && !usesImport(f, mainImp) {
deleteImport(f, mainImp)
}
return fixed
}
// ctx may be nil.
func insertContext(f *ast.File, call *ast.CallExpr, ctx *ast.Ident) {
if ctx == nil {
// context is unknown, so use a plain "ctx".
ctx = ast.NewIdent("ctx")
} else {
// Create a fresh *ast.Ident so we drop the position information.
ctx = ast.NewIdent(ctx.Name)
}
call.Args = append([]ast.Expr{ctx}, call.Args...)
}
appengine-1.6.7/cmd/aefix/ae_test.go 0000664 0000000 0000000 00000005061 13741174344 0017317 0 ustar 00root root 0000000 0000000 // Copyright 2016 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package main
func init() {
addTestCases(aeTests, nil)
}
var aeTests = []testCase{
// Collection of fixes:
// - imports
// - appengine.Timeout -> context.WithTimeout
// - add ctx arg to appengine.Datacenter
// - logging API
{
Name: "ae.0",
In: `package foo
import (
"net/http"
"time"
"appengine"
"appengine/datastore"
)
func f(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
c = appengine.Timeout(c, 5*time.Second)
err := datastore.ErrNoSuchEntity
c.Errorf("Something interesting happened: %v", err)
_ = appengine.Datacenter()
}
`,
Out: `package foo
import (
"net/http"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/log"
)
func f(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
c, _ = context.WithTimeout(c, 5*time.Second)
err := datastore.ErrNoSuchEntity
log.Errorf(c, "Something interesting happened: %v", err)
_ = appengine.Datacenter(c)
}
`,
},
// Updating a function that takes an appengine.Context arg.
{
Name: "ae.1",
In: `package foo
import (
"appengine"
)
func LogSomething(c2 appengine.Context) {
c2.Warningf("Stand back! I'm going to try science!")
}
`,
Out: `package foo
import (
"golang.org/x/net/context"
"google.golang.org/appengine/log"
)
func LogSomething(c2 context.Context) {
log.Warningf(c2, "Stand back! I'm going to try science!")
}
`,
},
// Less widely used API changes:
// - drop maxTasks arg to taskqueue.QueueStats
{
Name: "ae.2",
In: `package foo
import (
"appengine"
"appengine/taskqueue"
)
func f(ctx appengine.Context) {
stats, err := taskqueue.QueueStats(ctx, []string{"one", "two"}, 0)
}
`,
Out: `package foo
import (
"golang.org/x/net/context"
"google.golang.org/appengine/taskqueue"
)
func f(ctx context.Context) {
stats, err := taskqueue.QueueStats(ctx, []string{"one", "two"})
}
`,
},
// Check that the main "appengine" import will not be dropped
// if an appengine.Context -> context.Context change happens
// but the appengine package is still referenced.
{
Name: "ae.3",
In: `package foo
import (
"appengine"
"io"
)
func f(ctx appengine.Context, w io.Writer) {
_ = appengine.IsDevAppServer()
}
`,
Out: `package foo
import (
"golang.org/x/net/context"
"google.golang.org/appengine"
"io"
)
func f(ctx context.Context, w io.Writer) {
_ = appengine.IsDevAppServer()
}
`,
},
}
appengine-1.6.7/cmd/aefix/fix.go 0000664 0000000 0000000 00000051104 13741174344 0016460 0 ustar 00root root 0000000 0000000 // Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package main
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"os"
"path"
"reflect"
"strconv"
"strings"
)
type fix struct {
name string
date string // date that fix was introduced, in YYYY-MM-DD format
f func(*ast.File) bool
desc string
}
// main runs sort.Sort(byName(fixes)) before printing list of fixes.
type byName []fix
func (f byName) Len() int { return len(f) }
func (f byName) Swap(i, j int) { f[i], f[j] = f[j], f[i] }
func (f byName) Less(i, j int) bool { return f[i].name < f[j].name }
// main runs sort.Sort(byDate(fixes)) before applying fixes.
type byDate []fix
func (f byDate) Len() int { return len(f) }
func (f byDate) Swap(i, j int) { f[i], f[j] = f[j], f[i] }
func (f byDate) Less(i, j int) bool { return f[i].date < f[j].date }
var fixes []fix
func register(f fix) {
fixes = append(fixes, f)
}
// walk traverses the AST x, calling visit(y) for each node y in the tree but
// also with a pointer to each ast.Expr, ast.Stmt, and *ast.BlockStmt,
// in a bottom-up traversal.
func walk(x interface{}, visit func(interface{})) {
walkBeforeAfter(x, nop, visit)
}
func nop(interface{}) {}
// walkBeforeAfter is like walk but calls before(x) before traversing
// x's children and after(x) afterward.
func walkBeforeAfter(x interface{}, before, after func(interface{})) {
before(x)
switch n := x.(type) {
default:
panic(fmt.Errorf("unexpected type %T in walkBeforeAfter", x))
case nil:
// pointers to interfaces
case *ast.Decl:
walkBeforeAfter(*n, before, after)
case *ast.Expr:
walkBeforeAfter(*n, before, after)
case *ast.Spec:
walkBeforeAfter(*n, before, after)
case *ast.Stmt:
walkBeforeAfter(*n, before, after)
// pointers to struct pointers
case **ast.BlockStmt:
walkBeforeAfter(*n, before, after)
case **ast.CallExpr:
walkBeforeAfter(*n, before, after)
case **ast.FieldList:
walkBeforeAfter(*n, before, after)
case **ast.FuncType:
walkBeforeAfter(*n, before, after)
case **ast.Ident:
walkBeforeAfter(*n, before, after)
case **ast.BasicLit:
walkBeforeAfter(*n, before, after)
// pointers to slices
case *[]ast.Decl:
walkBeforeAfter(*n, before, after)
case *[]ast.Expr:
walkBeforeAfter(*n, before, after)
case *[]*ast.File:
walkBeforeAfter(*n, before, after)
case *[]*ast.Ident:
walkBeforeAfter(*n, before, after)
case *[]ast.Spec:
walkBeforeAfter(*n, before, after)
case *[]ast.Stmt:
walkBeforeAfter(*n, before, after)
// These are ordered and grouped to match ../../pkg/go/ast/ast.go
case *ast.Field:
walkBeforeAfter(&n.Names, before, after)
walkBeforeAfter(&n.Type, before, after)
walkBeforeAfter(&n.Tag, before, after)
case *ast.FieldList:
for _, field := range n.List {
walkBeforeAfter(field, before, after)
}
case *ast.BadExpr:
case *ast.Ident:
case *ast.Ellipsis:
walkBeforeAfter(&n.Elt, before, after)
case *ast.BasicLit:
case *ast.FuncLit:
walkBeforeAfter(&n.Type, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.CompositeLit:
walkBeforeAfter(&n.Type, before, after)
walkBeforeAfter(&n.Elts, before, after)
case *ast.ParenExpr:
walkBeforeAfter(&n.X, before, after)
case *ast.SelectorExpr:
walkBeforeAfter(&n.X, before, after)
case *ast.IndexExpr:
walkBeforeAfter(&n.X, before, after)
walkBeforeAfter(&n.Index, before, after)
case *ast.SliceExpr:
walkBeforeAfter(&n.X, before, after)
if n.Low != nil {
walkBeforeAfter(&n.Low, before, after)
}
if n.High != nil {
walkBeforeAfter(&n.High, before, after)
}
case *ast.TypeAssertExpr:
walkBeforeAfter(&n.X, before, after)
walkBeforeAfter(&n.Type, before, after)
case *ast.CallExpr:
walkBeforeAfter(&n.Fun, before, after)
walkBeforeAfter(&n.Args, before, after)
case *ast.StarExpr:
walkBeforeAfter(&n.X, before, after)
case *ast.UnaryExpr:
walkBeforeAfter(&n.X, before, after)
case *ast.BinaryExpr:
walkBeforeAfter(&n.X, before, after)
walkBeforeAfter(&n.Y, before, after)
case *ast.KeyValueExpr:
walkBeforeAfter(&n.Key, before, after)
walkBeforeAfter(&n.Value, before, after)
case *ast.ArrayType:
walkBeforeAfter(&n.Len, before, after)
walkBeforeAfter(&n.Elt, before, after)
case *ast.StructType:
walkBeforeAfter(&n.Fields, before, after)
case *ast.FuncType:
walkBeforeAfter(&n.Params, before, after)
if n.Results != nil {
walkBeforeAfter(&n.Results, before, after)
}
case *ast.InterfaceType:
walkBeforeAfter(&n.Methods, before, after)
case *ast.MapType:
walkBeforeAfter(&n.Key, before, after)
walkBeforeAfter(&n.Value, before, after)
case *ast.ChanType:
walkBeforeAfter(&n.Value, before, after)
case *ast.BadStmt:
case *ast.DeclStmt:
walkBeforeAfter(&n.Decl, before, after)
case *ast.EmptyStmt:
case *ast.LabeledStmt:
walkBeforeAfter(&n.Stmt, before, after)
case *ast.ExprStmt:
walkBeforeAfter(&n.X, before, after)
case *ast.SendStmt:
walkBeforeAfter(&n.Chan, before, after)
walkBeforeAfter(&n.Value, before, after)
case *ast.IncDecStmt:
walkBeforeAfter(&n.X, before, after)
case *ast.AssignStmt:
walkBeforeAfter(&n.Lhs, before, after)
walkBeforeAfter(&n.Rhs, before, after)
case *ast.GoStmt:
walkBeforeAfter(&n.Call, before, after)
case *ast.DeferStmt:
walkBeforeAfter(&n.Call, before, after)
case *ast.ReturnStmt:
walkBeforeAfter(&n.Results, before, after)
case *ast.BranchStmt:
case *ast.BlockStmt:
walkBeforeAfter(&n.List, before, after)
case *ast.IfStmt:
walkBeforeAfter(&n.Init, before, after)
walkBeforeAfter(&n.Cond, before, after)
walkBeforeAfter(&n.Body, before, after)
walkBeforeAfter(&n.Else, before, after)
case *ast.CaseClause:
walkBeforeAfter(&n.List, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.SwitchStmt:
walkBeforeAfter(&n.Init, before, after)
walkBeforeAfter(&n.Tag, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.TypeSwitchStmt:
walkBeforeAfter(&n.Init, before, after)
walkBeforeAfter(&n.Assign, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.CommClause:
walkBeforeAfter(&n.Comm, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.SelectStmt:
walkBeforeAfter(&n.Body, before, after)
case *ast.ForStmt:
walkBeforeAfter(&n.Init, before, after)
walkBeforeAfter(&n.Cond, before, after)
walkBeforeAfter(&n.Post, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.RangeStmt:
walkBeforeAfter(&n.Key, before, after)
walkBeforeAfter(&n.Value, before, after)
walkBeforeAfter(&n.X, before, after)
walkBeforeAfter(&n.Body, before, after)
case *ast.ImportSpec:
case *ast.ValueSpec:
walkBeforeAfter(&n.Type, before, after)
walkBeforeAfter(&n.Values, before, after)
walkBeforeAfter(&n.Names, before, after)
case *ast.TypeSpec:
walkBeforeAfter(&n.Type, before, after)
case *ast.BadDecl:
case *ast.GenDecl:
walkBeforeAfter(&n.Specs, before, after)
case *ast.FuncDecl:
if n.Recv != nil {
walkBeforeAfter(&n.Recv, before, after)
}
walkBeforeAfter(&n.Type, before, after)
if n.Body != nil {
walkBeforeAfter(&n.Body, before, after)
}
case *ast.File:
walkBeforeAfter(&n.Decls, before, after)
case *ast.Package:
walkBeforeAfter(&n.Files, before, after)
case []*ast.File:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
case []ast.Decl:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
case []ast.Expr:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
case []*ast.Ident:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
case []ast.Stmt:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
case []ast.Spec:
for i := range n {
walkBeforeAfter(&n[i], before, after)
}
}
after(x)
}
// imports returns true if f imports path.
func imports(f *ast.File, path string) bool {
return importSpec(f, path) != nil
}
// importSpec returns the import spec if f imports path,
// or nil otherwise.
func importSpec(f *ast.File, path string) *ast.ImportSpec {
for _, s := range f.Imports {
if importPath(s) == path {
return s
}
}
return nil
}
// importPath returns the unquoted import path of s,
// or "" if the path is not properly quoted.
func importPath(s *ast.ImportSpec) string {
t, err := strconv.Unquote(s.Path.Value)
if err == nil {
return t
}
return ""
}
// declImports reports whether gen contains an import of path.
func declImports(gen *ast.GenDecl, path string) bool {
if gen.Tok != token.IMPORT {
return false
}
for _, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
if importPath(impspec) == path {
return true
}
}
return false
}
// isPkgDot returns true if t is the expression "pkg.name"
// where pkg is an imported identifier.
func isPkgDot(t ast.Expr, pkg, name string) bool {
sel, ok := t.(*ast.SelectorExpr)
return ok && isTopName(sel.X, pkg) && sel.Sel.String() == name
}
// isPtrPkgDot returns true if f is the expression "*pkg.name"
// where pkg is an imported identifier.
func isPtrPkgDot(t ast.Expr, pkg, name string) bool {
ptr, ok := t.(*ast.StarExpr)
return ok && isPkgDot(ptr.X, pkg, name)
}
// isTopName returns true if n is a top-level unresolved identifier with the given name.
func isTopName(n ast.Expr, name string) bool {
id, ok := n.(*ast.Ident)
return ok && id.Name == name && id.Obj == nil
}
// isName returns true if n is an identifier with the given name.
func isName(n ast.Expr, name string) bool {
id, ok := n.(*ast.Ident)
return ok && id.String() == name
}
// isCall returns true if t is a call to pkg.name.
func isCall(t ast.Expr, pkg, name string) bool {
call, ok := t.(*ast.CallExpr)
return ok && isPkgDot(call.Fun, pkg, name)
}
// If n is an *ast.Ident, isIdent returns it; otherwise isIdent returns nil.
func isIdent(n interface{}) *ast.Ident {
id, _ := n.(*ast.Ident)
return id
}
// refersTo returns true if n is a reference to the same object as x.
func refersTo(n ast.Node, x *ast.Ident) bool {
id, ok := n.(*ast.Ident)
// The test of id.Name == x.Name handles top-level unresolved
// identifiers, which all have Obj == nil.
return ok && id.Obj == x.Obj && id.Name == x.Name
}
// isBlank returns true if n is the blank identifier.
func isBlank(n ast.Expr) bool {
return isName(n, "_")
}
// isEmptyString returns true if n is an empty string literal.
func isEmptyString(n ast.Expr) bool {
lit, ok := n.(*ast.BasicLit)
return ok && lit.Kind == token.STRING && len(lit.Value) == 2
}
func warn(pos token.Pos, msg string, args ...interface{}) {
if pos.IsValid() {
msg = "%s: " + msg
arg1 := []interface{}{fset.Position(pos).String()}
args = append(arg1, args...)
}
fmt.Fprintf(os.Stderr, msg+"\n", args...)
}
// countUses returns the number of uses of the identifier x in scope.
func countUses(x *ast.Ident, scope []ast.Stmt) int {
count := 0
ff := func(n interface{}) {
if n, ok := n.(ast.Node); ok && refersTo(n, x) {
count++
}
}
for _, n := range scope {
walk(n, ff)
}
return count
}
// rewriteUses replaces all uses of the identifier x and !x in scope
// with f(x.Pos()) and fnot(x.Pos()).
func rewriteUses(x *ast.Ident, f, fnot func(token.Pos) ast.Expr, scope []ast.Stmt) {
var lastF ast.Expr
ff := func(n interface{}) {
ptr, ok := n.(*ast.Expr)
if !ok {
return
}
nn := *ptr
// The child node was just walked and possibly replaced.
// If it was replaced and this is a negation, replace with fnot(p).
not, ok := nn.(*ast.UnaryExpr)
if ok && not.Op == token.NOT && not.X == lastF {
*ptr = fnot(nn.Pos())
return
}
if refersTo(nn, x) {
lastF = f(nn.Pos())
*ptr = lastF
}
}
for _, n := range scope {
walk(n, ff)
}
}
// assignsTo returns true if any of the code in scope assigns to or takes the address of x.
func assignsTo(x *ast.Ident, scope []ast.Stmt) bool {
assigned := false
ff := func(n interface{}) {
if assigned {
return
}
switch n := n.(type) {
case *ast.UnaryExpr:
// use of &x
if n.Op == token.AND && refersTo(n.X, x) {
assigned = true
return
}
case *ast.AssignStmt:
for _, l := range n.Lhs {
if refersTo(l, x) {
assigned = true
return
}
}
}
}
for _, n := range scope {
if assigned {
break
}
walk(n, ff)
}
return assigned
}
// newPkgDot returns an ast.Expr referring to "pkg.name" at position pos.
func newPkgDot(pos token.Pos, pkg, name string) ast.Expr {
return &ast.SelectorExpr{
X: &ast.Ident{
NamePos: pos,
Name: pkg,
},
Sel: &ast.Ident{
NamePos: pos,
Name: name,
},
}
}
// renameTop renames all references to the top-level name old.
// It returns true if it makes any changes.
func renameTop(f *ast.File, old, new string) bool {
var fixed bool
// Rename any conflicting imports
// (assuming package name is last element of path).
for _, s := range f.Imports {
if s.Name != nil {
if s.Name.Name == old {
s.Name.Name = new
fixed = true
}
} else {
_, thisName := path.Split(importPath(s))
if thisName == old {
s.Name = ast.NewIdent(new)
fixed = true
}
}
}
// Rename any top-level declarations.
for _, d := range f.Decls {
switch d := d.(type) {
case *ast.FuncDecl:
if d.Recv == nil && d.Name.Name == old {
d.Name.Name = new
d.Name.Obj.Name = new
fixed = true
}
case *ast.GenDecl:
for _, s := range d.Specs {
switch s := s.(type) {
case *ast.TypeSpec:
if s.Name.Name == old {
s.Name.Name = new
s.Name.Obj.Name = new
fixed = true
}
case *ast.ValueSpec:
for _, n := range s.Names {
if n.Name == old {
n.Name = new
n.Obj.Name = new
fixed = true
}
}
}
}
}
}
// Rename top-level old to new, both unresolved names
// (probably defined in another file) and names that resolve
// to a declaration we renamed.
walk(f, func(n interface{}) {
id, ok := n.(*ast.Ident)
if ok && isTopName(id, old) {
id.Name = new
fixed = true
}
if ok && id.Obj != nil && id.Name == old && id.Obj.Name == new {
id.Name = id.Obj.Name
fixed = true
}
})
return fixed
}
// matchLen returns the length of the longest prefix shared by x and y.
func matchLen(x, y string) int {
i := 0
for i < len(x) && i < len(y) && x[i] == y[i] {
i++
}
return i
}
// addImport adds the import path to the file f, if absent.
func addImport(f *ast.File, ipath string) (added bool) {
if imports(f, ipath) {
return false
}
// Determine name of import.
// Assume added imports follow convention of using last element.
_, name := path.Split(ipath)
// Rename any conflicting top-level references from name to name_.
renameTop(f, name, name+"_")
newImport := &ast.ImportSpec{
Path: &ast.BasicLit{
Kind: token.STRING,
Value: strconv.Quote(ipath),
},
}
// Find an import decl to add to.
var (
bestMatch = -1
lastImport = -1
impDecl *ast.GenDecl
impIndex = -1
)
for i, decl := range f.Decls {
gen, ok := decl.(*ast.GenDecl)
if ok && gen.Tok == token.IMPORT {
lastImport = i
// Do not add to import "C", to avoid disrupting the
// association with its doc comment, breaking cgo.
if declImports(gen, "C") {
continue
}
// Compute longest shared prefix with imports in this block.
for j, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
n := matchLen(importPath(impspec), ipath)
if n > bestMatch {
bestMatch = n
impDecl = gen
impIndex = j
}
}
}
}
// If no import decl found, add one after the last import.
if impDecl == nil {
impDecl = &ast.GenDecl{
Tok: token.IMPORT,
}
f.Decls = append(f.Decls, nil)
copy(f.Decls[lastImport+2:], f.Decls[lastImport+1:])
f.Decls[lastImport+1] = impDecl
}
// Ensure the import decl has parentheses, if needed.
if len(impDecl.Specs) > 0 && !impDecl.Lparen.IsValid() {
impDecl.Lparen = impDecl.Pos()
}
insertAt := impIndex + 1
if insertAt == 0 {
insertAt = len(impDecl.Specs)
}
impDecl.Specs = append(impDecl.Specs, nil)
copy(impDecl.Specs[insertAt+1:], impDecl.Specs[insertAt:])
impDecl.Specs[insertAt] = newImport
if insertAt > 0 {
// Assign same position as the previous import,
// so that the sorter sees it as being in the same block.
prev := impDecl.Specs[insertAt-1]
newImport.Path.ValuePos = prev.Pos()
newImport.EndPos = prev.Pos()
}
f.Imports = append(f.Imports, newImport)
return true
}
// deleteImport deletes the import path from the file f, if present.
func deleteImport(f *ast.File, path string) (deleted bool) {
oldImport := importSpec(f, path)
// Find the import node that imports path, if any.
for i, decl := range f.Decls {
gen, ok := decl.(*ast.GenDecl)
if !ok || gen.Tok != token.IMPORT {
continue
}
for j, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
if oldImport != impspec {
continue
}
// We found an import spec that imports path.
// Delete it.
deleted = true
copy(gen.Specs[j:], gen.Specs[j+1:])
gen.Specs = gen.Specs[:len(gen.Specs)-1]
// If this was the last import spec in this decl,
// delete the decl, too.
if len(gen.Specs) == 0 {
copy(f.Decls[i:], f.Decls[i+1:])
f.Decls = f.Decls[:len(f.Decls)-1]
} else if len(gen.Specs) == 1 {
gen.Lparen = token.NoPos // drop parens
}
if j > 0 {
// We deleted an entry but now there will be
// a blank line-sized hole where the import was.
// Close the hole by making the previous
// import appear to "end" where this one did.
gen.Specs[j-1].(*ast.ImportSpec).EndPos = impspec.End()
}
break
}
}
// Delete it from f.Imports.
for i, imp := range f.Imports {
if imp == oldImport {
copy(f.Imports[i:], f.Imports[i+1:])
f.Imports = f.Imports[:len(f.Imports)-1]
break
}
}
return
}
// rewriteImport rewrites any import of path oldPath to path newPath.
func rewriteImport(f *ast.File, oldPath, newPath string) (rewrote bool) {
for _, imp := range f.Imports {
if importPath(imp) == oldPath {
rewrote = true
// record old End, because the default is to compute
// it using the length of imp.Path.Value.
imp.EndPos = imp.End()
imp.Path.Value = strconv.Quote(newPath)
}
}
return
}
func usesImport(f *ast.File, path string) (used bool) {
spec := importSpec(f, path)
if spec == nil {
return
}
name := spec.Name.String()
switch name {
case "":
// If the package name is not explicitly specified,
// make an educated guess. This is not guaranteed to be correct.
lastSlash := strings.LastIndex(path, "/")
if lastSlash == -1 {
name = path
} else {
name = path[lastSlash+1:]
}
case "_", ".":
// Not sure if this import is used - err on the side of caution.
return true
}
walk(f, func(n interface{}) {
sel, ok := n.(*ast.SelectorExpr)
if ok && isTopName(sel.X, name) {
used = true
}
})
return
}
func expr(s string) ast.Expr {
x, err := parser.ParseExpr(s)
if err != nil {
panic("parsing " + s + ": " + err.Error())
}
// Remove position information to avoid spurious newlines.
killPos(reflect.ValueOf(x))
return x
}
var posType = reflect.TypeOf(token.Pos(0))
func killPos(v reflect.Value) {
switch v.Kind() {
case reflect.Ptr, reflect.Interface:
if !v.IsNil() {
killPos(v.Elem())
}
case reflect.Slice:
n := v.Len()
for i := 0; i < n; i++ {
killPos(v.Index(i))
}
case reflect.Struct:
n := v.NumField()
for i := 0; i < n; i++ {
f := v.Field(i)
if f.Type() == posType {
f.SetInt(0)
continue
}
killPos(f)
}
}
}
// A Rename describes a single renaming.
type rename struct {
OldImport string // only apply rename if this import is present
NewImport string // add this import during rewrite
Old string // old name: p.T or *p.T
New string // new name: p.T or *p.T
}
func renameFix(tab []rename) func(*ast.File) bool {
return func(f *ast.File) bool {
return renameFixTab(f, tab)
}
}
func parseName(s string) (ptr bool, pkg, nam string) {
i := strings.Index(s, ".")
if i < 0 {
panic("parseName: invalid name " + s)
}
if strings.HasPrefix(s, "*") {
ptr = true
s = s[1:]
i--
}
pkg = s[:i]
nam = s[i+1:]
return
}
func renameFixTab(f *ast.File, tab []rename) bool {
fixed := false
added := map[string]bool{}
check := map[string]bool{}
for _, t := range tab {
if !imports(f, t.OldImport) {
continue
}
optr, opkg, onam := parseName(t.Old)
walk(f, func(n interface{}) {
np, ok := n.(*ast.Expr)
if !ok {
return
}
x := *np
if optr {
p, ok := x.(*ast.StarExpr)
if !ok {
return
}
x = p.X
}
if !isPkgDot(x, opkg, onam) {
return
}
if t.NewImport != "" && !added[t.NewImport] {
addImport(f, t.NewImport)
added[t.NewImport] = true
}
*np = expr(t.New)
check[t.OldImport] = true
fixed = true
})
}
for ipath := range check {
if !usesImport(f, ipath) {
deleteImport(f, ipath)
}
}
return fixed
}
appengine-1.6.7/cmd/aefix/main.go 0000664 0000000 0000000 00000012477 13741174344 0016630 0 ustar 00root root 0000000 0000000 // Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package main
import (
"bytes"
"flag"
"fmt"
"go/ast"
"go/format"
"go/parser"
"go/scanner"
"go/token"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
)
var (
fset = token.NewFileSet()
exitCode = 0
)
var allowedRewrites = flag.String("r", "",
"restrict the rewrites to this comma-separated list")
var forceRewrites = flag.String("force", "",
"force these fixes to run even if the code looks updated")
var allowed, force map[string]bool
var doDiff = flag.Bool("diff", false, "display diffs instead of rewriting files")
// enable for debugging fix failures
const debug = false // display incorrectly reformatted source and exit
func usage() {
fmt.Fprintf(os.Stderr, "usage: aefix [-diff] [-r fixname,...] [-force fixname,...] [path ...]\n")
flag.PrintDefaults()
fmt.Fprintf(os.Stderr, "\nAvailable rewrites are:\n")
sort.Sort(byName(fixes))
for _, f := range fixes {
fmt.Fprintf(os.Stderr, "\n%s\n", f.name)
desc := strings.TrimSpace(f.desc)
desc = strings.Replace(desc, "\n", "\n\t", -1)
fmt.Fprintf(os.Stderr, "\t%s\n", desc)
}
os.Exit(2)
}
func main() {
flag.Usage = usage
flag.Parse()
sort.Sort(byDate(fixes))
if *allowedRewrites != "" {
allowed = make(map[string]bool)
for _, f := range strings.Split(*allowedRewrites, ",") {
allowed[f] = true
}
}
if *forceRewrites != "" {
force = make(map[string]bool)
for _, f := range strings.Split(*forceRewrites, ",") {
force[f] = true
}
}
if flag.NArg() == 0 {
if err := processFile("standard input", true); err != nil {
report(err)
}
os.Exit(exitCode)
}
for i := 0; i < flag.NArg(); i++ {
path := flag.Arg(i)
switch dir, err := os.Stat(path); {
case err != nil:
report(err)
case dir.IsDir():
walkDir(path)
default:
if err := processFile(path, false); err != nil {
report(err)
}
}
}
os.Exit(exitCode)
}
const parserMode = parser.ParseComments
func gofmtFile(f *ast.File) ([]byte, error) {
var buf bytes.Buffer
if err := format.Node(&buf, fset, f); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func processFile(filename string, useStdin bool) error {
var f *os.File
var err error
var fixlog bytes.Buffer
if useStdin {
f = os.Stdin
} else {
f, err = os.Open(filename)
if err != nil {
return err
}
defer f.Close()
}
src, err := ioutil.ReadAll(f)
if err != nil {
return err
}
file, err := parser.ParseFile(fset, filename, src, parserMode)
if err != nil {
return err
}
// Apply all fixes to file.
newFile := file
fixed := false
for _, fix := range fixes {
if allowed != nil && !allowed[fix.name] {
continue
}
if fix.f(newFile) {
fixed = true
fmt.Fprintf(&fixlog, " %s", fix.name)
// AST changed.
// Print and parse, to update any missing scoping
// or position information for subsequent fixers.
newSrc, err := gofmtFile(newFile)
if err != nil {
return err
}
newFile, err = parser.ParseFile(fset, filename, newSrc, parserMode)
if err != nil {
if debug {
fmt.Printf("%s", newSrc)
report(err)
os.Exit(exitCode)
}
return err
}
}
}
if !fixed {
return nil
}
fmt.Fprintf(os.Stderr, "%s: fixed %s\n", filename, fixlog.String()[1:])
// Print AST. We did that after each fix, so this appears
// redundant, but it is necessary to generate gofmt-compatible
// source code in a few cases. The official gofmt style is the
// output of the printer run on a standard AST generated by the parser,
// but the source we generated inside the loop above is the
// output of the printer run on a mangled AST generated by a fixer.
newSrc, err := gofmtFile(newFile)
if err != nil {
return err
}
if *doDiff {
data, err := diff(src, newSrc)
if err != nil {
return fmt.Errorf("computing diff: %s", err)
}
fmt.Printf("diff %s fixed/%s\n", filename, filename)
os.Stdout.Write(data)
return nil
}
if useStdin {
os.Stdout.Write(newSrc)
return nil
}
return ioutil.WriteFile(f.Name(), newSrc, 0)
}
var gofmtBuf bytes.Buffer
func gofmt(n interface{}) string {
gofmtBuf.Reset()
if err := format.Node(&gofmtBuf, fset, n); err != nil {
return "<" + err.Error() + ">"
}
return gofmtBuf.String()
}
func report(err error) {
scanner.PrintError(os.Stderr, err)
exitCode = 2
}
func walkDir(path string) {
filepath.Walk(path, visitFile)
}
func visitFile(path string, f os.FileInfo, err error) error {
if err == nil && isGoFile(f) {
err = processFile(path, false)
}
if err != nil {
report(err)
}
return nil
}
func isGoFile(f os.FileInfo) bool {
// ignore non-Go files
name := f.Name()
return !f.IsDir() && !strings.HasPrefix(name, ".") && strings.HasSuffix(name, ".go")
}
func diff(b1, b2 []byte) (data []byte, err error) {
f1, err := ioutil.TempFile("", "go-fix")
if err != nil {
return nil, err
}
defer os.Remove(f1.Name())
defer f1.Close()
f2, err := ioutil.TempFile("", "go-fix")
if err != nil {
return nil, err
}
defer os.Remove(f2.Name())
defer f2.Close()
f1.Write(b1)
f2.Write(b2)
data, err = exec.Command("diff", "-u", f1.Name(), f2.Name()).CombinedOutput()
if len(data) > 0 {
// diff exits with a non-zero status when the files don't match.
// Ignore that failure as long as we get output.
err = nil
}
return
}
appengine-1.6.7/cmd/aefix/main_test.go 0000664 0000000 0000000 00000005204 13741174344 0017655 0 ustar 00root root 0000000 0000000 // Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package main
import (
"go/ast"
"go/parser"
"strings"
"testing"
)
type testCase struct {
Name string
Fn func(*ast.File) bool
In string
Out string
}
var testCases []testCase
func addTestCases(t []testCase, fn func(*ast.File) bool) {
// Fill in fn to avoid repetition in definitions.
if fn != nil {
for i := range t {
if t[i].Fn == nil {
t[i].Fn = fn
}
}
}
testCases = append(testCases, t...)
}
func fnop(*ast.File) bool { return false }
func parseFixPrint(t *testing.T, fn func(*ast.File) bool, desc, in string, mustBeGofmt bool) (out string, fixed, ok bool) {
file, err := parser.ParseFile(fset, desc, in, parserMode)
if err != nil {
t.Errorf("%s: parsing: %v", desc, err)
return
}
outb, err := gofmtFile(file)
if err != nil {
t.Errorf("%s: printing: %v", desc, err)
return
}
if s := string(outb); in != s && mustBeGofmt {
t.Errorf("%s: not gofmt-formatted.\n--- %s\n%s\n--- %s | gofmt\n%s",
desc, desc, in, desc, s)
tdiff(t, in, s)
return
}
if fn == nil {
for _, fix := range fixes {
if fix.f(file) {
fixed = true
}
}
} else {
fixed = fn(file)
}
outb, err = gofmtFile(file)
if err != nil {
t.Errorf("%s: printing: %v", desc, err)
return
}
return string(outb), fixed, true
}
func TestRewrite(t *testing.T) {
for _, tt := range testCases {
// Apply fix: should get tt.Out.
out, fixed, ok := parseFixPrint(t, tt.Fn, tt.Name, tt.In, true)
if !ok {
continue
}
// reformat to get printing right
out, _, ok = parseFixPrint(t, fnop, tt.Name, out, false)
if !ok {
continue
}
if out != tt.Out {
t.Errorf("%s: incorrect output.\n", tt.Name)
if !strings.HasPrefix(tt.Name, "testdata/") {
t.Errorf("--- have\n%s\n--- want\n%s", out, tt.Out)
}
tdiff(t, out, tt.Out)
continue
}
if changed := out != tt.In; changed != fixed {
t.Errorf("%s: changed=%v != fixed=%v", tt.Name, changed, fixed)
continue
}
// Should not change if run again.
out2, fixed2, ok := parseFixPrint(t, tt.Fn, tt.Name+" output", out, true)
if !ok {
continue
}
if fixed2 {
t.Errorf("%s: applied fixes during second round", tt.Name)
continue
}
if out2 != out {
t.Errorf("%s: changed output after second round of fixes.\n--- output after first round\n%s\n--- output after second round\n%s",
tt.Name, out, out2)
tdiff(t, out, out2)
}
}
}
func tdiff(t *testing.T, a, b string) {
data, err := diff([]byte(a), []byte(b))
if err != nil {
t.Error(err)
return
}
t.Error(string(data))
}
appengine-1.6.7/cmd/aefix/typecheck.go 0000664 0000000 0000000 00000040347 13741174344 0017660 0 ustar 00root root 0000000 0000000 // Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package main
import (
"fmt"
"go/ast"
"go/token"
"os"
"reflect"
"strings"
)
// Partial type checker.
//
// The fact that it is partial is very important: the input is
// an AST and a description of some type information to
// assume about one or more packages, but not all the
// packages that the program imports. The checker is
// expected to do as much as it can with what it has been
// given. There is not enough information supplied to do
// a full type check, but the type checker is expected to
// apply information that can be derived from variable
// declarations, function and method returns, and type switches
// as far as it can, so that the caller can still tell the types
// of expression relevant to a particular fix.
//
// TODO(rsc,gri): Replace with go/typechecker.
// Doing that could be an interesting test case for go/typechecker:
// the constraints about working with partial information will
// likely exercise it in interesting ways. The ideal interface would
// be to pass typecheck a map from importpath to package API text
// (Go source code), but for now we use data structures (TypeConfig, Type).
//
// The strings mostly use gofmt form.
//
// A Field or FieldList has as its type a comma-separated list
// of the types of the fields. For example, the field list
// x, y, z int
// has type "int, int, int".
// The prefix "type " is the type of a type.
// For example, given
// var x int
// type T int
// x's type is "int" but T's type is "type int".
// mkType inserts the "type " prefix.
// getType removes it.
// isType tests for it.
func mkType(t string) string {
return "type " + t
}
func getType(t string) string {
if !isType(t) {
return ""
}
return t[len("type "):]
}
func isType(t string) bool {
return strings.HasPrefix(t, "type ")
}
// TypeConfig describes the universe of relevant types.
// For ease of creation, the types are all referred to by string
// name (e.g., "reflect.Value"). TypeByName is the only place
// where the strings are resolved.
type TypeConfig struct {
Type map[string]*Type
Var map[string]string
Func map[string]string
}
// typeof returns the type of the given name, which may be of
// the form "x" or "p.X".
func (cfg *TypeConfig) typeof(name string) string {
if cfg.Var != nil {
if t := cfg.Var[name]; t != "" {
return t
}
}
if cfg.Func != nil {
if t := cfg.Func[name]; t != "" {
return "func()" + t
}
}
return ""
}
// Type describes the Fields and Methods of a type.
// If the field or method cannot be found there, it is next
// looked for in the Embed list.
type Type struct {
Field map[string]string // map field name to type
Method map[string]string // map method name to comma-separated return types (should start with "func ")
Embed []string // list of types this type embeds (for extra methods)
Def string // definition of named type
}
// dot returns the type of "typ.name", making its decision
// using the type information in cfg.
func (typ *Type) dot(cfg *TypeConfig, name string) string {
if typ.Field != nil {
if t := typ.Field[name]; t != "" {
return t
}
}
if typ.Method != nil {
if t := typ.Method[name]; t != "" {
return t
}
}
for _, e := range typ.Embed {
etyp := cfg.Type[e]
if etyp != nil {
if t := etyp.dot(cfg, name); t != "" {
return t
}
}
}
return ""
}
// typecheck type checks the AST f assuming the information in cfg.
// It returns two maps with type information:
// typeof maps AST nodes to type information in gofmt string form.
// assign maps type strings to lists of expressions that were assigned
// to values of another type that were assigned to that type.
func typecheck(cfg *TypeConfig, f *ast.File) (typeof map[interface{}]string, assign map[string][]interface{}) {
typeof = make(map[interface{}]string)
assign = make(map[string][]interface{})
cfg1 := &TypeConfig{}
*cfg1 = *cfg // make copy so we can add locally
copied := false
// gather function declarations
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok {
continue
}
typecheck1(cfg, fn.Type, typeof, assign)
t := typeof[fn.Type]
if fn.Recv != nil {
// The receiver must be a type.
rcvr := typeof[fn.Recv]
if !isType(rcvr) {
if len(fn.Recv.List) != 1 {
continue
}
rcvr = mkType(gofmt(fn.Recv.List[0].Type))
typeof[fn.Recv.List[0].Type] = rcvr
}
rcvr = getType(rcvr)
if rcvr != "" && rcvr[0] == '*' {
rcvr = rcvr[1:]
}
typeof[rcvr+"."+fn.Name.Name] = t
} else {
if isType(t) {
t = getType(t)
} else {
t = gofmt(fn.Type)
}
typeof[fn.Name] = t
// Record typeof[fn.Name.Obj] for future references to fn.Name.
typeof[fn.Name.Obj] = t
}
}
// gather struct declarations
for _, decl := range f.Decls {
d, ok := decl.(*ast.GenDecl)
if ok {
for _, s := range d.Specs {
switch s := s.(type) {
case *ast.TypeSpec:
if cfg1.Type[s.Name.Name] != nil {
break
}
if !copied {
copied = true
// Copy map lazily: it's time.
cfg1.Type = make(map[string]*Type)
for k, v := range cfg.Type {
cfg1.Type[k] = v
}
}
t := &Type{Field: map[string]string{}}
cfg1.Type[s.Name.Name] = t
switch st := s.Type.(type) {
case *ast.StructType:
for _, f := range st.Fields.List {
for _, n := range f.Names {
t.Field[n.Name] = gofmt(f.Type)
}
}
case *ast.ArrayType, *ast.StarExpr, *ast.MapType:
t.Def = gofmt(st)
}
}
}
}
}
typecheck1(cfg1, f, typeof, assign)
return typeof, assign
}
func makeExprList(a []*ast.Ident) []ast.Expr {
var b []ast.Expr
for _, x := range a {
b = append(b, x)
}
return b
}
// Typecheck1 is the recursive form of typecheck.
// It is like typecheck but adds to the information in typeof
// instead of allocating a new map.
func typecheck1(cfg *TypeConfig, f interface{}, typeof map[interface{}]string, assign map[string][]interface{}) {
// set sets the type of n to typ.
// If isDecl is true, n is being declared.
set := func(n ast.Expr, typ string, isDecl bool) {
if typeof[n] != "" || typ == "" {
if typeof[n] != typ {
assign[typ] = append(assign[typ], n)
}
return
}
typeof[n] = typ
// If we obtained typ from the declaration of x
// propagate the type to all the uses.
// The !isDecl case is a cheat here, but it makes
// up in some cases for not paying attention to
// struct fields. The real type checker will be
// more accurate so we won't need the cheat.
if id, ok := n.(*ast.Ident); ok && id.Obj != nil && (isDecl || typeof[id.Obj] == "") {
typeof[id.Obj] = typ
}
}
// Type-check an assignment lhs = rhs.
// If isDecl is true, this is := so we can update
// the types of the objects that lhs refers to.
typecheckAssign := func(lhs, rhs []ast.Expr, isDecl bool) {
if len(lhs) > 1 && len(rhs) == 1 {
if _, ok := rhs[0].(*ast.CallExpr); ok {
t := split(typeof[rhs[0]])
// Lists should have same length but may not; pair what can be paired.
for i := 0; i < len(lhs) && i < len(t); i++ {
set(lhs[i], t[i], isDecl)
}
return
}
}
if len(lhs) == 1 && len(rhs) == 2 {
// x = y, ok
rhs = rhs[:1]
} else if len(lhs) == 2 && len(rhs) == 1 {
// x, ok = y
lhs = lhs[:1]
}
// Match as much as we can.
for i := 0; i < len(lhs) && i < len(rhs); i++ {
x, y := lhs[i], rhs[i]
if typeof[y] != "" {
set(x, typeof[y], isDecl)
} else {
set(y, typeof[x], false)
}
}
}
expand := func(s string) string {
typ := cfg.Type[s]
if typ != nil && typ.Def != "" {
return typ.Def
}
return s
}
// The main type check is a recursive algorithm implemented
// by walkBeforeAfter(n, before, after).
// Most of it is bottom-up, but in a few places we need
// to know the type of the function we are checking.
// The before function records that information on
// the curfn stack.
var curfn []*ast.FuncType
before := func(n interface{}) {
// push function type on stack
switch n := n.(type) {
case *ast.FuncDecl:
curfn = append(curfn, n.Type)
case *ast.FuncLit:
curfn = append(curfn, n.Type)
}
}
// After is the real type checker.
after := func(n interface{}) {
if n == nil {
return
}
if false && reflect.TypeOf(n).Kind() == reflect.Ptr { // debugging trace
defer func() {
if t := typeof[n]; t != "" {
pos := fset.Position(n.(ast.Node).Pos())
fmt.Fprintf(os.Stderr, "%s: typeof[%s] = %s\n", pos, gofmt(n), t)
}
}()
}
switch n := n.(type) {
case *ast.FuncDecl, *ast.FuncLit:
// pop function type off stack
curfn = curfn[:len(curfn)-1]
case *ast.FuncType:
typeof[n] = mkType(joinFunc(split(typeof[n.Params]), split(typeof[n.Results])))
case *ast.FieldList:
// Field list is concatenation of sub-lists.
t := ""
for _, field := range n.List {
if t != "" {
t += ", "
}
t += typeof[field]
}
typeof[n] = t
case *ast.Field:
// Field is one instance of the type per name.
all := ""
t := typeof[n.Type]
if !isType(t) {
// Create a type, because it is typically *T or *p.T
// and we might care about that type.
t = mkType(gofmt(n.Type))
typeof[n.Type] = t
}
t = getType(t)
if len(n.Names) == 0 {
all = t
} else {
for _, id := range n.Names {
if all != "" {
all += ", "
}
all += t
typeof[id.Obj] = t
typeof[id] = t
}
}
typeof[n] = all
case *ast.ValueSpec:
// var declaration. Use type if present.
if n.Type != nil {
t := typeof[n.Type]
if !isType(t) {
t = mkType(gofmt(n.Type))
typeof[n.Type] = t
}
t = getType(t)
for _, id := range n.Names {
set(id, t, true)
}
}
// Now treat same as assignment.
typecheckAssign(makeExprList(n.Names), n.Values, true)
case *ast.AssignStmt:
typecheckAssign(n.Lhs, n.Rhs, n.Tok == token.DEFINE)
case *ast.Ident:
// Identifier can take its type from underlying object.
if t := typeof[n.Obj]; t != "" {
typeof[n] = t
}
case *ast.SelectorExpr:
// Field or method.
name := n.Sel.Name
if t := typeof[n.X]; t != "" {
if strings.HasPrefix(t, "*") {
t = t[1:] // implicit *
}
if typ := cfg.Type[t]; typ != nil {
if t := typ.dot(cfg, name); t != "" {
typeof[n] = t
return
}
}
tt := typeof[t+"."+name]
if isType(tt) {
typeof[n] = getType(tt)
return
}
}
// Package selector.
if x, ok := n.X.(*ast.Ident); ok && x.Obj == nil {
str := x.Name + "." + name
if cfg.Type[str] != nil {
typeof[n] = mkType(str)
return
}
if t := cfg.typeof(x.Name + "." + name); t != "" {
typeof[n] = t
return
}
}
case *ast.CallExpr:
// make(T) has type T.
if isTopName(n.Fun, "make") && len(n.Args) >= 1 {
typeof[n] = gofmt(n.Args[0])
return
}
// new(T) has type *T
if isTopName(n.Fun, "new") && len(n.Args) == 1 {
typeof[n] = "*" + gofmt(n.Args[0])
return
}
// Otherwise, use type of function to determine arguments.
t := typeof[n.Fun]
in, out := splitFunc(t)
if in == nil && out == nil {
return
}
typeof[n] = join(out)
for i, arg := range n.Args {
if i >= len(in) {
break
}
if typeof[arg] == "" {
typeof[arg] = in[i]
}
}
case *ast.TypeAssertExpr:
// x.(type) has type of x.
if n.Type == nil {
typeof[n] = typeof[n.X]
return
}
// x.(T) has type T.
if t := typeof[n.Type]; isType(t) {
typeof[n] = getType(t)
} else {
typeof[n] = gofmt(n.Type)
}
case *ast.SliceExpr:
// x[i:j] has type of x.
typeof[n] = typeof[n.X]
case *ast.IndexExpr:
// x[i] has key type of x's type.
t := expand(typeof[n.X])
if strings.HasPrefix(t, "[") || strings.HasPrefix(t, "map[") {
// Lazy: assume there are no nested [] in the array
// length or map key type.
if i := strings.Index(t, "]"); i >= 0 {
typeof[n] = t[i+1:]
}
}
case *ast.StarExpr:
// *x for x of type *T has type T when x is an expr.
// We don't use the result when *x is a type, but
// compute it anyway.
t := expand(typeof[n.X])
if isType(t) {
typeof[n] = "type *" + getType(t)
} else if strings.HasPrefix(t, "*") {
typeof[n] = t[len("*"):]
}
case *ast.UnaryExpr:
// &x for x of type T has type *T.
t := typeof[n.X]
if t != "" && n.Op == token.AND {
typeof[n] = "*" + t
}
case *ast.CompositeLit:
// T{...} has type T.
typeof[n] = gofmt(n.Type)
case *ast.ParenExpr:
// (x) has type of x.
typeof[n] = typeof[n.X]
case *ast.RangeStmt:
t := expand(typeof[n.X])
if t == "" {
return
}
var key, value string
if t == "string" {
key, value = "int", "rune"
} else if strings.HasPrefix(t, "[") {
key = "int"
if i := strings.Index(t, "]"); i >= 0 {
value = t[i+1:]
}
} else if strings.HasPrefix(t, "map[") {
if i := strings.Index(t, "]"); i >= 0 {
key, value = t[4:i], t[i+1:]
}
}
changed := false
if n.Key != nil && key != "" {
changed = true
set(n.Key, key, n.Tok == token.DEFINE)
}
if n.Value != nil && value != "" {
changed = true
set(n.Value, value, n.Tok == token.DEFINE)
}
// Ugly failure of vision: already type-checked body.
// Do it again now that we have that type info.
if changed {
typecheck1(cfg, n.Body, typeof, assign)
}
case *ast.TypeSwitchStmt:
// Type of variable changes for each case in type switch,
// but go/parser generates just one variable.
// Repeat type check for each case with more precise
// type information.
as, ok := n.Assign.(*ast.AssignStmt)
if !ok {
return
}
varx, ok := as.Lhs[0].(*ast.Ident)
if !ok {
return
}
t := typeof[varx]
for _, cas := range n.Body.List {
cas := cas.(*ast.CaseClause)
if len(cas.List) == 1 {
// Variable has specific type only when there is
// exactly one type in the case list.
if tt := typeof[cas.List[0]]; isType(tt) {
tt = getType(tt)
typeof[varx] = tt
typeof[varx.Obj] = tt
typecheck1(cfg, cas.Body, typeof, assign)
}
}
}
// Restore t.
typeof[varx] = t
typeof[varx.Obj] = t
case *ast.ReturnStmt:
if len(curfn) == 0 {
// Probably can't happen.
return
}
f := curfn[len(curfn)-1]
res := n.Results
if f.Results != nil {
t := split(typeof[f.Results])
for i := 0; i < len(res) && i < len(t); i++ {
set(res[i], t[i], false)
}
}
}
}
walkBeforeAfter(f, before, after)
}
// Convert between function type strings and lists of types.
// Using strings makes this a little harder, but it makes
// a lot of the rest of the code easier. This will all go away
// when we can use go/typechecker directly.
// splitFunc splits "func(x,y,z) (a,b,c)" into ["x", "y", "z"] and ["a", "b", "c"].
func splitFunc(s string) (in, out []string) {
if !strings.HasPrefix(s, "func(") {
return nil, nil
}
i := len("func(") // index of beginning of 'in' arguments
nparen := 0
for j := i; j < len(s); j++ {
switch s[j] {
case '(':
nparen++
case ')':
nparen--
if nparen < 0 {
// found end of parameter list
out := strings.TrimSpace(s[j+1:])
if len(out) >= 2 && out[0] == '(' && out[len(out)-1] == ')' {
out = out[1 : len(out)-1]
}
return split(s[i:j]), split(out)
}
}
}
return nil, nil
}
// joinFunc is the inverse of splitFunc.
func joinFunc(in, out []string) string {
outs := ""
if len(out) == 1 {
outs = " " + out[0]
} else if len(out) > 1 {
outs = " (" + join(out) + ")"
}
return "func(" + join(in) + ")" + outs
}
// split splits "int, float" into ["int", "float"] and splits "" into [].
func split(s string) []string {
out := []string{}
i := 0 // current type being scanned is s[i:j].
nparen := 0
for j := 0; j < len(s); j++ {
switch s[j] {
case ' ':
if i == j {
i++
}
case '(':
nparen++
case ')':
nparen--
if nparen < 0 {
// probably can't happen
return nil
}
case ',':
if nparen == 0 {
if i < j {
out = append(out, s[i:j])
}
i = j + 1
}
}
}
if nparen != 0 {
// probably can't happen
return nil
}
if i < len(s) {
out = append(out, s[i:])
}
return out
}
// join is the inverse of split.
func join(x []string) string {
return strings.Join(x, ", ")
}
appengine-1.6.7/datastore/ 0000775 0000000 0000000 00000000000 13741174344 0015471 5 ustar 00root root 0000000 0000000 appengine-1.6.7/datastore/datastore.go 0000664 0000000 0000000 00000027163 13741174344 0020017 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"errors"
"fmt"
"reflect"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/internal"
pb "google.golang.org/appengine/internal/datastore"
)
var (
// ErrInvalidEntityType is returned when functions like Get or Next are
// passed a dst or src argument of invalid type.
ErrInvalidEntityType = errors.New("datastore: invalid entity type")
// ErrInvalidKey is returned when an invalid key is presented.
ErrInvalidKey = errors.New("datastore: invalid key")
// ErrNoSuchEntity is returned when no entity was found for a given key.
ErrNoSuchEntity = errors.New("datastore: no such entity")
)
// ErrFieldMismatch is returned when a field is to be loaded into a different
// type than the one it was stored from, or when a field is missing or
// unexported in the destination struct.
// StructType is the type of the struct pointed to by the destination argument
// passed to Get or to Iterator.Next.
type ErrFieldMismatch struct {
StructType reflect.Type
FieldName string
Reason string
}
func (e *ErrFieldMismatch) Error() string {
return fmt.Sprintf("datastore: cannot load field %q into a %q: %s",
e.FieldName, e.StructType, e.Reason)
}
// protoToKey converts a Reference proto to a *Key. If the key is invalid,
// protoToKey will return the invalid key along with ErrInvalidKey.
func protoToKey(r *pb.Reference) (k *Key, err error) {
appID := r.GetApp()
namespace := r.GetNameSpace()
for _, e := range r.Path.Element {
k = &Key{
kind: e.GetType(),
stringID: e.GetName(),
intID: e.GetId(),
parent: k,
appID: appID,
namespace: namespace,
}
if !k.valid() {
return k, ErrInvalidKey
}
}
return
}
// keyToProto converts a *Key to a Reference proto.
func keyToProto(defaultAppID string, k *Key) *pb.Reference {
appID := k.appID
if appID == "" {
appID = defaultAppID
}
n := 0
for i := k; i != nil; i = i.parent {
n++
}
e := make([]*pb.Path_Element, n)
for i := k; i != nil; i = i.parent {
n--
e[n] = &pb.Path_Element{
Type: &i.kind,
}
// At most one of {Name,Id} should be set.
// Neither will be set for incomplete keys.
if i.stringID != "" {
e[n].Name = &i.stringID
} else if i.intID != 0 {
e[n].Id = &i.intID
}
}
var namespace *string
if k.namespace != "" {
namespace = proto.String(k.namespace)
}
return &pb.Reference{
App: proto.String(appID),
NameSpace: namespace,
Path: &pb.Path{
Element: e,
},
}
}
// multiKeyToProto is a batch version of keyToProto.
func multiKeyToProto(appID string, key []*Key) []*pb.Reference {
ret := make([]*pb.Reference, len(key))
for i, k := range key {
ret[i] = keyToProto(appID, k)
}
return ret
}
// multiValid is a batch version of Key.valid. It returns an error, not a
// []bool.
func multiValid(key []*Key) error {
invalid := false
for _, k := range key {
if !k.valid() {
invalid = true
break
}
}
if !invalid {
return nil
}
err := make(appengine.MultiError, len(key))
for i, k := range key {
if !k.valid() {
err[i] = ErrInvalidKey
}
}
return err
}
// It's unfortunate that the two semantically equivalent concepts pb.Reference
// and pb.PropertyValue_ReferenceValue aren't the same type. For example, the
// two have different protobuf field numbers.
// referenceValueToKey is the same as protoToKey except the input is a
// PropertyValue_ReferenceValue instead of a Reference.
func referenceValueToKey(r *pb.PropertyValue_ReferenceValue) (k *Key, err error) {
appID := r.GetApp()
namespace := r.GetNameSpace()
for _, e := range r.Pathelement {
k = &Key{
kind: e.GetType(),
stringID: e.GetName(),
intID: e.GetId(),
parent: k,
appID: appID,
namespace: namespace,
}
if !k.valid() {
return nil, ErrInvalidKey
}
}
return
}
// keyToReferenceValue is the same as keyToProto except the output is a
// PropertyValue_ReferenceValue instead of a Reference.
func keyToReferenceValue(defaultAppID string, k *Key) *pb.PropertyValue_ReferenceValue {
ref := keyToProto(defaultAppID, k)
pe := make([]*pb.PropertyValue_ReferenceValue_PathElement, len(ref.Path.Element))
for i, e := range ref.Path.Element {
pe[i] = &pb.PropertyValue_ReferenceValue_PathElement{
Type: e.Type,
Id: e.Id,
Name: e.Name,
}
}
return &pb.PropertyValue_ReferenceValue{
App: ref.App,
NameSpace: ref.NameSpace,
Pathelement: pe,
}
}
type multiArgType int
const (
multiArgTypeInvalid multiArgType = iota
multiArgTypePropertyLoadSaver
multiArgTypeStruct
multiArgTypeStructPtr
multiArgTypeInterface
)
// checkMultiArg checks that v has type []S, []*S, []I, or []P, for some struct
// type S, for some interface type I, or some non-interface non-pointer type P
// such that P or *P implements PropertyLoadSaver.
//
// It returns what category the slice's elements are, and the reflect.Type
// that represents S, I or P.
//
// As a special case, PropertyList is an invalid type for v.
func checkMultiArg(v reflect.Value) (m multiArgType, elemType reflect.Type) {
if v.Kind() != reflect.Slice {
return multiArgTypeInvalid, nil
}
if v.Type() == typeOfPropertyList {
return multiArgTypeInvalid, nil
}
elemType = v.Type().Elem()
if reflect.PtrTo(elemType).Implements(typeOfPropertyLoadSaver) {
return multiArgTypePropertyLoadSaver, elemType
}
switch elemType.Kind() {
case reflect.Struct:
return multiArgTypeStruct, elemType
case reflect.Interface:
return multiArgTypeInterface, elemType
case reflect.Ptr:
elemType = elemType.Elem()
if elemType.Kind() == reflect.Struct {
return multiArgTypeStructPtr, elemType
}
}
return multiArgTypeInvalid, nil
}
// Get loads the entity stored for k into dst, which must be a struct pointer
// or implement PropertyLoadSaver. If there is no such entity for the key, Get
// returns ErrNoSuchEntity.
//
// The values of dst's unmatched struct fields are not modified, and matching
// slice-typed fields are not reset before appending to them. In particular, it
// is recommended to pass a pointer to a zero valued struct on each Get call.
//
// ErrFieldMismatch is returned when a field is to be loaded into a different
// type than the one it was stored from, or when a field is missing or
// unexported in the destination struct. ErrFieldMismatch is only returned if
// dst is a struct pointer.
func Get(c context.Context, key *Key, dst interface{}) error {
if dst == nil { // GetMulti catches nil interface; we need to catch nil ptr here
return ErrInvalidEntityType
}
err := GetMulti(c, []*Key{key}, []interface{}{dst})
if me, ok := err.(appengine.MultiError); ok {
return me[0]
}
return err
}
// GetMulti is a batch version of Get.
//
// dst must be a []S, []*S, []I or []P, for some struct type S, some interface
// type I, or some non-interface non-pointer type P such that P or *P
// implements PropertyLoadSaver. If an []I, each element must be a valid dst
// for Get: it must be a struct pointer or implement PropertyLoadSaver.
//
// As a special case, PropertyList is an invalid type for dst, even though a
// PropertyList is a slice of structs. It is treated as invalid to avoid being
// mistakenly passed when []PropertyList was intended.
func GetMulti(c context.Context, key []*Key, dst interface{}) error {
v := reflect.ValueOf(dst)
multiArgType, _ := checkMultiArg(v)
if multiArgType == multiArgTypeInvalid {
return errors.New("datastore: dst has invalid type")
}
if len(key) != v.Len() {
return errors.New("datastore: key and dst slices have different length")
}
if len(key) == 0 {
return nil
}
if err := multiValid(key); err != nil {
return err
}
req := &pb.GetRequest{
Key: multiKeyToProto(internal.FullyQualifiedAppID(c), key),
}
res := &pb.GetResponse{}
if err := internal.Call(c, "datastore_v3", "Get", req, res); err != nil {
return err
}
if len(key) != len(res.Entity) {
return errors.New("datastore: internal error: server returned the wrong number of entities")
}
multiErr, any := make(appengine.MultiError, len(key)), false
for i, e := range res.Entity {
if e.Entity == nil {
multiErr[i] = ErrNoSuchEntity
} else {
elem := v.Index(i)
if multiArgType == multiArgTypePropertyLoadSaver || multiArgType == multiArgTypeStruct {
elem = elem.Addr()
}
if multiArgType == multiArgTypeStructPtr && elem.IsNil() {
elem.Set(reflect.New(elem.Type().Elem()))
}
multiErr[i] = loadEntity(elem.Interface(), e.Entity)
}
if multiErr[i] != nil {
any = true
}
}
if any {
return multiErr
}
return nil
}
// Put saves the entity src into the datastore with key k. src must be a struct
// pointer or implement PropertyLoadSaver; if a struct pointer then any
// unexported fields of that struct will be skipped. If k is an incomplete key,
// the returned key will be a unique key generated by the datastore.
func Put(c context.Context, key *Key, src interface{}) (*Key, error) {
k, err := PutMulti(c, []*Key{key}, []interface{}{src})
if err != nil {
if me, ok := err.(appengine.MultiError); ok {
return nil, me[0]
}
return nil, err
}
return k[0], nil
}
// PutMulti is a batch version of Put.
//
// src must satisfy the same conditions as the dst argument to GetMulti.
func PutMulti(c context.Context, key []*Key, src interface{}) ([]*Key, error) {
v := reflect.ValueOf(src)
multiArgType, _ := checkMultiArg(v)
if multiArgType == multiArgTypeInvalid {
return nil, errors.New("datastore: src has invalid type")
}
if len(key) != v.Len() {
return nil, errors.New("datastore: key and src slices have different length")
}
if len(key) == 0 {
return nil, nil
}
appID := internal.FullyQualifiedAppID(c)
if err := multiValid(key); err != nil {
return nil, err
}
req := &pb.PutRequest{}
for i := range key {
elem := v.Index(i)
if multiArgType == multiArgTypePropertyLoadSaver || multiArgType == multiArgTypeStruct {
elem = elem.Addr()
}
sProto, err := saveEntity(appID, key[i], elem.Interface())
if err != nil {
return nil, err
}
req.Entity = append(req.Entity, sProto)
}
res := &pb.PutResponse{}
if err := internal.Call(c, "datastore_v3", "Put", req, res); err != nil {
return nil, err
}
if len(key) != len(res.Key) {
return nil, errors.New("datastore: internal error: server returned the wrong number of keys")
}
ret := make([]*Key, len(key))
for i := range ret {
var err error
ret[i], err = protoToKey(res.Key[i])
if err != nil || ret[i].Incomplete() {
return nil, errors.New("datastore: internal error: server returned an invalid key")
}
}
return ret, nil
}
// Delete deletes the entity for the given key.
func Delete(c context.Context, key *Key) error {
err := DeleteMulti(c, []*Key{key})
if me, ok := err.(appengine.MultiError); ok {
return me[0]
}
return err
}
// DeleteMulti is a batch version of Delete.
func DeleteMulti(c context.Context, key []*Key) error {
if len(key) == 0 {
return nil
}
if err := multiValid(key); err != nil {
return err
}
req := &pb.DeleteRequest{
Key: multiKeyToProto(internal.FullyQualifiedAppID(c), key),
}
res := &pb.DeleteResponse{}
return internal.Call(c, "datastore_v3", "Delete", req, res)
}
func namespaceMod(m proto.Message, namespace string) {
// pb.Query is the only type that has a name_space field.
// All other namespace support in datastore is in the keys.
switch m := m.(type) {
case *pb.Query:
if m.NameSpace == nil {
m.NameSpace = &namespace
}
}
}
func init() {
internal.NamespaceMods["datastore_v3"] = namespaceMod
internal.RegisterErrorCodeMap("datastore_v3", pb.Error_ErrorCode_name)
internal.RegisterTimeoutErrorCode("datastore_v3", int32(pb.Error_TIMEOUT))
}
appengine-1.6.7/datastore/datastore_test.go 0000664 0000000 0000000 00000104260 13741174344 0021050 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"encoding/json"
"errors"
"fmt"
"os"
"reflect"
"sort"
"strings"
"testing"
"time"
"google.golang.org/appengine"
"google.golang.org/appengine/internal/aetesting"
pb "google.golang.org/appengine/internal/datastore"
)
const testAppID = "testApp"
type (
myBlob []byte
myByte byte
myString string
)
func makeMyByteSlice(n int) []myByte {
b := make([]myByte, n)
for i := range b {
b[i] = myByte(i)
}
return b
}
func makeInt8Slice(n int) []int8 {
b := make([]int8, n)
for i := range b {
b[i] = int8(i)
}
return b
}
func makeUint8Slice(n int) []uint8 {
b := make([]uint8, n)
for i := range b {
b[i] = uint8(i)
}
return b
}
func newKey(stringID string, parent *Key) *Key {
return &Key{
kind: "kind",
stringID: stringID,
intID: 0,
parent: parent,
appID: testAppID,
}
}
var (
testKey0 = newKey("name0", nil)
testKey1a = newKey("name1", nil)
testKey1b = newKey("name1", nil)
testKey2a = newKey("name2", testKey0)
testKey2b = newKey("name2", testKey0)
testGeoPt0 = appengine.GeoPoint{Lat: 1.2, Lng: 3.4}
testGeoPt1 = appengine.GeoPoint{Lat: 5, Lng: 10}
testBadGeoPt = appengine.GeoPoint{Lat: 1000, Lng: 34}
now = time.Unix(1e9, 0).UTC()
)
type B0 struct {
B []byte
}
type B1 struct {
B []int8
}
type B2 struct {
B myBlob
}
type B3 struct {
B []myByte
}
type B4 struct {
B [][]byte
}
type B5 struct {
B ByteString
}
type C0 struct {
I int
C chan int
}
type C1 struct {
I int
C *chan int
}
type C2 struct {
I int
C []chan int
}
type C3 struct {
C string
}
type E struct{}
type G0 struct {
G appengine.GeoPoint
}
type G1 struct {
G []appengine.GeoPoint
}
type K0 struct {
K *Key
}
type K1 struct {
K []*Key
}
type S struct {
St string
}
type NoOmit struct {
A string
B int `datastore:"Bb"`
C bool `datastore:",noindex"`
}
type OmitAll struct {
A string `datastore:",omitempty"`
B int `datastore:"Bb,omitempty"`
C bool `datastore:",omitempty,noindex"`
D time.Time `datastore:",omitempty"`
F []int `datastore:",omitempty"`
}
type Omit struct {
A string `datastore:",omitempty"`
B int `datastore:"Bb,omitempty"`
C bool `datastore:",omitempty,noindex"`
D time.Time `datastore:",omitempty"`
F []int `datastore:",omitempty"`
S `datastore:",omitempty"`
}
type NoOmits struct {
No []NoOmit `datastore:",omitempty"`
S `datastore:",omitempty"`
Ss S `datastore:",omitempty"`
}
type N0 struct {
X0
Nonymous X0
Ignore string `datastore:"-"`
Other string
}
type N1 struct {
X0
Nonymous []X0
Ignore string `datastore:"-"`
Other string
}
type N2 struct {
N1 `datastore:"red"`
Green N1 `datastore:"green"`
Blue N1
White N1 `datastore:"-"`
}
type O0 struct {
I int64
}
type O1 struct {
I int32
}
type U0 struct {
U uint
}
type U1 struct {
U string
}
type T struct {
T time.Time
}
type X0 struct {
S string
I int
i int
}
type X1 struct {
S myString
I int32
J int64
}
type X2 struct {
Z string
i int
}
type X3 struct {
S bool
I int
}
type Y0 struct {
B bool
F []float64
G []float64
}
type Y1 struct {
B bool
F float64
}
type Y2 struct {
B bool
F []int64
}
type Tagged struct {
A int `datastore:"a,noindex"`
B []int `datastore:"b"`
C int `datastore:",noindex"`
D int `datastore:""`
E int
// The "flatten" option is parsed but ignored for now.
F int `datastore:",noindex,flatten"`
G int `datastore:",flatten"`
I int `datastore:"-"`
J int `datastore:",noindex" json:"j"`
Y0 `datastore:"-"`
Z chan int `datastore:"-,"`
}
type InvalidTagged1 struct {
I int `datastore:"\t"`
}
type InvalidTagged2 struct {
I int
J int `datastore:"I"`
}
type Inner1 struct {
W int32
X string
}
type Inner2 struct {
Y float64
}
type Inner3 struct {
Z bool
}
type Outer struct {
A int16
I []Inner1
J Inner2
Inner3
}
type OuterEquivalent struct {
A int16
IDotW []int32 `datastore:"I.W"`
IDotX []string `datastore:"I.X"`
JDotY float64 `datastore:"J.Y"`
Z bool
}
type Dotted struct {
A DottedA `datastore:"A0.A1.A2"`
}
type DottedA struct {
B DottedB `datastore:"B3"`
}
type DottedB struct {
C int `datastore:"C4.C5"`
}
type SliceOfSlices struct {
I int
S []struct {
J int
F []float64
}
}
type Recursive struct {
I int
R []Recursive
}
type MutuallyRecursive0 struct {
I int
R []MutuallyRecursive1
}
type MutuallyRecursive1 struct {
I int
R []MutuallyRecursive0
}
type Doubler struct {
S string
I int64
B bool
}
type Repeat struct {
Key string
Value []byte
}
type Repeated struct {
Repeats []Repeat
}
func (d *Doubler) Load(props []Property) error {
return LoadStruct(d, props)
}
type EmbeddedTime struct {
time.Time
}
type SpecialTime struct {
MyTime EmbeddedTime
}
func (d *Doubler) Save() ([]Property, error) {
// Save the default Property slice to an in-memory buffer (a PropertyList).
props, err := SaveStruct(d)
if err != nil {
return nil, err
}
var list PropertyList
if err := list.Load(props); err != nil {
return nil, err
}
// Edit that PropertyList, and send it on.
for i := range list {
switch v := list[i].Value.(type) {
case string:
// + means string concatenation.
list[i].Value = v + v
case int64:
// + means integer addition.
list[i].Value = v + v
}
}
return list.Save()
}
var _ PropertyLoadSaver = (*Doubler)(nil)
type Deriver struct {
S, Derived, Ignored string
}
func (e *Deriver) Load(props []Property) error {
for _, p := range props {
if p.Name != "S" {
continue
}
e.S = p.Value.(string)
e.Derived = "derived+" + e.S
}
return nil
}
func (e *Deriver) Save() ([]Property, error) {
return []Property{
{
Name: "S",
Value: e.S,
},
}, nil
}
var _ PropertyLoadSaver = (*Deriver)(nil)
type BadMultiPropEntity struct{}
func (e *BadMultiPropEntity) Load(props []Property) error {
return errors.New("unimplemented")
}
func (e *BadMultiPropEntity) Save() ([]Property, error) {
// Write multiple properties with the same name "I", but Multiple is false.
var props []Property
for i := 0; i < 3; i++ {
props = append(props, Property{
Name: "I",
Value: int64(i),
})
}
return props, nil
}
var _ PropertyLoadSaver = (*BadMultiPropEntity)(nil)
type BK struct {
Key appengine.BlobKey
}
type testCase struct {
desc string
src interface{}
want interface{}
putErr string
getErr string
}
var testCases = []testCase{
{
"chan save fails",
&C0{I: -1},
&E{},
"unsupported struct field",
"",
},
{
"*chan save fails",
&C1{I: -1},
&E{},
"unsupported struct field",
"",
},
{
"[]chan save fails",
&C2{I: -1, C: make([]chan int, 8)},
&E{},
"unsupported struct field",
"",
},
{
"chan load fails",
&C3{C: "not a chan"},
&C0{},
"",
"type mismatch",
},
{
"*chan load fails",
&C3{C: "not a *chan"},
&C1{},
"",
"type mismatch",
},
{
"[]chan load fails",
&C3{C: "not a []chan"},
&C2{},
"",
"type mismatch",
},
{
"empty struct",
&E{},
&E{},
"",
"",
},
{
"geopoint",
&G0{G: testGeoPt0},
&G0{G: testGeoPt0},
"",
"",
},
{
"geopoint invalid",
&G0{G: testBadGeoPt},
&G0{},
"invalid GeoPoint value",
"",
},
{
"geopoint as props",
&G0{G: testGeoPt0},
&PropertyList{
Property{Name: "G", Value: testGeoPt0, NoIndex: false, Multiple: false},
},
"",
"",
},
{
"geopoint slice",
&G1{G: []appengine.GeoPoint{testGeoPt0, testGeoPt1}},
&G1{G: []appengine.GeoPoint{testGeoPt0, testGeoPt1}},
"",
"",
},
{
"omit empty, all",
&OmitAll{},
new(PropertyList),
"",
"",
},
{
"omit empty",
&Omit{},
&PropertyList{
Property{Name: "St", Value: "", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"omit empty, fields populated",
&Omit{
A: "a",
B: 10,
C: true,
D: now,
F: []int{11},
},
&PropertyList{
Property{Name: "A", Value: "a", NoIndex: false, Multiple: false},
Property{Name: "Bb", Value: int64(10), NoIndex: false, Multiple: false},
Property{Name: "C", Value: true, NoIndex: true, Multiple: false},
Property{Name: "D", Value: now, NoIndex: false, Multiple: false},
Property{Name: "F", Value: int64(11), NoIndex: false, Multiple: true},
Property{Name: "St", Value: "", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"omit empty, fields populated",
&Omit{
A: "a",
B: 10,
C: true,
D: now,
F: []int{11},
S: S{St: "string"},
},
&PropertyList{
Property{Name: "A", Value: "a", NoIndex: false, Multiple: false},
Property{Name: "Bb", Value: int64(10), NoIndex: false, Multiple: false},
Property{Name: "C", Value: true, NoIndex: true, Multiple: false},
Property{Name: "D", Value: now, NoIndex: false, Multiple: false},
Property{Name: "F", Value: int64(11), NoIndex: false, Multiple: true},
Property{Name: "St", Value: "string", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"omit empty does not propagate",
&NoOmits{
No: []NoOmit{
NoOmit{},
},
S: S{},
Ss: S{},
},
&PropertyList{
Property{Name: "No.A", Value: "", NoIndex: false, Multiple: true},
Property{Name: "No.Bb", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "No.C", Value: false, NoIndex: true, Multiple: true},
Property{Name: "Ss.St", Value: "", NoIndex: false, Multiple: false},
Property{Name: "St", Value: "", NoIndex: false, Multiple: false}},
"",
"",
},
{
"key",
&K0{K: testKey1a},
&K0{K: testKey1b},
"",
"",
},
{
"key with parent",
&K0{K: testKey2a},
&K0{K: testKey2b},
"",
"",
},
{
"nil key",
&K0{},
&K0{},
"",
"",
},
{
"all nil keys in slice",
&K1{[]*Key{nil, nil}},
&K1{[]*Key{nil, nil}},
"",
"",
},
{
"some nil keys in slice",
&K1{[]*Key{testKey1a, nil, testKey2a}},
&K1{[]*Key{testKey1b, nil, testKey2b}},
"",
"",
},
{
"overflow",
&O0{I: 1 << 48},
&O1{},
"",
"overflow",
},
{
"time",
&T{T: time.Unix(1e9, 0)},
&T{T: time.Unix(1e9, 0)},
"",
"",
},
{
"time as props",
&T{T: time.Unix(1e9, 0)},
&PropertyList{
Property{Name: "T", Value: time.Unix(1e9, 0).UTC(), NoIndex: false, Multiple: false},
},
"",
"",
},
{
"uint save",
&U0{U: 1},
&U0{},
"unsupported struct field",
"",
},
{
"uint load",
&U1{U: "not a uint"},
&U0{},
"",
"type mismatch",
},
{
"zero",
&X0{},
&X0{},
"",
"",
},
{
"basic",
&X0{S: "one", I: 2, i: 3},
&X0{S: "one", I: 2},
"",
"",
},
{
"save string/int load myString/int32",
&X0{S: "one", I: 2, i: 3},
&X1{S: "one", I: 2},
"",
"",
},
{
"missing fields",
&X0{S: "one", I: 2, i: 3},
&X2{},
"",
"no such struct field",
},
{
"save string load bool",
&X0{S: "one", I: 2, i: 3},
&X3{I: 2},
"",
"type mismatch",
},
{
"basic slice",
&Y0{B: true, F: []float64{7, 8, 9}},
&Y0{B: true, F: []float64{7, 8, 9}},
"",
"",
},
{
"save []float64 load float64",
&Y0{B: true, F: []float64{7, 8, 9}},
&Y1{B: true},
"",
"requires a slice",
},
{
"save []float64 load []int64",
&Y0{B: true, F: []float64{7, 8, 9}},
&Y2{B: true},
"",
"type mismatch",
},
{
"single slice is too long",
&Y0{F: make([]float64, maxIndexedProperties+1)},
&Y0{},
"too many indexed properties",
"",
},
{
"two slices are too long",
&Y0{F: make([]float64, maxIndexedProperties), G: make([]float64, maxIndexedProperties)},
&Y0{},
"too many indexed properties",
"",
},
{
"one slice and one scalar are too long",
&Y0{F: make([]float64, maxIndexedProperties), B: true},
&Y0{},
"too many indexed properties",
"",
},
{
"slice of slices of bytes",
&Repeated{
Repeats: []Repeat{
{
Key: "key 1",
Value: []byte("value 1"),
},
{
Key: "key 2",
Value: []byte("value 2"),
},
},
},
&Repeated{
Repeats: []Repeat{
{
Key: "key 1",
Value: []byte("value 1"),
},
{
Key: "key 2",
Value: []byte("value 2"),
},
},
},
"",
"",
},
{
"long blob",
&B0{B: makeUint8Slice(maxIndexedProperties + 1)},
&B0{B: makeUint8Slice(maxIndexedProperties + 1)},
"",
"",
},
{
"long []int8 is too long",
&B1{B: makeInt8Slice(maxIndexedProperties + 1)},
&B1{},
"too many indexed properties",
"",
},
{
"short []int8",
&B1{B: makeInt8Slice(3)},
&B1{B: makeInt8Slice(3)},
"",
"",
},
{
"long myBlob",
&B2{B: makeUint8Slice(maxIndexedProperties + 1)},
&B2{B: makeUint8Slice(maxIndexedProperties + 1)},
"",
"",
},
{
"short myBlob",
&B2{B: makeUint8Slice(3)},
&B2{B: makeUint8Slice(3)},
"",
"",
},
{
"long []myByte",
&B3{B: makeMyByteSlice(maxIndexedProperties + 1)},
&B3{B: makeMyByteSlice(maxIndexedProperties + 1)},
"",
"",
},
{
"short []myByte",
&B3{B: makeMyByteSlice(3)},
&B3{B: makeMyByteSlice(3)},
"",
"",
},
{
"slice of blobs",
&B4{B: [][]byte{
makeUint8Slice(3),
makeUint8Slice(4),
makeUint8Slice(5),
}},
&B4{B: [][]byte{
makeUint8Slice(3),
makeUint8Slice(4),
makeUint8Slice(5),
}},
"",
"",
},
{
"short ByteString",
&B5{B: ByteString(makeUint8Slice(3))},
&B5{B: ByteString(makeUint8Slice(3))},
"",
"",
},
{
"short ByteString as props",
&B5{B: ByteString(makeUint8Slice(3))},
&PropertyList{
Property{Name: "B", Value: ByteString(makeUint8Slice(3)), NoIndex: false, Multiple: false},
},
"",
"",
},
{
"short ByteString into string",
&B5{B: ByteString("legacy")},
&struct{ B string }{"legacy"},
"",
"",
},
{
"[]byte must be noindex",
&PropertyList{
Property{Name: "B", Value: makeUint8Slice(3), NoIndex: false},
},
nil,
"cannot index a []byte valued Property",
"",
},
{
"save tagged load props",
&Tagged{A: 1, B: []int{21, 22, 23}, C: 3, D: 4, E: 5, F: 6, G: 7, I: 8, J: 9},
&PropertyList{
// A and B are renamed to a and b; A and C are noindex, I is ignored.
// Indexed properties are loaded before raw properties. Thus, the
// result is: b, b, b, D, E, a, c.
Property{Name: "C", Value: int64(3), NoIndex: true, Multiple: false},
Property{Name: "D", Value: int64(4), NoIndex: false, Multiple: false},
Property{Name: "E", Value: int64(5), NoIndex: false, Multiple: false},
Property{Name: "F", Value: int64(6), NoIndex: true, Multiple: false},
Property{Name: "G", Value: int64(7), NoIndex: false, Multiple: false},
Property{Name: "J", Value: int64(9), NoIndex: true, Multiple: false},
Property{Name: "a", Value: int64(1), NoIndex: true, Multiple: false},
Property{Name: "b", Value: int64(21), NoIndex: false, Multiple: true},
Property{Name: "b", Value: int64(22), NoIndex: false, Multiple: true},
Property{Name: "b", Value: int64(23), NoIndex: false, Multiple: true},
},
"",
"",
},
{
"save tagged load tagged",
&Tagged{A: 1, B: []int{21, 22, 23}, C: 3, D: 4, E: 5, I: 6, J: 7},
&Tagged{A: 1, B: []int{21, 22, 23}, C: 3, D: 4, E: 5, J: 7},
"",
"",
},
{
"save props load tagged",
&PropertyList{
Property{Name: "A", Value: int64(11), NoIndex: true, Multiple: false},
Property{Name: "a", Value: int64(12), NoIndex: true, Multiple: false},
},
&Tagged{A: 12},
"",
`cannot load field "A"`,
},
{
"invalid tagged1",
&InvalidTagged1{I: 1},
&InvalidTagged1{},
"struct tag has invalid property name",
"",
},
{
"invalid tagged2",
&InvalidTagged2{I: 1, J: 2},
&InvalidTagged2{},
"struct tag has repeated property name",
"",
},
{
"doubler",
&Doubler{S: "s", I: 1, B: true},
&Doubler{S: "ss", I: 2, B: true},
"",
"",
},
{
"save struct load props",
&X0{S: "s", I: 1},
&PropertyList{
Property{Name: "I", Value: int64(1), NoIndex: false, Multiple: false},
Property{Name: "S", Value: "s", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"save props load struct",
&PropertyList{
Property{Name: "S", Value: "s", NoIndex: false, Multiple: false},
Property{Name: "I", Value: int64(1), NoIndex: false, Multiple: false},
},
&X0{S: "s", I: 1},
"",
"",
},
{
"nil-value props",
&PropertyList{
Property{Name: "I", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "B", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "S", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "F", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "K", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "T", Value: nil, NoIndex: false, Multiple: false},
Property{Name: "J", Value: nil, NoIndex: false, Multiple: true},
Property{Name: "J", Value: int64(7), NoIndex: false, Multiple: true},
Property{Name: "J", Value: nil, NoIndex: false, Multiple: true},
},
&struct {
I int64
B bool
S string
F float64
K *Key
T time.Time
J []int64
}{
J: []int64{0, 7, 0},
},
"",
"",
},
{
"save outer load props",
&Outer{
A: 1,
I: []Inner1{
{10, "ten"},
{20, "twenty"},
{30, "thirty"},
},
J: Inner2{
Y: 3.14,
},
Inner3: Inner3{
Z: true,
},
},
&PropertyList{
Property{Name: "A", Value: int64(1), NoIndex: false, Multiple: false},
Property{Name: "I.W", Value: int64(10), NoIndex: false, Multiple: true},
Property{Name: "I.W", Value: int64(20), NoIndex: false, Multiple: true},
Property{Name: "I.W", Value: int64(30), NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "ten", NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "twenty", NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "thirty", NoIndex: false, Multiple: true},
Property{Name: "J.Y", Value: float64(3.14), NoIndex: false, Multiple: false},
Property{Name: "Z", Value: true, NoIndex: false, Multiple: false},
},
"",
"",
},
{
"save props load outer-equivalent",
&PropertyList{
Property{Name: "A", Value: int64(1), NoIndex: false, Multiple: false},
Property{Name: "I.W", Value: int64(10), NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "ten", NoIndex: false, Multiple: true},
Property{Name: "I.W", Value: int64(20), NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "twenty", NoIndex: false, Multiple: true},
Property{Name: "I.W", Value: int64(30), NoIndex: false, Multiple: true},
Property{Name: "I.X", Value: "thirty", NoIndex: false, Multiple: true},
Property{Name: "J.Y", Value: float64(3.14), NoIndex: false, Multiple: false},
Property{Name: "Z", Value: true, NoIndex: false, Multiple: false},
},
&OuterEquivalent{
A: 1,
IDotW: []int32{10, 20, 30},
IDotX: []string{"ten", "twenty", "thirty"},
JDotY: 3.14,
Z: true,
},
"",
"",
},
{
"save outer-equivalent load outer",
&OuterEquivalent{
A: 1,
IDotW: []int32{10, 20, 30},
IDotX: []string{"ten", "twenty", "thirty"},
JDotY: 3.14,
Z: true,
},
&Outer{
A: 1,
I: []Inner1{
{10, "ten"},
{20, "twenty"},
{30, "thirty"},
},
J: Inner2{
Y: 3.14,
},
Inner3: Inner3{
Z: true,
},
},
"",
"",
},
{
"dotted names save",
&Dotted{A: DottedA{B: DottedB{C: 88}}},
&PropertyList{
Property{Name: "A0.A1.A2.B3.C4.C5", Value: int64(88), NoIndex: false, Multiple: false},
},
"",
"",
},
{
"dotted names load",
&PropertyList{
Property{Name: "A0.A1.A2.B3.C4.C5", Value: int64(99), NoIndex: false, Multiple: false},
},
&Dotted{A: DottedA{B: DottedB{C: 99}}},
"",
"",
},
{
"save struct load deriver",
&X0{S: "s", I: 1},
&Deriver{S: "s", Derived: "derived+s"},
"",
"",
},
{
"save deriver load struct",
&Deriver{S: "s", Derived: "derived+s", Ignored: "ignored"},
&X0{S: "s"},
"",
"",
},
{
"bad multi-prop entity",
&BadMultiPropEntity{},
&BadMultiPropEntity{},
"Multiple is false",
"",
},
// Regression: CL 25062824 broke handling of appengine.BlobKey fields.
{
"appengine.BlobKey",
&BK{Key: "blah"},
&BK{Key: "blah"},
"",
"",
},
{
"zero time.Time",
&T{T: time.Time{}},
&T{T: time.Time{}},
"",
"",
},
{
"time.Time near Unix zero time",
&T{T: time.Unix(0, 4e3)},
&T{T: time.Unix(0, 4e3)},
"",
"",
},
{
"time.Time, far in the future",
&T{T: time.Date(99999, 1, 1, 0, 0, 0, 0, time.UTC)},
&T{T: time.Date(99999, 1, 1, 0, 0, 0, 0, time.UTC)},
"",
"",
},
{
"time.Time, very far in the past",
&T{T: time.Date(-300000, 1, 1, 0, 0, 0, 0, time.UTC)},
&T{},
"time value out of range",
"",
},
{
"time.Time, very far in the future",
&T{T: time.Date(294248, 1, 1, 0, 0, 0, 0, time.UTC)},
&T{},
"time value out of range",
"",
},
{
"structs",
&N0{
X0: X0{S: "one", I: 2, i: 3},
Nonymous: X0{S: "four", I: 5, i: 6},
Ignore: "ignore",
Other: "other",
},
&N0{
X0: X0{S: "one", I: 2},
Nonymous: X0{S: "four", I: 5},
Other: "other",
},
"",
"",
},
{
"slice of structs",
&N1{
X0: X0{S: "one", I: 2, i: 3},
Nonymous: []X0{
{S: "four", I: 5, i: 6},
{S: "seven", I: 8, i: 9},
{S: "ten", I: 11, i: 12},
{S: "thirteen", I: 14, i: 15},
},
Ignore: "ignore",
Other: "other",
},
&N1{
X0: X0{S: "one", I: 2},
Nonymous: []X0{
{S: "four", I: 5},
{S: "seven", I: 8},
{S: "ten", I: 11},
{S: "thirteen", I: 14},
},
Other: "other",
},
"",
"",
},
{
"structs with slices of structs",
&N2{
N1: N1{
X0: X0{S: "rouge"},
Nonymous: []X0{
{S: "rosso0"},
{S: "rosso1"},
},
},
Green: N1{
X0: X0{S: "vert"},
Nonymous: []X0{
{S: "verde0"},
{S: "verde1"},
{S: "verde2"},
},
},
Blue: N1{
X0: X0{S: "bleu"},
Nonymous: []X0{
{S: "blu0"},
{S: "blu1"},
{S: "blu2"},
{S: "blu3"},
},
},
},
&N2{
N1: N1{
X0: X0{S: "rouge"},
Nonymous: []X0{
{S: "rosso0"},
{S: "rosso1"},
},
},
Green: N1{
X0: X0{S: "vert"},
Nonymous: []X0{
{S: "verde0"},
{S: "verde1"},
{S: "verde2"},
},
},
Blue: N1{
X0: X0{S: "bleu"},
Nonymous: []X0{
{S: "blu0"},
{S: "blu1"},
{S: "blu2"},
{S: "blu3"},
},
},
},
"",
"",
},
{
"save structs load props",
&N2{
N1: N1{
X0: X0{S: "rouge"},
Nonymous: []X0{
{S: "rosso0"},
{S: "rosso1"},
},
},
Green: N1{
X0: X0{S: "vert"},
Nonymous: []X0{
{S: "verde0"},
{S: "verde1"},
{S: "verde2"},
},
},
Blue: N1{
X0: X0{S: "bleu"},
Nonymous: []X0{
{S: "blu0"},
{S: "blu1"},
{S: "blu2"},
{S: "blu3"},
},
},
},
&PropertyList{
Property{Name: "Blue.I", Value: int64(0), NoIndex: false, Multiple: false},
Property{Name: "Blue.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blu0", NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blu1", NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blu2", NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blu3", NoIndex: false, Multiple: true},
Property{Name: "Blue.Other", Value: "", NoIndex: false, Multiple: false},
Property{Name: "Blue.S", Value: "bleu", NoIndex: false, Multiple: false},
Property{Name: "green.I", Value: int64(0), NoIndex: false, Multiple: false},
Property{Name: "green.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.S", Value: "verde0", NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.S", Value: "verde1", NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.S", Value: "verde2", NoIndex: false, Multiple: true},
Property{Name: "green.Other", Value: "", NoIndex: false, Multiple: false},
Property{Name: "green.S", Value: "vert", NoIndex: false, Multiple: false},
Property{Name: "red.I", Value: int64(0), NoIndex: false, Multiple: false},
Property{Name: "red.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "red.Nonymous.I", Value: int64(0), NoIndex: false, Multiple: true},
Property{Name: "red.Nonymous.S", Value: "rosso0", NoIndex: false, Multiple: true},
Property{Name: "red.Nonymous.S", Value: "rosso1", NoIndex: false, Multiple: true},
Property{Name: "red.Other", Value: "", NoIndex: false, Multiple: false},
Property{Name: "red.S", Value: "rouge", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"save props load structs with ragged fields",
&PropertyList{
Property{Name: "red.S", Value: "rot", NoIndex: false, Multiple: false},
Property{Name: "green.Nonymous.I", Value: int64(10), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.I", Value: int64(11), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.I", Value: int64(12), NoIndex: false, Multiple: true},
Property{Name: "green.Nonymous.I", Value: int64(13), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blau0", NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.I", Value: int64(20), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blau1", NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.I", Value: int64(21), NoIndex: false, Multiple: true},
Property{Name: "Blue.Nonymous.S", Value: "blau2", NoIndex: false, Multiple: true},
},
&N2{
N1: N1{
X0: X0{S: "rot"},
},
Green: N1{
Nonymous: []X0{
{I: 10},
{I: 11},
{I: 12},
{I: 13},
},
},
Blue: N1{
Nonymous: []X0{
{S: "blau0", I: 20},
{S: "blau1", I: 21},
{S: "blau2"},
},
},
},
"",
"",
},
{
"save structs with noindex tags",
&struct {
A struct {
X string `datastore:",noindex"`
Y string
} `datastore:",noindex"`
B struct {
X string `datastore:",noindex"`
Y string
}
}{},
&PropertyList{
Property{Name: "A.X", Value: "", NoIndex: true, Multiple: false},
Property{Name: "A.Y", Value: "", NoIndex: true, Multiple: false},
Property{Name: "B.X", Value: "", NoIndex: true, Multiple: false},
Property{Name: "B.Y", Value: "", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"embedded struct with name override",
&struct {
Inner1 `datastore:"foo"`
}{},
&PropertyList{
Property{Name: "foo.W", Value: int64(0), NoIndex: false, Multiple: false},
Property{Name: "foo.X", Value: "", NoIndex: false, Multiple: false},
},
"",
"",
},
{
"slice of slices",
&SliceOfSlices{},
nil,
"flattening nested structs leads to a slice of slices",
"",
},
{
"recursive struct",
&Recursive{},
nil,
"recursive struct",
"",
},
{
"mutually recursive struct",
&MutuallyRecursive0{},
nil,
"recursive struct",
"",
},
{
"non-exported struct fields",
&struct {
i, J int64
}{i: 1, J: 2},
&PropertyList{
Property{Name: "J", Value: int64(2), NoIndex: false, Multiple: false},
},
"",
"",
},
{
"json.RawMessage",
&struct {
J json.RawMessage
}{
J: json.RawMessage("rawr"),
},
&PropertyList{
Property{Name: "J", Value: []byte("rawr"), NoIndex: true, Multiple: false},
},
"",
"",
},
{
"json.RawMessage to myBlob",
&struct {
B json.RawMessage
}{
B: json.RawMessage("rawr"),
},
&B2{B: myBlob("rawr")},
"",
"",
},
{
"embedded time field",
&SpecialTime{MyTime: EmbeddedTime{now}},
&SpecialTime{MyTime: EmbeddedTime{now}},
"",
"",
},
{
"embedded time load",
&PropertyList{
Property{Name: "MyTime.", Value: now, NoIndex: false, Multiple: false},
},
&SpecialTime{MyTime: EmbeddedTime{now}},
"",
"",
},
}
// checkErr returns the empty string if either both want and err are zero,
// or if want is a non-empty substring of err's string representation.
func checkErr(want string, err error) string {
if err != nil {
got := err.Error()
if want == "" || strings.Index(got, want) == -1 {
return got
}
} else if want != "" {
return fmt.Sprintf("want error %q", want)
}
return ""
}
func TestRoundTrip(t *testing.T) {
for _, tc := range testCases {
p, err := saveEntity(testAppID, testKey0, tc.src)
if s := checkErr(tc.putErr, err); s != "" {
t.Errorf("%s: save: %s", tc.desc, s)
continue
}
if p == nil {
continue
}
var got interface{}
if _, ok := tc.want.(*PropertyList); ok {
got = new(PropertyList)
} else {
got = reflect.New(reflect.TypeOf(tc.want).Elem()).Interface()
}
err = loadEntity(got, p)
if s := checkErr(tc.getErr, err); s != "" {
t.Errorf("%s: load: %s", tc.desc, s)
continue
}
if pl, ok := got.(*PropertyList); ok {
// Sort by name to make sure we have a deterministic order.
sort.Stable(byName(*pl))
}
equal := false
if gotT, ok := got.(*T); ok {
// Round tripping a time.Time can result in a different time.Location: Local instead of UTC.
// We therefore test equality explicitly, instead of relying on reflect.DeepEqual.
equal = gotT.T.Equal(tc.want.(*T).T)
} else {
equal = reflect.DeepEqual(got, tc.want)
}
if !equal {
t.Errorf("%s: compare: got %v want %v", tc.desc, got, tc.want)
continue
}
}
}
type byName PropertyList
func (s byName) Len() int { return len(s) }
func (s byName) Less(i, j int) bool { return s[i].Name < s[j].Name }
func (s byName) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func TestQueryConstruction(t *testing.T) {
tests := []struct {
q, exp *Query
err string
}{
{
q: NewQuery("Foo"),
exp: &Query{
kind: "Foo",
limit: -1,
},
},
{
// Regular filtered query with standard spacing.
q: NewQuery("Foo").Filter("foo >", 7),
exp: &Query{
kind: "Foo",
filter: []filter{
{
FieldName: "foo",
Op: greaterThan,
Value: 7,
},
},
limit: -1,
},
},
{
// Filtered query with no spacing.
q: NewQuery("Foo").Filter("foo=", 6),
exp: &Query{
kind: "Foo",
filter: []filter{
{
FieldName: "foo",
Op: equal,
Value: 6,
},
},
limit: -1,
},
},
{
// Filtered query with funky spacing.
q: NewQuery("Foo").Filter(" foo< ", 8),
exp: &Query{
kind: "Foo",
filter: []filter{
{
FieldName: "foo",
Op: lessThan,
Value: 8,
},
},
limit: -1,
},
},
{
// Filtered query with multicharacter op.
q: NewQuery("Foo").Filter("foo >=", 9),
exp: &Query{
kind: "Foo",
filter: []filter{
{
FieldName: "foo",
Op: greaterEq,
Value: 9,
},
},
limit: -1,
},
},
{
// Query with ordering.
q: NewQuery("Foo").Order("bar"),
exp: &Query{
kind: "Foo",
order: []order{
{
FieldName: "bar",
Direction: ascending,
},
},
limit: -1,
},
},
{
// Query with reverse ordering, and funky spacing.
q: NewQuery("Foo").Order(" - bar"),
exp: &Query{
kind: "Foo",
order: []order{
{
FieldName: "bar",
Direction: descending,
},
},
limit: -1,
},
},
{
// Query with an empty ordering.
q: NewQuery("Foo").Order(""),
err: "empty order",
},
{
// Query with a + ordering.
q: NewQuery("Foo").Order("+bar"),
err: "invalid order",
},
}
for i, test := range tests {
if test.q.err != nil {
got := test.q.err.Error()
if !strings.Contains(got, test.err) {
t.Errorf("%d: error mismatch: got %q want something containing %q", i, got, test.err)
}
continue
}
if !reflect.DeepEqual(test.q, test.exp) {
t.Errorf("%d: mismatch: got %v want %v", i, test.q, test.exp)
}
}
}
func TestStringMeaning(t *testing.T) {
var xx [4]interface{}
xx[0] = &struct {
X string
}{"xx0"}
xx[1] = &struct {
X string `datastore:",noindex"`
}{"xx1"}
xx[2] = &struct {
X []byte
}{[]byte("xx2")}
xx[3] = &struct {
X []byte `datastore:",noindex"`
}{[]byte("xx3")}
indexed := [4]bool{
true,
false,
false, // A []byte is always no-index.
false,
}
want := [4]pb.Property_Meaning{
pb.Property_NO_MEANING,
pb.Property_TEXT,
pb.Property_BLOB,
pb.Property_BLOB,
}
for i, x := range xx {
props, err := SaveStruct(x)
if err != nil {
t.Errorf("i=%d: SaveStruct: %v", i, err)
continue
}
e, err := propertiesToProto("appID", testKey0, props)
if err != nil {
t.Errorf("i=%d: propertiesToProto: %v", i, err)
continue
}
var p *pb.Property
switch {
case indexed[i] && len(e.Property) == 1:
p = e.Property[0]
case !indexed[i] && len(e.RawProperty) == 1:
p = e.RawProperty[0]
default:
t.Errorf("i=%d: EntityProto did not have expected property slice", i)
continue
}
if got := p.GetMeaning(); got != want[i] {
t.Errorf("i=%d: meaning: got %v, want %v", i, got, want[i])
continue
}
}
}
func TestNamespaceResetting(t *testing.T) {
// These environment variables are necessary because *Query.Run will
// call internal.FullyQualifiedAppID which checks these variables or falls
// back to the Metadata service that is not available in tests.
environ := []struct {
key, value string
}{
{"GAE_LONG_APP_ID", "my-app-id"},
{"GAE_PARTITION", "1"},
}
for _, v := range environ {
old := os.Getenv(v.key)
os.Setenv(v.key, v.value)
v.value = old
}
defer func() { // Restore old environment after the test completes.
for _, v := range environ {
if v.value == "" {
os.Unsetenv(v.key)
continue
}
os.Setenv(v.key, v.value)
}
}()
namec := make(chan *string, 1)
c0 := aetesting.FakeSingleContext(t, "datastore_v3", "RunQuery", func(req *pb.Query, res *pb.QueryResult) error {
namec <- req.NameSpace
return fmt.Errorf("RPC error")
})
// Check that wrapping c0 in a namespace twice works correctly.
c1, err := appengine.Namespace(c0, "A")
if err != nil {
t.Fatalf("appengine.Namespace: %v", err)
}
c2, err := appengine.Namespace(c1, "") // should act as the original context
if err != nil {
t.Fatalf("appengine.Namespace: %v", err)
}
q := NewQuery("SomeKind")
q.Run(c0)
if ns := <-namec; ns != nil {
t.Errorf(`RunQuery with c0: ns = %q, want nil`, *ns)
}
q.Run(c1)
if ns := <-namec; ns == nil {
t.Error(`RunQuery with c1: ns = nil, want "A"`)
} else if *ns != "A" {
t.Errorf(`RunQuery with c1: ns = %q, want "A"`, *ns)
}
q.Run(c2)
if ns := <-namec; ns != nil {
t.Errorf(`RunQuery with c2: ns = %q, want nil`, *ns)
}
}
appengine-1.6.7/datastore/doc.go 0000664 0000000 0000000 00000026500 13741174344 0016570 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
/*
Package datastore provides a client for App Engine's datastore service.
Basic Operations
Entities are the unit of storage and are associated with a key. A key
consists of an optional parent key, a string application ID, a string kind
(also known as an entity type), and either a StringID or an IntID. A
StringID is also known as an entity name or key name.
It is valid to create a key with a zero StringID and a zero IntID; this is
called an incomplete key, and does not refer to any saved entity. Putting an
entity into the datastore under an incomplete key will cause a unique key
to be generated for that entity, with a non-zero IntID.
An entity's contents are a mapping from case-sensitive field names to values.
Valid value types are:
- signed integers (int, int8, int16, int32 and int64),
- bool,
- string,
- float32 and float64,
- []byte (up to 1 megabyte in length),
- any type whose underlying type is one of the above predeclared types,
- ByteString,
- *Key,
- time.Time (stored with microsecond precision),
- appengine.BlobKey,
- appengine.GeoPoint,
- structs whose fields are all valid value types,
- slices of any of the above.
Slices of structs are valid, as are structs that contain slices. However, if
one struct contains another, then at most one of those can be repeated. This
disqualifies recursively defined struct types: any struct T that (directly or
indirectly) contains a []T.
The Get and Put functions load and save an entity's contents. An entity's
contents are typically represented by a struct pointer.
Example code:
type Entity struct {
Value string
}
func handle(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
k := datastore.NewKey(ctx, "Entity", "stringID", 0, nil)
e := new(Entity)
if err := datastore.Get(ctx, k, e); err != nil {
http.Error(w, err.Error(), 500)
return
}
old := e.Value
e.Value = r.URL.Path
if _, err := datastore.Put(ctx, k, e); err != nil {
http.Error(w, err.Error(), 500)
return
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
fmt.Fprintf(w, "old=%q\nnew=%q\n", old, e.Value)
}
GetMulti, PutMulti and DeleteMulti are batch versions of the Get, Put and
Delete functions. They take a []*Key instead of a *Key, and may return an
appengine.MultiError when encountering partial failure.
Properties
An entity's contents can be represented by a variety of types. These are
typically struct pointers, but can also be any type that implements the
PropertyLoadSaver interface. If using a struct pointer, you do not have to
explicitly implement the PropertyLoadSaver interface; the datastore will
automatically convert via reflection. If a struct pointer does implement that
interface then those methods will be used in preference to the default
behavior for struct pointers. Struct pointers are more strongly typed and are
easier to use; PropertyLoadSavers are more flexible.
The actual types passed do not have to match between Get and Put calls or even
across different calls to datastore. It is valid to put a *PropertyList and
get that same entity as a *myStruct, or put a *myStruct0 and get a *myStruct1.
Conceptually, any entity is saved as a sequence of properties, and is loaded
into the destination value on a property-by-property basis. When loading into
a struct pointer, an entity that cannot be completely represented (such as a
missing field) will result in an ErrFieldMismatch error but it is up to the
caller whether this error is fatal, recoverable or ignorable.
By default, for struct pointers, all properties are potentially indexed, and
the property name is the same as the field name (and hence must start with an
upper case letter).
Fields may have a `datastore:"name,options"` tag. The tag name is the
property name, which must be one or more valid Go identifiers joined by ".",
but may start with a lower case letter. An empty tag name means to just use the
field name. A "-" tag name means that the datastore will ignore that field.
The only valid options are "omitempty" and "noindex".
If the options include "omitempty" and the value of the field is empty, then the field will be omitted on Save.
The empty values are false, 0, any nil interface value, and any array, slice, map, or string of length zero.
Struct field values will never be empty.
If options include "noindex" then the field will not be indexed. All fields are indexed
by default. Strings or byte slices longer than 1500 bytes cannot be indexed;
fields used to store long strings and byte slices must be tagged with "noindex"
or they will cause Put operations to fail.
To use multiple options together, separate them by a comma.
The order does not matter.
If the options is "" then the comma may be omitted.
Example code:
// A and B are renamed to a and b.
// A, C and J are not indexed.
// D's tag is equivalent to having no tag at all (E).
// I is ignored entirely by the datastore.
// J has tag information for both the datastore and json packages.
type TaggedStruct struct {
A int `datastore:"a,noindex"`
B int `datastore:"b"`
C int `datastore:",noindex"`
D int `datastore:""`
E int
I int `datastore:"-"`
J int `datastore:",noindex" json:"j"`
}
Structured Properties
If the struct pointed to contains other structs, then the nested or embedded
structs are flattened. For example, given these definitions:
type Inner1 struct {
W int32
X string
}
type Inner2 struct {
Y float64
}
type Inner3 struct {
Z bool
}
type Outer struct {
A int16
I []Inner1
J Inner2
Inner3
}
then an Outer's properties would be equivalent to those of:
type OuterEquivalent struct {
A int16
IDotW []int32 `datastore:"I.W"`
IDotX []string `datastore:"I.X"`
JDotY float64 `datastore:"J.Y"`
Z bool
}
If Outer's embedded Inner3 field was tagged as `datastore:"Foo"` then the
equivalent field would instead be: FooDotZ bool `datastore:"Foo.Z"`.
If an outer struct is tagged "noindex" then all of its implicit flattened
fields are effectively "noindex".
The PropertyLoadSaver Interface
An entity's contents can also be represented by any type that implements the
PropertyLoadSaver interface. This type may be a struct pointer, but it does
not have to be. The datastore package will call Load when getting the entity's
contents, and Save when putting the entity's contents.
Possible uses include deriving non-stored fields, verifying fields, or indexing
a field only if its value is positive.
Example code:
type CustomPropsExample struct {
I, J int
// Sum is not stored, but should always be equal to I + J.
Sum int `datastore:"-"`
}
func (x *CustomPropsExample) Load(ps []datastore.Property) error {
// Load I and J as usual.
if err := datastore.LoadStruct(x, ps); err != nil {
return err
}
// Derive the Sum field.
x.Sum = x.I + x.J
return nil
}
func (x *CustomPropsExample) Save() ([]datastore.Property, error) {
// Validate the Sum field.
if x.Sum != x.I + x.J {
return nil, errors.New("CustomPropsExample has inconsistent sum")
}
// Save I and J as usual. The code below is equivalent to calling
// "return datastore.SaveStruct(x)", but is done manually for
// demonstration purposes.
return []datastore.Property{
{
Name: "I",
Value: int64(x.I),
},
{
Name: "J",
Value: int64(x.J),
},
}, nil
}
The *PropertyList type implements PropertyLoadSaver, and can therefore hold an
arbitrary entity's contents.
Queries
Queries retrieve entities based on their properties or key's ancestry. Running
a query yields an iterator of results: either keys or (key, entity) pairs.
Queries are re-usable and it is safe to call Query.Run from concurrent
goroutines. Iterators are not safe for concurrent use.
Queries are immutable, and are either created by calling NewQuery, or derived
from an existing query by calling a method like Filter or Order that returns a
new query value. A query is typically constructed by calling NewQuery followed
by a chain of zero or more such methods. These methods are:
- Ancestor and Filter constrain the entities returned by running a query.
- Order affects the order in which they are returned.
- Project constrains the fields returned.
- Distinct de-duplicates projected entities.
- KeysOnly makes the iterator return only keys, not (key, entity) pairs.
- Start, End, Offset and Limit define which sub-sequence of matching entities
to return. Start and End take cursors, Offset and Limit take integers. Start
and Offset affect the first result, End and Limit affect the last result.
If both Start and Offset are set, then the offset is relative to Start.
If both End and Limit are set, then the earliest constraint wins. Limit is
relative to Start+Offset, not relative to End. As a special case, a
negative limit means unlimited.
Example code:
type Widget struct {
Description string
Price int
}
func handle(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
q := datastore.NewQuery("Widget").
Filter("Price <", 1000).
Order("-Price")
b := new(bytes.Buffer)
for t := q.Run(ctx); ; {
var x Widget
key, err := t.Next(&x)
if err == datastore.Done {
break
}
if err != nil {
serveError(ctx, w, err)
return
}
fmt.Fprintf(b, "Key=%v\nWidget=%#v\n\n", key, x)
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
io.Copy(w, b)
}
Transactions
RunInTransaction runs a function in a transaction.
Example code:
type Counter struct {
Count int
}
func inc(ctx context.Context, key *datastore.Key) (int, error) {
var x Counter
if err := datastore.Get(ctx, key, &x); err != nil && err != datastore.ErrNoSuchEntity {
return 0, err
}
x.Count++
if _, err := datastore.Put(ctx, key, &x); err != nil {
return 0, err
}
return x.Count, nil
}
func handle(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
var count int
err := datastore.RunInTransaction(ctx, func(ctx context.Context) error {
var err1 error
count, err1 = inc(ctx, datastore.NewKey(ctx, "Counter", "singleton", 0, nil))
return err1
}, nil)
if err != nil {
serveError(ctx, w, err)
return
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
fmt.Fprintf(w, "Count=%d", count)
}
Metadata
The datastore package provides access to some of App Engine's datastore
metadata. This metadata includes information about the entity groups,
namespaces, entity kinds, and properties in the datastore, as well as the
property representations for each property.
Example code:
func handle(w http.ResponseWriter, r *http.Request) {
// Print all the kinds in the datastore, with all the indexed
// properties (and their representations) for each.
ctx := appengine.NewContext(r)
kinds, err := datastore.Kinds(ctx)
if err != nil {
serveError(ctx, w, err)
return
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
for _, kind := range kinds {
fmt.Fprintf(w, "%s:\n", kind)
props, err := datastore.KindProperties(ctx, kind)
if err != nil {
fmt.Fprintln(w, "\t(unable to retrieve properties)")
continue
}
for p, rep := range props {
fmt.Fprintf(w, "\t-%s (%s)\n", p, strings.Join(rep, ", "))
}
}
}
*/
package datastore // import "google.golang.org/appengine/datastore"
appengine-1.6.7/datastore/internal/ 0000775 0000000 0000000 00000000000 13741174344 0017305 5 ustar 00root root 0000000 0000000 appengine-1.6.7/datastore/internal/cloudkey/ 0000775 0000000 0000000 00000000000 13741174344 0021124 5 ustar 00root root 0000000 0000000 appengine-1.6.7/datastore/internal/cloudkey/cloudkey.go 0000664 0000000 0000000 00000006454 13741174344 0023303 0 ustar 00root root 0000000 0000000 // Copyright 2019 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package cloudpb is a subset of types and functions, copied from cloud.google.com/go/datastore.
//
// They are copied here to provide compatibility to decode keys generated by the cloud.google.com/go/datastore package.
package cloudkey
import (
"encoding/base64"
"errors"
"strings"
"github.com/golang/protobuf/proto"
cloudpb "google.golang.org/appengine/datastore/internal/cloudpb"
)
/////////////////////////////////////////////////////////////////////
// Code below is copied from https://github.com/googleapis/google-cloud-go/blob/master/datastore/datastore.go
/////////////////////////////////////////////////////////////////////
var (
// ErrInvalidKey is returned when an invalid key is presented.
ErrInvalidKey = errors.New("datastore: invalid key")
)
/////////////////////////////////////////////////////////////////////
// Code below is copied from https://github.com/googleapis/google-cloud-go/blob/master/datastore/key.go
/////////////////////////////////////////////////////////////////////
// Key represents the datastore key for a stored entity.
type Key struct {
// Kind cannot be empty.
Kind string
// Either ID or Name must be zero for the Key to be valid.
// If both are zero, the Key is incomplete.
ID int64
Name string
// Parent must either be a complete Key or nil.
Parent *Key
// Namespace provides the ability to partition your data for multiple
// tenants. In most cases, it is not necessary to specify a namespace.
// See docs on datastore multitenancy for details:
// https://cloud.google.com/datastore/docs/concepts/multitenancy
Namespace string
}
// DecodeKey decodes a key from the opaque representation returned by Encode.
func DecodeKey(encoded string) (*Key, error) {
// Re-add padding.
if m := len(encoded) % 4; m != 0 {
encoded += strings.Repeat("=", 4-m)
}
b, err := base64.URLEncoding.DecodeString(encoded)
if err != nil {
return nil, err
}
pKey := new(cloudpb.Key)
if err := proto.Unmarshal(b, pKey); err != nil {
return nil, err
}
return protoToKey(pKey)
}
// valid returns whether the key is valid.
func (k *Key) valid() bool {
if k == nil {
return false
}
for ; k != nil; k = k.Parent {
if k.Kind == "" {
return false
}
if k.Name != "" && k.ID != 0 {
return false
}
if k.Parent != nil {
if k.Parent.Incomplete() {
return false
}
if k.Parent.Namespace != k.Namespace {
return false
}
}
}
return true
}
// Incomplete reports whether the key does not refer to a stored entity.
func (k *Key) Incomplete() bool {
return k.Name == "" && k.ID == 0
}
// protoToKey decodes a protocol buffer representation of a key into an
// equivalent *Key object. If the key is invalid, protoToKey will return the
// invalid key along with ErrInvalidKey.
func protoToKey(p *cloudpb.Key) (*Key, error) {
var key *Key
var namespace string
if partition := p.PartitionId; partition != nil {
namespace = partition.NamespaceId
}
for _, el := range p.Path {
key = &Key{
Namespace: namespace,
Kind: el.Kind,
ID: el.GetId(),
Name: el.GetName(),
Parent: key,
}
}
if !key.valid() { // Also detects key == nil.
return key, ErrInvalidKey
}
return key, nil
}
appengine-1.6.7/datastore/internal/cloudpb/ 0000775 0000000 0000000 00000000000 13741174344 0020735 5 ustar 00root root 0000000 0000000 appengine-1.6.7/datastore/internal/cloudpb/entity.pb.go 0000664 0000000 0000000 00000034555 13741174344 0023214 0 ustar 00root root 0000000 0000000 // Copyright 2019 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// Package cloudpb is a subset of protobufs, copied from google.golang.org/genproto/googleapis/datastore/v1.
//
// They are copied here to provide compatibility to decode keys generated by the cloud.google.com/go/datastore package.
package cloudpb
import (
"fmt"
"github.com/golang/protobuf/proto"
)
// A partition ID identifies a grouping of entities. The grouping is always
// by project and namespace, however the namespace ID may be empty.
//
// A partition ID contains several dimensions:
// project ID and namespace ID.
//
// Partition dimensions:
//
// - May be `""`.
// - Must be valid UTF-8 bytes.
// - Must have values that match regex `[A-Za-z\d\.\-_]{1,100}`
// If the value of any dimension matches regex `__.*__`, the partition is
// reserved/read-only.
// A reserved/read-only partition ID is forbidden in certain documented
// contexts.
//
// Foreign partition IDs (in which the project ID does
// not match the context project ID ) are discouraged.
// Reads and writes of foreign partition IDs may fail if the project is not in
// an active state.
type PartitionId struct {
// The ID of the project to which the entities belong.
ProjectId string `protobuf:"bytes,2,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"`
// If not empty, the ID of the namespace to which the entities belong.
NamespaceId string `protobuf:"bytes,4,opt,name=namespace_id,json=namespaceId,proto3" json:"namespace_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *PartitionId) Reset() { *m = PartitionId{} }
func (m *PartitionId) String() string { return proto.CompactTextString(m) }
func (*PartitionId) ProtoMessage() {}
func (*PartitionId) Descriptor() ([]byte, []int) {
return fileDescriptor_entity_096a297364b049a5, []int{0}
}
func (m *PartitionId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PartitionId.Unmarshal(m, b)
}
func (m *PartitionId) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_PartitionId.Marshal(b, m, deterministic)
}
func (dst *PartitionId) XXX_Merge(src proto.Message) {
xxx_messageInfo_PartitionId.Merge(dst, src)
}
func (m *PartitionId) XXX_Size() int {
return xxx_messageInfo_PartitionId.Size(m)
}
func (m *PartitionId) XXX_DiscardUnknown() {
xxx_messageInfo_PartitionId.DiscardUnknown(m)
}
var xxx_messageInfo_PartitionId proto.InternalMessageInfo
func (m *PartitionId) GetProjectId() string {
if m != nil {
return m.ProjectId
}
return ""
}
func (m *PartitionId) GetNamespaceId() string {
if m != nil {
return m.NamespaceId
}
return ""
}
// A unique identifier for an entity.
// If a key's partition ID or any of its path kinds or names are
// reserved/read-only, the key is reserved/read-only.
// A reserved/read-only key is forbidden in certain documented contexts.
type Key struct {
// Entities are partitioned into subsets, currently identified by a project
// ID and namespace ID.
// Queries are scoped to a single partition.
PartitionId *PartitionId `protobuf:"bytes,1,opt,name=partition_id,json=partitionId,proto3" json:"partition_id,omitempty"`
// The entity path.
// An entity path consists of one or more elements composed of a kind and a
// string or numerical identifier, which identify entities. The first
// element identifies a _root entity_, the second element identifies
// a _child_ of the root entity, the third element identifies a child of the
// second entity, and so forth. The entities identified by all prefixes of
// the path are called the element's _ancestors_.
//
// An entity path is always fully complete: *all* of the entity's ancestors
// are required to be in the path along with the entity identifier itself.
// The only exception is that in some documented cases, the identifier in the
// last path element (for the entity) itself may be omitted. For example,
// the last path element of the key of `Mutation.insert` may have no
// identifier.
//
// A path can never be empty, and a path can have at most 100 elements.
Path []*Key_PathElement `protobuf:"bytes,2,rep,name=path,proto3" json:"path,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *Key) Reset() { *m = Key{} }
func (m *Key) String() string { return proto.CompactTextString(m) }
func (*Key) ProtoMessage() {}
func (*Key) Descriptor() ([]byte, []int) {
return fileDescriptor_entity_096a297364b049a5, []int{1}
}
func (m *Key) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Key.Unmarshal(m, b)
}
func (m *Key) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_Key.Marshal(b, m, deterministic)
}
func (dst *Key) XXX_Merge(src proto.Message) {
xxx_messageInfo_Key.Merge(dst, src)
}
func (m *Key) XXX_Size() int {
return xxx_messageInfo_Key.Size(m)
}
func (m *Key) XXX_DiscardUnknown() {
xxx_messageInfo_Key.DiscardUnknown(m)
}
// A (kind, ID/name) pair used to construct a key path.
//
// If either name or ID is set, the element is complete.
// If neither is set, the element is incomplete.
type Key_PathElement struct {
// The kind of the entity.
// A kind matching regex `__.*__` is reserved/read-only.
// A kind must not contain more than 1500 bytes when UTF-8 encoded.
// Cannot be `""`.
Kind string `protobuf:"bytes,1,opt,name=kind,proto3" json:"kind,omitempty"`
// The type of ID.
//
// Types that are valid to be assigned to IdType:
// *Key_PathElement_Id
// *Key_PathElement_Name
IdType isKey_PathElement_IdType `protobuf_oneof:"id_type"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *Key_PathElement) Reset() { *m = Key_PathElement{} }
func (m *Key_PathElement) String() string { return proto.CompactTextString(m) }
func (*Key_PathElement) ProtoMessage() {}
func (*Key_PathElement) Descriptor() ([]byte, []int) {
return fileDescriptor_entity_096a297364b049a5, []int{1, 0}
}
func (m *Key_PathElement) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Key_PathElement.Unmarshal(m, b)
}
func (m *Key_PathElement) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_Key_PathElement.Marshal(b, m, deterministic)
}
func (dst *Key_PathElement) XXX_Merge(src proto.Message) {
xxx_messageInfo_Key_PathElement.Merge(dst, src)
}
func (m *Key_PathElement) XXX_Size() int {
return xxx_messageInfo_Key_PathElement.Size(m)
}
func (m *Key_PathElement) XXX_DiscardUnknown() {
xxx_messageInfo_Key_PathElement.DiscardUnknown(m)
}
var xxx_messageInfo_Key_PathElement proto.InternalMessageInfo
func (m *Key_PathElement) GetKind() string {
if m != nil {
return m.Kind
}
return ""
}
type isKey_PathElement_IdType interface {
isKey_PathElement_IdType()
}
type Key_PathElement_Id struct {
Id int64 `protobuf:"varint,2,opt,name=id,proto3,oneof"`
}
type Key_PathElement_Name struct {
Name string `protobuf:"bytes,3,opt,name=name,proto3,oneof"`
}
func (*Key_PathElement_Id) isKey_PathElement_IdType() {}
func (*Key_PathElement_Name) isKey_PathElement_IdType() {}
func (m *Key_PathElement) GetIdType() isKey_PathElement_IdType {
if m != nil {
return m.IdType
}
return nil
}
func (m *Key_PathElement) GetId() int64 {
if x, ok := m.GetIdType().(*Key_PathElement_Id); ok {
return x.Id
}
return 0
}
func (m *Key_PathElement) GetName() string {
if x, ok := m.GetIdType().(*Key_PathElement_Name); ok {
return x.Name
}
return ""
}
// XXX_OneofFuncs is for the internal use of the proto package.
func (*Key_PathElement) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
return _Key_PathElement_OneofMarshaler, _Key_PathElement_OneofUnmarshaler, _Key_PathElement_OneofSizer, []interface{}{
(*Key_PathElement_Id)(nil),
(*Key_PathElement_Name)(nil),
}
}
func _Key_PathElement_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
m := msg.(*Key_PathElement)
// id_type
switch x := m.IdType.(type) {
case *Key_PathElement_Id:
b.EncodeVarint(2<<3 | proto.WireVarint)
b.EncodeVarint(uint64(x.Id))
case *Key_PathElement_Name:
b.EncodeVarint(3<<3 | proto.WireBytes)
b.EncodeStringBytes(x.Name)
case nil:
default:
return fmt.Errorf("Key_PathElement.IdType has unexpected type %T", x)
}
return nil
}
func _Key_PathElement_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
m := msg.(*Key_PathElement)
switch tag {
case 2: // id_type.id
if wire != proto.WireVarint {
return true, proto.ErrInternalBadWireType
}
x, err := b.DecodeVarint()
m.IdType = &Key_PathElement_Id{int64(x)}
return true, err
case 3: // id_type.name
if wire != proto.WireBytes {
return true, proto.ErrInternalBadWireType
}
x, err := b.DecodeStringBytes()
m.IdType = &Key_PathElement_Name{x}
return true, err
default:
return false, nil
}
}
func _Key_PathElement_OneofSizer(msg proto.Message) (n int) {
m := msg.(*Key_PathElement)
// id_type
switch x := m.IdType.(type) {
case *Key_PathElement_Id:
n += 1 // tag and wire
n += proto.SizeVarint(uint64(x.Id))
case *Key_PathElement_Name:
n += 1 // tag and wire
n += proto.SizeVarint(uint64(len(x.Name)))
n += len(x.Name)
case nil:
default:
panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
}
return n
}
var fileDescriptor_entity_096a297364b049a5 = []byte{
// 780 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x7c, 0x94, 0xff, 0x6e, 0xdc, 0x44,
0x10, 0xc7, 0xed, 0xbb, 0x5c, 0x1a, 0x8f, 0xdd, 0xa4, 0x6c, 0x2a, 0x61, 0x02, 0x28, 0x26, 0x80,
0x74, 0x02, 0xc9, 0x6e, 0xc2, 0x1f, 0x54, 0x14, 0xa4, 0x72, 0x25, 0xe0, 0x28, 0x15, 0x9c, 0x56,
0x55, 0x24, 0x50, 0xa4, 0xd3, 0xde, 0x79, 0xeb, 0x2e, 0x67, 0xef, 0x5a, 0xf6, 0x3a, 0xaa, 0xdf,
0x05, 0xf1, 0x00, 0x3c, 0x0a, 0x8f, 0x80, 0x78, 0x18, 0xb4, 0x3f, 0xec, 0x0b, 0xed, 0x35, 0xff,
0x79, 0x67, 0x3e, 0xdf, 0xd9, 0xef, 0xec, 0xce, 0x1a, 0xa2, 0x5c, 0x88, 0xbc, 0xa0, 0x49, 0x46,
0x24, 0x69, 0xa4, 0xa8, 0x69, 0x72, 0x73, 0x9a, 0x50, 0x2e, 0x99, 0xec, 0xe2, 0xaa, 0x16, 0x52,
0xa0, 0x43, 0x43, 0xc4, 0x03, 0x11, 0xdf, 0x9c, 0x1e, 0x7d, 0x64, 0x65, 0xa4, 0x62, 0x09, 0xe1,
0x5c, 0x48, 0x22, 0x99, 0xe0, 0x8d, 0x91, 0x0c, 0x59, 0xbd, 0x5a, 0xb6, 0x2f, 0x93, 0x46, 0xd6,
0xed, 0x4a, 0xda, 0xec, 0xf1, 0x9b, 0x59, 0xc9, 0x4a, 0xda, 0x48, 0x52, 0x56, 0x16, 0x08, 0x2d,
0x20, 0xbb, 0x8a, 0x26, 0x05, 0x91, 0x05, 0xcf, 0x4d, 0xe6, 0xe4, 0x17, 0xf0, 0xe7, 0xa4, 0x96,
0x4c, 0x6d, 0x76, 0x91, 0xa1, 0x8f, 0x01, 0xaa, 0x5a, 0xfc, 0x4e, 0x57, 0x72, 0xc1, 0xb2, 0x70,
0x14, 0xb9, 0x53, 0x0f, 0x7b, 0x36, 0x72, 0x91, 0xa1, 0x4f, 0x20, 0xe0, 0xa4, 0xa4, 0x4d, 0x45,
0x56, 0x54, 0x01, 0x3b, 0x1a, 0xf0, 0x87, 0xd8, 0x45, 0x76, 0xf2, 0x8f, 0x0b, 0xe3, 0x4b, 0xda,
0xa1, 0x67, 0x10, 0x54, 0x7d, 0x61, 0x85, 0xba, 0x91, 0x3b, 0xf5, 0xcf, 0xa2, 0x78, 0x4b, 0xef,
0xf1, 0x2d, 0x07, 0xd8, 0xaf, 0x6e, 0xd9, 0x79, 0x0c, 0x3b, 0x15, 0x91, 0xaf, 0xc2, 0x51, 0x34,
0x9e, 0xfa, 0x67, 0x9f, 0x6d, 0x15, 0x5f, 0xd2, 0x2e, 0x9e, 0x13, 0xf9, 0xea, 0xbc, 0xa0, 0x25,
0xe5, 0x12, 0x6b, 0xc5, 0xd1, 0x0b, 0xd5, 0xd7, 0x10, 0x44, 0x08, 0x76, 0xd6, 0x8c, 0x1b, 0x17,
0x1e, 0xd6, 0xdf, 0xe8, 0x01, 0x8c, 0x6c, 0x8f, 0xe3, 0xd4, 0xc1, 0x23, 0x96, 0xa1, 0x87, 0xb0,
0xa3, 0x5a, 0x09, 0xc7, 0x8a, 0x4a, 0x1d, 0xac, 0x57, 0x33, 0x0f, 0xee, 0xb1, 0x6c, 0xa1, 0x8e,
0xee, 0xe4, 0x29, 0xc0, 0xf7, 0x75, 0x4d, 0xba, 0x2b, 0x52, 0xb4, 0x14, 0x9d, 0xc1, 0xee, 0x8d,
0xfa, 0x68, 0x42, 0x57, 0xfb, 0x3b, 0xda, 0xea, 0x4f, 0xb3, 0xd8, 0x92, 0x27, 0x7f, 0x4c, 0x60,
0x62, 0xd4, 0x4f, 0x00, 0x78, 0x5b, 0x14, 0x0b, 0x9d, 0x08, 0xfd, 0xc8, 0x9d, 0xee, 0x6f, 0x2a,
0xf4, 0x37, 0x19, 0xff, 0xdc, 0x16, 0x85, 0xe6, 0x53, 0x07, 0x7b, 0xbc, 0x5f, 0xa0, 0xcf, 0xe1,
0xfe, 0x52, 0x88, 0x82, 0x12, 0x6e, 0xf5, 0xaa, 0xb1, 0xbd, 0xd4, 0xc1, 0x81, 0x0d, 0x0f, 0x18,
0xe3, 0x92, 0xe6, 0xb4, 0xb6, 0x58, 0xdf, 0x6d, 0x60, 0xc3, 0x06, 0xfb, 0x14, 0x82, 0x4c, 0xb4,
0xcb, 0x82, 0x5a, 0x4a, 0xf5, 0xef, 0xa6, 0x0e, 0xf6, 0x4d, 0xd4, 0x40, 0xe7, 0x70, 0x30, 0x8c,
0x95, 0xe5, 0x40, 0xdf, 0xe9, 0xdb, 0xa6, 0x5f, 0xf4, 0x5c, 0xea, 0xe0, 0xfd, 0x41, 0x64, 0xca,
0x7c, 0x0d, 0xde, 0x9a, 0x76, 0xb6, 0xc0, 0x44, 0x17, 0x08, 0xdf, 0x75, 0xaf, 0xa9, 0x83, 0xf7,
0xd6, 0xb4, 0x1b, 0x4c, 0x36, 0xb2, 0x66, 0x3c, 0xb7, 0xda, 0xf7, 0xec, 0x25, 0xf9, 0x26, 0x6a,
0xa0, 0x63, 0x80, 0x65, 0x21, 0x96, 0x16, 0x41, 0x91, 0x3b, 0x0d, 0xd4, 0xc1, 0xa9, 0x98, 0x01,
0xbe, 0x83, 0x83, 0x9c, 0x8a, 0x45, 0x25, 0x18, 0x97, 0x96, 0xda, 0xd3, 0x26, 0x0e, 0x7b, 0x13,
0xea, 0xa2, 0xe3, 0xe7, 0x44, 0x3e, 0xe7, 0x79, 0xea, 0xe0, 0xfb, 0x39, 0x15, 0x73, 0x05, 0x1b,
0xf9, 0x53, 0x08, 0xcc, 0x53, 0xb6, 0xda, 0x5d, 0xad, 0xfd, 0x70, 0x6b, 0x03, 0xe7, 0x1a, 0x54,
0x0e, 0x8d, 0xc4, 0x54, 0x98, 0x81, 0x4f, 0xd4, 0x08, 0xd9, 0x02, 0x9e, 0x2e, 0x70, 0xbc, 0xb5,
0xc0, 0x66, 0xd4, 0x52, 0x07, 0x03, 0xd9, 0x0c, 0x5e, 0x08, 0xf7, 0x4a, 0x4a, 0x38, 0xe3, 0x79,
0xb8, 0x1f, 0xb9, 0xd3, 0x09, 0xee, 0x97, 0xe8, 0x11, 0x3c, 0xa4, 0xaf, 0x57, 0x45, 0x9b, 0xd1,
0xc5, 0xcb, 0x5a, 0x94, 0x0b, 0xc6, 0x33, 0xfa, 0x9a, 0x36, 0xe1, 0xa1, 0x1a, 0x0f, 0x8c, 0x6c,
0xee, 0xc7, 0x5a, 0x94, 0x17, 0x26, 0x33, 0x0b, 0x00, 0xb4, 0x13, 0x33, 0xe0, 0xff, 0xba, 0xb0,
0x6b, 0x7c, 0xa3, 0x2f, 0x60, 0xbc, 0xa6, 0x9d, 0x7d, 0xb7, 0xef, 0xbc, 0x22, 0xac, 0x20, 0x74,
0xa9, 0x7f, 0x1b, 0x15, 0xad, 0x25, 0xa3, 0x4d, 0x38, 0xd6, 0xaf, 0xe1, 0xcb, 0x3b, 0x0e, 0x25,
0x9e, 0x0f, 0xf4, 0x39, 0x97, 0x75, 0x87, 0x6f, 0xc9, 0x8f, 0x7e, 0x85, 0x83, 0x37, 0xd2, 0xe8,
0xc1, 0xc6, 0x8b, 0x67, 0x76, 0x7c, 0x04, 0x93, 0xcd, 0x44, 0xdf, 0xfd, 0xf4, 0x0c, 0xf8, 0xcd,
0xe8, 0xb1, 0x3b, 0xfb, 0xd3, 0x85, 0xf7, 0x57, 0xa2, 0xdc, 0x06, 0xcf, 0x7c, 0x63, 0x6d, 0xae,
0x86, 0x78, 0xee, 0xfe, 0xf6, 0xad, 0x65, 0x72, 0x51, 0x10, 0x9e, 0xc7, 0xa2, 0xce, 0x93, 0x9c,
0x72, 0x3d, 0xe2, 0x89, 0x49, 0x91, 0x8a, 0x35, 0xff, 0xfb, 0xcb, 0x3f, 0x19, 0x16, 0x7f, 0x8d,
0x3e, 0xf8, 0xc9, 0xc8, 0x9f, 0x15, 0xa2, 0xcd, 0xe2, 0x1f, 0x86, 0x8d, 0xae, 0x4e, 0xff, 0xee,
0x73, 0xd7, 0x3a, 0x77, 0x3d, 0xe4, 0xae, 0xaf, 0x4e, 0x97, 0xbb, 0x7a, 0x83, 0xaf, 0xfe, 0x0b,
0x00, 0x00, 0xff, 0xff, 0xf3, 0xdd, 0x11, 0x96, 0x45, 0x06, 0x00, 0x00,
}
var xxx_messageInfo_Key proto.InternalMessageInfo
appengine-1.6.7/datastore/key.go 0000664 0000000 0000000 00000025304 13741174344 0016614 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"bytes"
"encoding/base64"
"encoding/gob"
"errors"
"fmt"
"strconv"
"strings"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
pb "google.golang.org/appengine/internal/datastore"
)
type KeyRangeCollisionError struct {
start int64
end int64
}
func (e *KeyRangeCollisionError) Error() string {
return fmt.Sprintf("datastore: Collision when attempting to allocate range [%d, %d]",
e.start, e.end)
}
type KeyRangeContentionError struct {
start int64
end int64
}
func (e *KeyRangeContentionError) Error() string {
return fmt.Sprintf("datastore: Contention when attempting to allocate range [%d, %d]",
e.start, e.end)
}
// Key represents the datastore key for a stored entity, and is immutable.
type Key struct {
kind string
stringID string
intID int64
parent *Key
appID string
namespace string
}
// Kind returns the key's kind (also known as entity type).
func (k *Key) Kind() string {
return k.kind
}
// StringID returns the key's string ID (also known as an entity name or key
// name), which may be "".
func (k *Key) StringID() string {
return k.stringID
}
// IntID returns the key's integer ID, which may be 0.
func (k *Key) IntID() int64 {
return k.intID
}
// Parent returns the key's parent key, which may be nil.
func (k *Key) Parent() *Key {
return k.parent
}
// AppID returns the key's application ID.
func (k *Key) AppID() string {
return k.appID
}
// Namespace returns the key's namespace.
func (k *Key) Namespace() string {
return k.namespace
}
// Incomplete returns whether the key does not refer to a stored entity.
// In particular, whether the key has a zero StringID and a zero IntID.
func (k *Key) Incomplete() bool {
return k.stringID == "" && k.intID == 0
}
// valid returns whether the key is valid.
func (k *Key) valid() bool {
if k == nil {
return false
}
for ; k != nil; k = k.parent {
if k.kind == "" || k.appID == "" {
return false
}
if k.stringID != "" && k.intID != 0 {
return false
}
if k.parent != nil {
if k.parent.Incomplete() {
return false
}
if k.parent.appID != k.appID || k.parent.namespace != k.namespace {
return false
}
}
}
return true
}
// Equal returns whether two keys are equal.
func (k *Key) Equal(o *Key) bool {
for k != nil && o != nil {
if k.kind != o.kind || k.stringID != o.stringID || k.intID != o.intID || k.appID != o.appID || k.namespace != o.namespace {
return false
}
k, o = k.parent, o.parent
}
return k == o
}
// root returns the furthest ancestor of a key, which may be itself.
func (k *Key) root() *Key {
for k.parent != nil {
k = k.parent
}
return k
}
// marshal marshals the key's string representation to the buffer.
func (k *Key) marshal(b *bytes.Buffer) {
if k.parent != nil {
k.parent.marshal(b)
}
b.WriteByte('/')
b.WriteString(k.kind)
b.WriteByte(',')
if k.stringID != "" {
b.WriteString(k.stringID)
} else {
b.WriteString(strconv.FormatInt(k.intID, 10))
}
}
// String returns a string representation of the key.
func (k *Key) String() string {
if k == nil {
return ""
}
b := bytes.NewBuffer(make([]byte, 0, 512))
k.marshal(b)
return b.String()
}
type gobKey struct {
Kind string
StringID string
IntID int64
Parent *gobKey
AppID string
Namespace string
}
func keyToGobKey(k *Key) *gobKey {
if k == nil {
return nil
}
return &gobKey{
Kind: k.kind,
StringID: k.stringID,
IntID: k.intID,
Parent: keyToGobKey(k.parent),
AppID: k.appID,
Namespace: k.namespace,
}
}
func gobKeyToKey(gk *gobKey) *Key {
if gk == nil {
return nil
}
return &Key{
kind: gk.Kind,
stringID: gk.StringID,
intID: gk.IntID,
parent: gobKeyToKey(gk.Parent),
appID: gk.AppID,
namespace: gk.Namespace,
}
}
func (k *Key) GobEncode() ([]byte, error) {
buf := new(bytes.Buffer)
if err := gob.NewEncoder(buf).Encode(keyToGobKey(k)); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (k *Key) GobDecode(buf []byte) error {
gk := new(gobKey)
if err := gob.NewDecoder(bytes.NewBuffer(buf)).Decode(gk); err != nil {
return err
}
*k = *gobKeyToKey(gk)
return nil
}
func (k *Key) MarshalJSON() ([]byte, error) {
return []byte(`"` + k.Encode() + `"`), nil
}
func (k *Key) UnmarshalJSON(buf []byte) error {
if len(buf) < 2 || buf[0] != '"' || buf[len(buf)-1] != '"' {
return errors.New("datastore: bad JSON key")
}
k2, err := DecodeKey(string(buf[1 : len(buf)-1]))
if err != nil {
return err
}
*k = *k2
return nil
}
// Encode returns an opaque representation of the key
// suitable for use in HTML and URLs.
// This is compatible with the Python and Java runtimes.
func (k *Key) Encode() string {
ref := keyToProto("", k)
b, err := proto.Marshal(ref)
if err != nil {
panic(err)
}
// Trailing padding is stripped.
return strings.TrimRight(base64.URLEncoding.EncodeToString(b), "=")
}
// DecodeKey decodes a key from the opaque representation returned by Encode.
func DecodeKey(encoded string) (*Key, error) {
// Re-add padding.
if m := len(encoded) % 4; m != 0 {
encoded += strings.Repeat("=", 4-m)
}
b, err := base64.URLEncoding.DecodeString(encoded)
if err != nil {
return nil, err
}
ref := new(pb.Reference)
if err := proto.Unmarshal(b, ref); err != nil {
// Couldn't decode it as an App Engine key, try decoding it as a key encoded by cloud.google.com/go/datastore.
if k := decodeCloudKey(encoded); k != nil {
return k, nil
}
return nil, err
}
return protoToKey(ref)
}
// NewIncompleteKey creates a new incomplete key.
// kind cannot be empty.
func NewIncompleteKey(c context.Context, kind string, parent *Key) *Key {
return NewKey(c, kind, "", 0, parent)
}
// NewKey creates a new key.
// kind cannot be empty.
// Either one or both of stringID and intID must be zero. If both are zero,
// the key returned is incomplete.
// parent must either be a complete key or nil.
func NewKey(c context.Context, kind, stringID string, intID int64, parent *Key) *Key {
// If there's a parent key, use its namespace.
// Otherwise, use any namespace attached to the context.
var namespace string
if parent != nil {
namespace = parent.namespace
} else {
namespace = internal.NamespaceFromContext(c)
}
return &Key{
kind: kind,
stringID: stringID,
intID: intID,
parent: parent,
appID: internal.FullyQualifiedAppID(c),
namespace: namespace,
}
}
// AllocateIDs returns a range of n integer IDs with the given kind and parent
// combination. kind cannot be empty; parent may be nil. The IDs in the range
// returned will not be used by the datastore's automatic ID sequence generator
// and may be used with NewKey without conflict.
//
// The range is inclusive at the low end and exclusive at the high end. In
// other words, valid intIDs x satisfy low <= x && x < high.
//
// If no error is returned, low + n == high.
func AllocateIDs(c context.Context, kind string, parent *Key, n int) (low, high int64, err error) {
if kind == "" {
return 0, 0, errors.New("datastore: AllocateIDs given an empty kind")
}
if n < 0 {
return 0, 0, fmt.Errorf("datastore: AllocateIDs given a negative count: %d", n)
}
if n == 0 {
return 0, 0, nil
}
req := &pb.AllocateIdsRequest{
ModelKey: keyToProto("", NewIncompleteKey(c, kind, parent)),
Size: proto.Int64(int64(n)),
}
res := &pb.AllocateIdsResponse{}
if err := internal.Call(c, "datastore_v3", "AllocateIds", req, res); err != nil {
return 0, 0, err
}
// The protobuf is inclusive at both ends. Idiomatic Go (e.g. slices, for loops)
// is inclusive at the low end and exclusive at the high end, so we add 1.
low = res.GetStart()
high = res.GetEnd() + 1
if low+int64(n) != high {
return 0, 0, fmt.Errorf("datastore: internal error: could not allocate %d IDs", n)
}
return low, high, nil
}
// AllocateIDRange allocates a range of IDs with specific endpoints.
// The range is inclusive at both the low and high end. Once these IDs have been
// allocated, you can manually assign them to newly created entities.
//
// The Datastore's automatic ID allocator never assigns a key that has already
// been allocated (either through automatic ID allocation or through an explicit
// AllocateIDs call). As a result, entities written to the given key range will
// never be overwritten. However, writing entities with manually assigned keys in
// this range may overwrite existing entities (or new entities written by a separate
// request), depending on the error returned.
//
// Use this only if you have an existing numeric ID range that you want to reserve
// (for example, bulk loading entities that already have IDs). If you don't care
// about which IDs you receive, use AllocateIDs instead.
//
// AllocateIDRange returns nil if the range is successfully allocated. If one or more
// entities with an ID in the given range already exist, it returns a KeyRangeCollisionError.
// If the Datastore has already cached IDs in this range (e.g. from a previous call to
// AllocateIDRange), it returns a KeyRangeContentionError. Errors of other types indicate
// problems with arguments or an error returned directly from the Datastore.
func AllocateIDRange(c context.Context, kind string, parent *Key, start, end int64) (err error) {
if kind == "" {
return errors.New("datastore: AllocateIDRange given an empty kind")
}
if start < 1 || end < 1 {
return errors.New("datastore: AllocateIDRange start and end must both be greater than 0")
}
if start > end {
return errors.New("datastore: AllocateIDRange start must be before end")
}
req := &pb.AllocateIdsRequest{
ModelKey: keyToProto("", NewIncompleteKey(c, kind, parent)),
Max: proto.Int64(end),
}
res := &pb.AllocateIdsResponse{}
if err := internal.Call(c, "datastore_v3", "AllocateIds", req, res); err != nil {
return err
}
// Check for collisions, i.e. existing entities with IDs in this range.
// We could do this before the allocation, but we'd still have to do it
// afterward as well to catch the race condition where an entity is inserted
// after that initial check but before the allocation. Skip the up-front check
// and just do it once.
q := NewQuery(kind).Filter("__key__ >=", NewKey(c, kind, "", start, parent)).
Filter("__key__ <=", NewKey(c, kind, "", end, parent)).KeysOnly().Limit(1)
keys, err := q.GetAll(c, nil)
if err != nil {
return err
}
if len(keys) != 0 {
return &KeyRangeCollisionError{start: start, end: end}
}
// Check for a race condition, i.e. cases where the datastore may have
// cached ID batches that contain IDs in this range.
if start < res.GetStart() {
return &KeyRangeContentionError{start: start, end: end}
}
return nil
}
appengine-1.6.7/datastore/key_test.go 0000664 0000000 0000000 00000010214 13741174344 0017645 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"bytes"
"encoding/gob"
"encoding/json"
"testing"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
)
func TestKeyEncoding(t *testing.T) {
testCases := []struct {
desc string
key *Key
exp string
}{
{
desc: "A simple key with an int ID",
key: &Key{
kind: "Person",
intID: 1,
appID: "glibrary",
},
exp: "aghnbGlicmFyeXIMCxIGUGVyc29uGAEM",
},
{
desc: "A simple key with a string ID",
key: &Key{
kind: "Graph",
stringID: "graph:7-day-active",
appID: "glibrary",
},
exp: "aghnbGlicmFyeXIdCxIFR3JhcGgiEmdyYXBoOjctZGF5LWFjdGl2ZQw",
},
{
desc: "A key with a parent",
key: &Key{
kind: "WordIndex",
intID: 1033,
parent: &Key{
kind: "WordIndex",
intID: 1020032,
appID: "glibrary",
},
appID: "glibrary",
},
exp: "aghnbGlicmFyeXIhCxIJV29yZEluZGV4GIChPgwLEglXb3JkSW5kZXgYiQgM",
},
}
for _, tc := range testCases {
enc := tc.key.Encode()
if enc != tc.exp {
t.Errorf("%s: got %q, want %q", tc.desc, enc, tc.exp)
}
key, err := DecodeKey(tc.exp)
if err != nil {
t.Errorf("%s: failed decoding key: %v", tc.desc, err)
continue
}
if !key.Equal(tc.key) {
t.Errorf("%s: decoded key %v, want %v", tc.desc, key, tc.key)
}
}
}
func TestKeyGob(t *testing.T) {
k := &Key{
kind: "Gopher",
intID: 3,
parent: &Key{
kind: "Mom",
stringID: "narwhal",
appID: "gopher-con",
},
appID: "gopher-con",
}
buf := new(bytes.Buffer)
if err := gob.NewEncoder(buf).Encode(k); err != nil {
t.Fatalf("gob encode failed: %v", err)
}
k2 := new(Key)
if err := gob.NewDecoder(buf).Decode(k2); err != nil {
t.Fatalf("gob decode failed: %v", err)
}
if !k2.Equal(k) {
t.Errorf("gob round trip of %v produced %v", k, k2)
}
}
func TestNilKeyGob(t *testing.T) {
type S struct {
Key *Key
}
s1 := new(S)
buf := new(bytes.Buffer)
if err := gob.NewEncoder(buf).Encode(s1); err != nil {
t.Fatalf("gob encode failed: %v", err)
}
s2 := new(S)
if err := gob.NewDecoder(buf).Decode(s2); err != nil {
t.Fatalf("gob decode failed: %v", err)
}
if s2.Key != nil {
t.Errorf("gob round trip of nil key produced %v", s2.Key)
}
}
func TestKeyJSON(t *testing.T) {
k := &Key{
kind: "Gopher",
intID: 2,
parent: &Key{
kind: "Mom",
stringID: "narwhal",
appID: "gopher-con",
},
appID: "gopher-con",
}
exp := `"` + k.Encode() + `"`
buf, err := json.Marshal(k)
if err != nil {
t.Fatalf("json.Marshal failed: %v", err)
}
if s := string(buf); s != exp {
t.Errorf("JSON encoding of key %v: got %q, want %q", k, s, exp)
}
k2 := new(Key)
if err := json.Unmarshal(buf, k2); err != nil {
t.Fatalf("json.Unmarshal failed: %v", err)
}
if !k2.Equal(k) {
t.Errorf("JSON round trip of %v produced %v", k, k2)
}
}
func TestNilKeyJSON(t *testing.T) {
type S struct {
Key *Key
}
s1 := new(S)
buf, err := json.Marshal(s1)
if err != nil {
t.Fatalf("json.Marshal failed: %v", err)
}
s2 := new(S)
if err := json.Unmarshal(buf, s2); err != nil {
t.Fatalf("json.Unmarshal failed: %v", err)
}
if s2.Key != nil {
t.Errorf("JSON round trip of nil key produced %v", s2.Key)
}
}
func TestIncompleteKeyWithParent(t *testing.T) {
c := internal.WithAppIDOverride(context.Background(), "s~some-app")
// fadduh is a complete key.
fadduh := NewKey(c, "Person", "", 1, nil)
if fadduh.Incomplete() {
t.Fatalf("fadduh is incomplete")
}
// robert is an incomplete key with fadduh as a parent.
robert := NewIncompleteKey(c, "Person", fadduh)
if !robert.Incomplete() {
t.Fatalf("robert is complete")
}
// Both should be valid keys.
if !fadduh.valid() {
t.Errorf("fadduh is invalid: %v", fadduh)
}
if !robert.valid() {
t.Errorf("robert is invalid: %v", robert)
}
}
func TestNamespace(t *testing.T) {
key := &Key{
kind: "Person",
intID: 1,
appID: "s~some-app",
namespace: "mynamespace",
}
if g, w := key.Namespace(), "mynamespace"; g != w {
t.Errorf("key.Namespace() = %q, want %q", g, w)
}
}
appengine-1.6.7/datastore/keycompat.go 0000664 0000000 0000000 00000005036 13741174344 0020020 0 ustar 00root root 0000000 0000000 // Copyright 2019 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"sync"
"golang.org/x/net/context"
"google.golang.org/appengine/datastore/internal/cloudkey"
"google.golang.org/appengine/internal"
)
var keyConversion struct {
mu sync.RWMutex
appID string // read using getKeyConversionAppID
}
// EnableKeyConversion enables encoded key compatibility with the Cloud
// Datastore client library (cloud.google.com/go/datastore). Encoded keys
// generated by the Cloud Datastore client library will be decoded into App
// Engine datastore keys.
//
// The context provided must be an App Engine context if running in App Engine
// first generation runtime. This can be called in the /_ah/start handler. It is
// safe to call multiple times, and is cheap to call, so can also be inserted as
// middleware.
//
// Enabling key compatibility does not affect the encoding format used by
// Key.Encode, it only expands the type of keys that are able to be decoded with
// DecodeKey.
func EnableKeyConversion(ctx context.Context) {
// Only attempt to set appID if it's unset.
// If already set, ignore.
if getKeyConversionAppID() != "" {
return
}
keyConversion.mu.Lock()
// Check again to avoid race where another goroutine set appID between the call
// to getKeyConversionAppID above and taking the write lock.
if keyConversion.appID == "" {
keyConversion.appID = internal.FullyQualifiedAppID(ctx)
}
keyConversion.mu.Unlock()
}
func getKeyConversionAppID() string {
keyConversion.mu.RLock()
appID := keyConversion.appID
keyConversion.mu.RUnlock()
return appID
}
// decodeCloudKey attempts to decode the given encoded key generated by the
// Cloud Datastore client library (cloud.google.com/go/datastore), returning nil
// if the key couldn't be decoded.
func decodeCloudKey(encoded string) *Key {
appID := getKeyConversionAppID()
if appID == "" {
return nil
}
k, err := cloudkey.DecodeKey(encoded)
if err != nil {
return nil
}
return convertCloudKey(k, appID)
}
// convertCloudKey converts a Cloud Datastore key and converts it to an App
// Engine Datastore key. Cloud Datastore keys don't include the project/app ID,
// so we must add it back in.
func convertCloudKey(key *cloudkey.Key, appID string) *Key {
if key == nil {
return nil
}
k := &Key{
intID: key.ID,
kind: key.Kind,
namespace: key.Namespace,
parent: convertCloudKey(key.Parent, appID),
stringID: key.Name,
appID: appID,
}
return k
}
appengine-1.6.7/datastore/keycompat_test.go 0000664 0000000 0000000 00000004226 13741174344 0021057 0 ustar 00root root 0000000 0000000 // Copyright 2019 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"reflect"
"testing"
)
func TestKeyConversion(t *testing.T) {
var tests = []struct {
desc string
key *Key
encodedKey string
}{
{
desc: "A control test for legacy to legacy key conversion int as the key",
key: &Key{
kind: "Person",
intID: 1,
appID: "glibrary",
},
encodedKey: "aghnbGlicmFyeXIMCxIGUGVyc29uGAEM",
},
{
desc: "A control test for legacy to legacy key conversion string as the key",
key: &Key{
kind: "Graph",
stringID: "graph:7-day-active",
appID: "glibrary",
},
encodedKey: "aghnbGlicmFyeXIdCxIFR3JhcGgiEmdyYXBoOjctZGF5LWFjdGl2ZQw",
},
// These are keys encoded with cloud.google.com/go/datastore
// Standard int as the key
{
desc: "Convert new key format to old key with int id",
key: &Key{
kind: "WordIndex",
intID: 1033,
appID: "glibrary",
},
encodedKey: "Eg4KCVdvcmRJbmRleBCJCA",
},
// These are keys encoded with cloud.google.com/go/datastore
// Standard string
{
desc: "Convert new key format to old key with string id",
key: &Key{
kind: "WordIndex",
stringID: "IAmAnID",
appID: "glibrary",
},
encodedKey: "EhQKCVdvcmRJbmRleBoHSUFtQW5JRA",
},
// These are keys encoded with cloud.google.com/go/datastore
// ID String with parent as string
{
desc: "Convert new key format to old key with string id with a parent",
key: &Key{
kind: "WordIndex",
stringID: "IAmAnID",
appID: "glibrary",
parent: &Key{
kind: "LetterIndex",
stringID: "IAmAnotherID",
appID: "glibrary",
},
},
encodedKey: "EhsKC0xldHRlckluZGV4GgxJQW1Bbm90aGVySUQSFAoJV29yZEluZGV4GgdJQW1BbklE",
},
}
// Simulate the key converter enablement
keyConversion.appID = "glibrary"
for _, tc := range tests {
dk, err := DecodeKey(tc.encodedKey)
if err != nil {
t.Fatalf("DecodeKey: %v", err)
}
if !reflect.DeepEqual(dk, tc.key) {
t.Errorf("%s: got %+v, want %+v", tc.desc, dk, tc.key)
}
}
}
appengine-1.6.7/datastore/load.go 0000664 0000000 0000000 00000026213 13741174344 0016743 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"fmt"
"reflect"
"strings"
"time"
"github.com/golang/protobuf/proto"
"google.golang.org/appengine"
pb "google.golang.org/appengine/internal/datastore"
)
var (
typeOfBlobKey = reflect.TypeOf(appengine.BlobKey(""))
typeOfByteSlice = reflect.TypeOf([]byte(nil))
typeOfByteString = reflect.TypeOf(ByteString(nil))
typeOfGeoPoint = reflect.TypeOf(appengine.GeoPoint{})
typeOfTime = reflect.TypeOf(time.Time{})
typeOfKeyPtr = reflect.TypeOf(&Key{})
typeOfEntityPtr = reflect.TypeOf(&Entity{})
)
// typeMismatchReason returns a string explaining why the property p could not
// be stored in an entity field of type v.Type().
func typeMismatchReason(pValue interface{}, v reflect.Value) string {
entityType := "empty"
switch pValue.(type) {
case int64:
entityType = "int"
case bool:
entityType = "bool"
case string:
entityType = "string"
case float64:
entityType = "float"
case *Key:
entityType = "*datastore.Key"
case time.Time:
entityType = "time.Time"
case appengine.BlobKey:
entityType = "appengine.BlobKey"
case appengine.GeoPoint:
entityType = "appengine.GeoPoint"
case ByteString:
entityType = "datastore.ByteString"
case []byte:
entityType = "[]byte"
}
return fmt.Sprintf("type mismatch: %s versus %v", entityType, v.Type())
}
type propertyLoader struct {
// m holds the number of times a substruct field like "Foo.Bar.Baz" has
// been seen so far. The map is constructed lazily.
m map[string]int
}
func (l *propertyLoader) load(codec *structCodec, structValue reflect.Value, p Property, requireSlice bool) string {
var v reflect.Value
var sliceIndex int
name := p.Name
// If name ends with a '.', the last field is anonymous.
// In this case, strings.Split will give us "" as the
// last element of our fields slice, which will match the ""
// field name in the substruct codec.
fields := strings.Split(name, ".")
for len(fields) > 0 {
var decoder fieldCodec
var ok bool
// Cut off the last field (delimited by ".") and find its parent
// in the codec.
// eg. for name "A.B.C.D", split off "A.B.C" and try to
// find a field in the codec with this name.
// Loop again with "A.B", etc.
for i := len(fields); i > 0; i-- {
parent := strings.Join(fields[:i], ".")
decoder, ok = codec.fields[parent]
if ok {
fields = fields[i:]
break
}
}
// If we never found a matching field in the codec, return
// error message.
if !ok {
return "no such struct field"
}
v = initField(structValue, decoder.path)
if !v.IsValid() {
return "no such struct field"
}
if !v.CanSet() {
return "cannot set struct field"
}
if decoder.structCodec != nil {
codec = decoder.structCodec
structValue = v
}
if v.Kind() == reflect.Slice && v.Type() != typeOfByteSlice {
if l.m == nil {
l.m = make(map[string]int)
}
sliceIndex = l.m[p.Name]
l.m[p.Name] = sliceIndex + 1
for v.Len() <= sliceIndex {
v.Set(reflect.Append(v, reflect.New(v.Type().Elem()).Elem()))
}
structValue = v.Index(sliceIndex)
requireSlice = false
}
}
var slice reflect.Value
if v.Kind() == reflect.Slice && v.Type().Elem().Kind() != reflect.Uint8 {
slice = v
v = reflect.New(v.Type().Elem()).Elem()
} else if requireSlice {
return "multiple-valued property requires a slice field type"
}
// Convert indexValues to a Go value with a meaning derived from the
// destination type.
pValue := p.Value
if iv, ok := pValue.(indexValue); ok {
meaning := pb.Property_NO_MEANING
switch v.Type() {
case typeOfBlobKey:
meaning = pb.Property_BLOBKEY
case typeOfByteSlice:
meaning = pb.Property_BLOB
case typeOfByteString:
meaning = pb.Property_BYTESTRING
case typeOfGeoPoint:
meaning = pb.Property_GEORSS_POINT
case typeOfTime:
meaning = pb.Property_GD_WHEN
case typeOfEntityPtr:
meaning = pb.Property_ENTITY_PROTO
}
var err error
pValue, err = propValue(iv.value, meaning)
if err != nil {
return err.Error()
}
}
if errReason := setVal(v, pValue); errReason != "" {
// Set the slice back to its zero value.
if slice.IsValid() {
slice.Set(reflect.Zero(slice.Type()))
}
return errReason
}
if slice.IsValid() {
slice.Index(sliceIndex).Set(v)
}
return ""
}
// setVal sets v to the value pValue.
func setVal(v reflect.Value, pValue interface{}) string {
switch v.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
x, ok := pValue.(int64)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
if v.OverflowInt(x) {
return fmt.Sprintf("value %v overflows struct field of type %v", x, v.Type())
}
v.SetInt(x)
case reflect.Bool:
x, ok := pValue.(bool)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
v.SetBool(x)
case reflect.String:
switch x := pValue.(type) {
case appengine.BlobKey:
v.SetString(string(x))
case ByteString:
v.SetString(string(x))
case string:
v.SetString(x)
default:
if pValue != nil {
return typeMismatchReason(pValue, v)
}
}
case reflect.Float32, reflect.Float64:
x, ok := pValue.(float64)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
if v.OverflowFloat(x) {
return fmt.Sprintf("value %v overflows struct field of type %v", x, v.Type())
}
v.SetFloat(x)
case reflect.Ptr:
x, ok := pValue.(*Key)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
if _, ok := v.Interface().(*Key); !ok {
return typeMismatchReason(pValue, v)
}
v.Set(reflect.ValueOf(x))
case reflect.Struct:
switch v.Type() {
case typeOfTime:
x, ok := pValue.(time.Time)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
v.Set(reflect.ValueOf(x))
case typeOfGeoPoint:
x, ok := pValue.(appengine.GeoPoint)
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
v.Set(reflect.ValueOf(x))
default:
ent, ok := pValue.(*Entity)
if !ok {
return typeMismatchReason(pValue, v)
}
// Recursively load nested struct
pls, err := newStructPLS(v.Addr().Interface())
if err != nil {
return err.Error()
}
// if ent has a Key value and our struct has a Key field,
// load the Entity's Key value into the Key field on the struct.
if ent.Key != nil && pls.codec.keyField != -1 {
pls.v.Field(pls.codec.keyField).Set(reflect.ValueOf(ent.Key))
}
err = pls.Load(ent.Properties)
if err != nil {
return err.Error()
}
}
case reflect.Slice:
x, ok := pValue.([]byte)
if !ok {
if y, yok := pValue.(ByteString); yok {
x, ok = []byte(y), true
}
}
if !ok && pValue != nil {
return typeMismatchReason(pValue, v)
}
if v.Type().Elem().Kind() != reflect.Uint8 {
return typeMismatchReason(pValue, v)
}
v.SetBytes(x)
default:
return typeMismatchReason(pValue, v)
}
return ""
}
// initField is similar to reflect's Value.FieldByIndex, in that it
// returns the nested struct field corresponding to index, but it
// initialises any nil pointers encountered when traversing the structure.
func initField(val reflect.Value, index []int) reflect.Value {
for _, i := range index[:len(index)-1] {
val = val.Field(i)
if val.Kind() == reflect.Ptr {
if val.IsNil() {
val.Set(reflect.New(val.Type().Elem()))
}
val = val.Elem()
}
}
return val.Field(index[len(index)-1])
}
// loadEntity loads an EntityProto into PropertyLoadSaver or struct pointer.
func loadEntity(dst interface{}, src *pb.EntityProto) (err error) {
ent, err := protoToEntity(src)
if err != nil {
return err
}
if e, ok := dst.(PropertyLoadSaver); ok {
return e.Load(ent.Properties)
}
return LoadStruct(dst, ent.Properties)
}
func (s structPLS) Load(props []Property) error {
var fieldName, reason string
var l propertyLoader
for _, p := range props {
if errStr := l.load(s.codec, s.v, p, p.Multiple); errStr != "" {
// We don't return early, as we try to load as many properties as possible.
// It is valid to load an entity into a struct that cannot fully represent it.
// That case returns an error, but the caller is free to ignore it.
fieldName, reason = p.Name, errStr
}
}
if reason != "" {
return &ErrFieldMismatch{
StructType: s.v.Type(),
FieldName: fieldName,
Reason: reason,
}
}
return nil
}
func protoToEntity(src *pb.EntityProto) (*Entity, error) {
props, rawProps := src.Property, src.RawProperty
outProps := make([]Property, 0, len(props)+len(rawProps))
for {
var (
x *pb.Property
noIndex bool
)
if len(props) > 0 {
x, props = props[0], props[1:]
} else if len(rawProps) > 0 {
x, rawProps = rawProps[0], rawProps[1:]
noIndex = true
} else {
break
}
var value interface{}
if x.Meaning != nil && *x.Meaning == pb.Property_INDEX_VALUE {
value = indexValue{x.Value}
} else {
var err error
value, err = propValue(x.Value, x.GetMeaning())
if err != nil {
return nil, err
}
}
outProps = append(outProps, Property{
Name: x.GetName(),
Value: value,
NoIndex: noIndex,
Multiple: x.GetMultiple(),
})
}
var key *Key
if src.Key != nil {
// Ignore any error, since nested entity values
// are allowed to have an invalid key.
key, _ = protoToKey(src.Key)
}
return &Entity{key, outProps}, nil
}
// propValue returns a Go value that combines the raw PropertyValue with a
// meaning. For example, an Int64Value with GD_WHEN becomes a time.Time.
func propValue(v *pb.PropertyValue, m pb.Property_Meaning) (interface{}, error) {
switch {
case v.Int64Value != nil:
if m == pb.Property_GD_WHEN {
return fromUnixMicro(*v.Int64Value), nil
} else {
return *v.Int64Value, nil
}
case v.BooleanValue != nil:
return *v.BooleanValue, nil
case v.StringValue != nil:
if m == pb.Property_BLOB {
return []byte(*v.StringValue), nil
} else if m == pb.Property_BLOBKEY {
return appengine.BlobKey(*v.StringValue), nil
} else if m == pb.Property_BYTESTRING {
return ByteString(*v.StringValue), nil
} else if m == pb.Property_ENTITY_PROTO {
var ent pb.EntityProto
err := proto.Unmarshal([]byte(*v.StringValue), &ent)
if err != nil {
return nil, err
}
return protoToEntity(&ent)
} else {
return *v.StringValue, nil
}
case v.DoubleValue != nil:
return *v.DoubleValue, nil
case v.Referencevalue != nil:
key, err := referenceValueToKey(v.Referencevalue)
if err != nil {
return nil, err
}
return key, nil
case v.Pointvalue != nil:
// NOTE: Strangely, latitude maps to X, longitude to Y.
return appengine.GeoPoint{Lat: v.Pointvalue.GetX(), Lng: v.Pointvalue.GetY()}, nil
}
return nil, nil
}
// indexValue is a Property value that is created when entities are loaded from
// an index, such as from a projection query.
//
// Such Property values do not contain all of the metadata required to be
// faithfully represented as a Go value, and are instead represented as an
// opaque indexValue. Load the properties into a concrete struct type (e.g. by
// passing a struct pointer to Iterator.Next) to reconstruct actual Go values
// of type int, string, time.Time, etc.
type indexValue struct {
value *pb.PropertyValue
}
appengine-1.6.7/datastore/load_test.go 0000664 0000000 0000000 00000032005 13741174344 0017776 0 ustar 00root root 0000000 0000000 // Copyright 2016 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"reflect"
"testing"
proto "github.com/golang/protobuf/proto"
pb "google.golang.org/appengine/internal/datastore"
)
type Simple struct {
I int64
}
type SimpleWithTag struct {
I int64 `datastore:"II"`
}
type NestedSimpleWithTag struct {
A SimpleWithTag `datastore:"AA"`
}
type NestedSliceOfSimple struct {
A []Simple
}
type SimpleTwoFields struct {
S string
SS string
}
type NestedSimpleAnonymous struct {
Simple
X string
}
type NestedSimple struct {
A Simple
I int64
}
type NestedSimple1 struct {
A Simple
X string
}
type NestedSimple2X struct {
AA NestedSimple
A SimpleTwoFields
S string
}
type BDotB struct {
B string `datastore:"B.B"`
}
type ABDotB struct {
A BDotB
}
type MultiAnonymous struct {
Simple
SimpleTwoFields
X string
}
var (
// these values need to be addressable
testString2 = "two"
testString3 = "three"
testInt64 = int64(2)
fieldNameI = "I"
fieldNameX = "X"
fieldNameS = "S"
fieldNameSS = "SS"
fieldNameADotI = "A.I"
fieldNameAADotII = "AA.II"
fieldNameADotBDotB = "A.B.B"
)
func TestLoadEntityNestedLegacy(t *testing.T) {
testCases := []struct {
desc string
src *pb.EntityProto
want interface{}
}{
{
"nested",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
&pb.Property{
Name: &fieldNameADotI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
},
},
&NestedSimple1{
A: Simple{I: testInt64},
X: testString2,
},
},
{
"nested with tag",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameAADotII,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
},
},
&NestedSimpleWithTag{
A: SimpleWithTag{I: testInt64},
},
},
{
"nested with anonymous struct field",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
},
},
&NestedSimpleAnonymous{
Simple: Simple{I: testInt64},
X: testString2,
},
},
{
"nested with dotted field tag",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameADotBDotB,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
},
},
&ABDotB{
A: BDotB{
B: testString2,
},
},
},
{
"nested with dotted field tag",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
&pb.Property{
Name: &fieldNameS,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
&pb.Property{
Name: &fieldNameSS,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
},
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
},
},
},
&MultiAnonymous{
Simple: Simple{I: testInt64},
SimpleTwoFields: SimpleTwoFields{S: "two", SS: "three"},
X: "three",
},
},
}
for _, tc := range testCases {
dst := reflect.New(reflect.TypeOf(tc.want).Elem()).Interface()
err := loadEntity(dst, tc.src)
if err != nil {
t.Errorf("loadEntity: %s: %v", tc.desc, err)
continue
}
if !reflect.DeepEqual(tc.want, dst) {
t.Errorf("%s: compare:\ngot: %#v\nwant: %#v", tc.desc, dst, tc.want)
}
}
}
type WithKey struct {
X string
I int64
K *Key `datastore:"__key__"`
}
type NestedWithKey struct {
N WithKey
Y string
}
var (
incompleteKey = newKey("", nil)
invalidKey = newKey("s", incompleteKey)
// these values need to be addressable
fieldNameA = "A"
fieldNameK = "K"
fieldNameN = "N"
fieldNameY = "Y"
fieldNameAA = "AA"
fieldNameII = "II"
fieldNameBDotB = "B.B"
entityProtoMeaning = pb.Property_ENTITY_PROTO
TRUE = true
FALSE = false
)
var (
simpleEntityProto, nestedSimpleEntityProto,
simpleTwoFieldsEntityProto, simpleWithTagEntityProto,
bDotBEntityProto, withKeyEntityProto string
)
func init() {
// simpleEntityProto corresponds to:
// Simple{I: testInt64}
simpleEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", incompleteKey),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
simpleEntityProto = string(simpleEntityProtob)
// nestedSimpleEntityProto corresponds to:
// NestedSimple{
// A: Simple{I: testInt64},
// I: testInt64,
// }
nestedSimpleEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", incompleteKey),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &simpleEntityProto,
},
Multiple: &FALSE,
},
&pb.Property{
Name: &fieldNameI,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
nestedSimpleEntityProto = string(nestedSimpleEntityProtob)
// simpleTwoFieldsEntityProto corresponds to:
// SimpleTwoFields{S: testString2, SS: testString3}
simpleTwoFieldsEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", incompleteKey),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameS,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
Multiple: &FALSE,
},
&pb.Property{
Name: &fieldNameSS,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
simpleTwoFieldsEntityProto = string(simpleTwoFieldsEntityProtob)
// simpleWithTagEntityProto corresponds to:
// SimpleWithTag{I: testInt64}
simpleWithTagEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", incompleteKey),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameII,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
simpleWithTagEntityProto = string(simpleWithTagEntityProtob)
// bDotBEntityProto corresponds to:
// BDotB{
// B: testString2,
// }
bDotBEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", incompleteKey),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameBDotB,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
bDotBEntityProto = string(bDotBEntityProtob)
// withKeyEntityProto corresponds to:
// WithKey{
// X: testString3,
// I: testInt64,
// K: testKey1a,
// }
withKeyEntityProtob, err := proto.Marshal(&pb.EntityProto{
Key: keyToProto("", testKey1a),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
Multiple: &FALSE,
},
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
Multiple: &FALSE,
},
},
EntityGroup: &pb.Path{},
})
if err != nil {
panic(err)
}
withKeyEntityProto = string(withKeyEntityProtob)
}
func TestLoadEntityNested(t *testing.T) {
testCases := []struct {
desc string
src *pb.EntityProto
want interface{}
}{
{
"nested basic",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &simpleEntityProto,
},
},
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
},
},
&NestedSimple{
A: Simple{I: 2},
I: 2,
},
},
{
"nested with struct tags",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameAA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &simpleWithTagEntityProto,
},
},
},
},
&NestedSimpleWithTag{
A: SimpleWithTag{I: testInt64},
},
},
{
"nested 2x",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameAA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &nestedSimpleEntityProto,
},
},
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &simpleTwoFieldsEntityProto,
},
},
&pb.Property{
Name: &fieldNameS,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
},
},
},
&NestedSimple2X{
AA: NestedSimple{
A: Simple{I: testInt64},
I: testInt64,
},
A: SimpleTwoFields{S: testString2, SS: testString3},
S: testString3,
},
},
{
"nested anonymous",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
},
},
&NestedSimpleAnonymous{
Simple: Simple{I: testInt64},
X: testString2,
},
},
{
"nested simple with slice",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Multiple: &TRUE,
Value: &pb.PropertyValue{
StringValue: &simpleEntityProto,
},
},
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Multiple: &TRUE,
Value: &pb.PropertyValue{
StringValue: &simpleEntityProto,
},
},
},
},
&NestedSliceOfSimple{
A: []Simple{Simple{I: testInt64}, Simple{I: testInt64}},
},
},
{
"nested with multiple anonymous fields",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameI,
Value: &pb.PropertyValue{
Int64Value: &testInt64,
},
},
&pb.Property{
Name: &fieldNameS,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
&pb.Property{
Name: &fieldNameSS,
Value: &pb.PropertyValue{
StringValue: &testString3,
},
},
&pb.Property{
Name: &fieldNameX,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
},
},
&MultiAnonymous{
Simple: Simple{I: testInt64},
SimpleTwoFields: SimpleTwoFields{S: testString2, SS: testString3},
X: testString2,
},
},
{
"nested with dotted field tag",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameA,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &bDotBEntityProto,
},
},
},
},
&ABDotB{
A: BDotB{
B: testString2,
},
},
},
{
"nested entity with key",
&pb.EntityProto{
Key: keyToProto("some-app-id", testKey0),
Property: []*pb.Property{
&pb.Property{
Name: &fieldNameY,
Value: &pb.PropertyValue{
StringValue: &testString2,
},
},
&pb.Property{
Name: &fieldNameN,
Meaning: &entityProtoMeaning,
Value: &pb.PropertyValue{
StringValue: &withKeyEntityProto,
},
},
},
},
&NestedWithKey{
Y: testString2,
N: WithKey{
X: testString3,
I: testInt64,
K: testKey1a,
},
},
},
}
for _, tc := range testCases {
dst := reflect.New(reflect.TypeOf(tc.want).Elem()).Interface()
err := loadEntity(dst, tc.src)
if err != nil {
t.Errorf("loadEntity: %s: %v", tc.desc, err)
continue
}
if !reflect.DeepEqual(tc.want, dst) {
t.Errorf("%s: compare:\ngot: %#v\nwant: %#v", tc.desc, dst, tc.want)
}
}
}
appengine-1.6.7/datastore/metadata.go 0000664 0000000 0000000 00000004367 13741174344 0017612 0 ustar 00root root 0000000 0000000 // Copyright 2016 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import "golang.org/x/net/context"
// Datastore kinds for the metadata entities.
const (
namespaceKind = "__namespace__"
kindKind = "__kind__"
propertyKind = "__property__"
)
// Namespaces returns all the datastore namespaces.
func Namespaces(ctx context.Context) ([]string, error) {
// TODO(djd): Support range queries.
q := NewQuery(namespaceKind).KeysOnly()
keys, err := q.GetAll(ctx, nil)
if err != nil {
return nil, err
}
// The empty namespace key uses a numeric ID (==1), but luckily
// the string ID defaults to "" for numeric IDs anyway.
return keyNames(keys), nil
}
// Kinds returns the names of all the kinds in the current namespace.
func Kinds(ctx context.Context) ([]string, error) {
// TODO(djd): Support range queries.
q := NewQuery(kindKind).KeysOnly()
keys, err := q.GetAll(ctx, nil)
if err != nil {
return nil, err
}
return keyNames(keys), nil
}
// keyNames returns a slice of the provided keys' names (string IDs).
func keyNames(keys []*Key) []string {
n := make([]string, 0, len(keys))
for _, k := range keys {
n = append(n, k.StringID())
}
return n
}
// KindProperties returns all the indexed properties for the given kind.
// The properties are returned as a map of property names to a slice of the
// representation types. The representation types for the supported Go property
// types are:
// "INT64": signed integers and time.Time
// "DOUBLE": float32 and float64
// "BOOLEAN": bool
// "STRING": string, []byte and ByteString
// "POINT": appengine.GeoPoint
// "REFERENCE": *Key
// "USER": (not used in the Go runtime)
func KindProperties(ctx context.Context, kind string) (map[string][]string, error) {
// TODO(djd): Support range queries.
kindKey := NewKey(ctx, kindKind, kind, 0, nil)
q := NewQuery(propertyKind).Ancestor(kindKey)
propMap := map[string][]string{}
props := []struct {
Repr []string `datastore:"property_representation"`
}{}
keys, err := q.GetAll(ctx, &props)
if err != nil {
return nil, err
}
for i, p := range props {
propMap[keys[i].StringID()] = p.Repr
}
return propMap, nil
}
appengine-1.6.7/datastore/prop.go 0000664 0000000 0000000 00000023143 13741174344 0017003 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"fmt"
"reflect"
"strings"
"sync"
"unicode"
)
// Entities with more than this many indexed properties will not be saved.
const maxIndexedProperties = 20000
// []byte fields more than 1 megabyte long will not be loaded or saved.
const maxBlobLen = 1 << 20
// Property is a name/value pair plus some metadata. A datastore entity's
// contents are loaded and saved as a sequence of Properties. An entity can
// have multiple Properties with the same name, provided that p.Multiple is
// true on all of that entity's Properties with that name.
type Property struct {
// Name is the property name.
Name string
// Value is the property value. The valid types are:
// - int64
// - bool
// - string
// - float64
// - ByteString
// - *Key
// - time.Time
// - appengine.BlobKey
// - appengine.GeoPoint
// - []byte (up to 1 megabyte in length)
// - *Entity (representing a nested struct)
// This set is smaller than the set of valid struct field types that the
// datastore can load and save. A Property Value cannot be a slice (apart
// from []byte); use multiple Properties instead. Also, a Value's type
// must be explicitly on the list above; it is not sufficient for the
// underlying type to be on that list. For example, a Value of "type
// myInt64 int64" is invalid. Smaller-width integers and floats are also
// invalid. Again, this is more restrictive than the set of valid struct
// field types.
//
// A Value will have an opaque type when loading entities from an index,
// such as via a projection query. Load entities into a struct instead
// of a PropertyLoadSaver when using a projection query.
//
// A Value may also be the nil interface value; this is equivalent to
// Python's None but not directly representable by a Go struct. Loading
// a nil-valued property into a struct will set that field to the zero
// value.
Value interface{}
// NoIndex is whether the datastore cannot index this property.
NoIndex bool
// Multiple is whether the entity can have multiple properties with
// the same name. Even if a particular instance only has one property with
// a certain name, Multiple should be true if a struct would best represent
// it as a field of type []T instead of type T.
Multiple bool
}
// An Entity is the value type for a nested struct.
// This type is only used for a Property's Value.
type Entity struct {
Key *Key
Properties []Property
}
// ByteString is a short byte slice (up to 1500 bytes) that can be indexed.
type ByteString []byte
// PropertyLoadSaver can be converted from and to a slice of Properties.
type PropertyLoadSaver interface {
Load([]Property) error
Save() ([]Property, error)
}
// PropertyList converts a []Property to implement PropertyLoadSaver.
type PropertyList []Property
var (
typeOfPropertyLoadSaver = reflect.TypeOf((*PropertyLoadSaver)(nil)).Elem()
typeOfPropertyList = reflect.TypeOf(PropertyList(nil))
)
// Load loads all of the provided properties into l.
// It does not first reset *l to an empty slice.
func (l *PropertyList) Load(p []Property) error {
*l = append(*l, p...)
return nil
}
// Save saves all of l's properties as a slice or Properties.
func (l *PropertyList) Save() ([]Property, error) {
return *l, nil
}
// validPropertyName returns whether name consists of one or more valid Go
// identifiers joined by ".".
func validPropertyName(name string) bool {
if name == "" {
return false
}
for _, s := range strings.Split(name, ".") {
if s == "" {
return false
}
first := true
for _, c := range s {
if first {
first = false
if c != '_' && !unicode.IsLetter(c) {
return false
}
} else {
if c != '_' && !unicode.IsLetter(c) && !unicode.IsDigit(c) {
return false
}
}
}
}
return true
}
// structCodec describes how to convert a struct to and from a sequence of
// properties.
type structCodec struct {
// fields gives the field codec for the structTag with the given name.
fields map[string]fieldCodec
// hasSlice is whether a struct or any of its nested or embedded structs
// has a slice-typed field (other than []byte).
hasSlice bool
// keyField is the index of a *Key field with structTag __key__.
// This field is not relevant for the top level struct, only for
// nested structs.
keyField int
// complete is whether the structCodec is complete. An incomplete
// structCodec may be encountered when walking a recursive struct.
complete bool
}
// fieldCodec is a struct field's index and, if that struct field's type is
// itself a struct, that substruct's structCodec.
type fieldCodec struct {
// path is the index path to the field
path []int
noIndex bool
// omitEmpty indicates that the field should be omitted on save
// if empty.
omitEmpty bool
// structCodec is the codec fot the struct field at index 'path',
// or nil if the field is not a struct.
structCodec *structCodec
}
// structCodecs collects the structCodecs that have already been calculated.
var (
structCodecsMutex sync.Mutex
structCodecs = make(map[reflect.Type]*structCodec)
)
// getStructCodec returns the structCodec for the given struct type.
func getStructCodec(t reflect.Type) (*structCodec, error) {
structCodecsMutex.Lock()
defer structCodecsMutex.Unlock()
return getStructCodecLocked(t)
}
// getStructCodecLocked implements getStructCodec. The structCodecsMutex must
// be held when calling this function.
func getStructCodecLocked(t reflect.Type) (ret *structCodec, retErr error) {
c, ok := structCodecs[t]
if ok {
return c, nil
}
c = &structCodec{
fields: make(map[string]fieldCodec),
// We initialize keyField to -1 so that the zero-value is not
// misinterpreted as index 0.
keyField: -1,
}
// Add c to the structCodecs map before we are sure it is good. If t is
// a recursive type, it needs to find the incomplete entry for itself in
// the map.
structCodecs[t] = c
defer func() {
if retErr != nil {
delete(structCodecs, t)
}
}()
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
// Skip unexported fields.
// Note that if f is an anonymous, unexported struct field,
// we will promote its fields.
if f.PkgPath != "" && !f.Anonymous {
continue
}
tags := strings.Split(f.Tag.Get("datastore"), ",")
name := tags[0]
opts := make(map[string]bool)
for _, t := range tags[1:] {
opts[t] = true
}
switch {
case name == "":
if !f.Anonymous {
name = f.Name
}
case name == "-":
continue
case name == "__key__":
if f.Type != typeOfKeyPtr {
return nil, fmt.Errorf("datastore: __key__ field on struct %v is not a *datastore.Key", t)
}
c.keyField = i
case !validPropertyName(name):
return nil, fmt.Errorf("datastore: struct tag has invalid property name: %q", name)
}
substructType, fIsSlice := reflect.Type(nil), false
switch f.Type.Kind() {
case reflect.Struct:
substructType = f.Type
case reflect.Slice:
if f.Type.Elem().Kind() == reflect.Struct {
substructType = f.Type.Elem()
}
fIsSlice = f.Type != typeOfByteSlice
c.hasSlice = c.hasSlice || fIsSlice
}
var sub *structCodec
if substructType != nil && substructType != typeOfTime && substructType != typeOfGeoPoint {
var err error
sub, err = getStructCodecLocked(substructType)
if err != nil {
return nil, err
}
if !sub.complete {
return nil, fmt.Errorf("datastore: recursive struct: field %q", f.Name)
}
if fIsSlice && sub.hasSlice {
return nil, fmt.Errorf(
"datastore: flattening nested structs leads to a slice of slices: field %q", f.Name)
}
c.hasSlice = c.hasSlice || sub.hasSlice
// If f is an anonymous struct field, we promote the substruct's fields up to this level
// in the linked list of struct codecs.
if f.Anonymous {
for subname, subfield := range sub.fields {
if name != "" {
subname = name + "." + subname
}
if _, ok := c.fields[subname]; ok {
return nil, fmt.Errorf("datastore: struct tag has repeated property name: %q", subname)
}
c.fields[subname] = fieldCodec{
path: append([]int{i}, subfield.path...),
noIndex: subfield.noIndex || opts["noindex"],
omitEmpty: subfield.omitEmpty,
structCodec: subfield.structCodec,
}
}
continue
}
}
if _, ok := c.fields[name]; ok {
return nil, fmt.Errorf("datastore: struct tag has repeated property name: %q", name)
}
c.fields[name] = fieldCodec{
path: []int{i},
noIndex: opts["noindex"],
omitEmpty: opts["omitempty"],
structCodec: sub,
}
}
c.complete = true
return c, nil
}
// structPLS adapts a struct to be a PropertyLoadSaver.
type structPLS struct {
v reflect.Value
codec *structCodec
}
// newStructPLS returns a structPLS, which implements the
// PropertyLoadSaver interface, for the struct pointer p.
func newStructPLS(p interface{}) (*structPLS, error) {
v := reflect.ValueOf(p)
if v.Kind() != reflect.Ptr || v.Elem().Kind() != reflect.Struct {
return nil, ErrInvalidEntityType
}
v = v.Elem()
codec, err := getStructCodec(v.Type())
if err != nil {
return nil, err
}
return &structPLS{v, codec}, nil
}
// LoadStruct loads the properties from p to dst.
// dst must be a struct pointer.
func LoadStruct(dst interface{}, p []Property) error {
x, err := newStructPLS(dst)
if err != nil {
return err
}
return x.Load(p)
}
// SaveStruct returns the properties from src as a slice of Properties.
// src must be a struct pointer.
func SaveStruct(src interface{}) ([]Property, error) {
x, err := newStructPLS(src)
if err != nil {
return nil, err
}
return x.Save()
}
appengine-1.6.7/datastore/prop_test.go 0000664 0000000 0000000 00000032754 13741174344 0020052 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"reflect"
"sort"
"testing"
"time"
"google.golang.org/appengine"
)
func TestValidPropertyName(t *testing.T) {
testCases := []struct {
name string
want bool
}{
// Invalid names.
{"", false},
{"'", false},
{".", false},
{"..", false},
{".foo", false},
{"0", false},
{"00", false},
{"X.X.4.X.X", false},
{"\n", false},
{"\x00", false},
{"abc\xffz", false},
{"foo.", false},
{"foo..", false},
{"foo..bar", false},
{"☃", false},
{`"`, false},
// Valid names.
{"AB", true},
{"Abc", true},
{"X.X.X.X.X", true},
{"_", true},
{"_0", true},
{"a", true},
{"a_B", true},
{"f00", true},
{"f0o", true},
{"fo0", true},
{"foo", true},
{"foo.bar", true},
{"foo.bar.baz", true},
{"世界", true},
}
for _, tc := range testCases {
got := validPropertyName(tc.name)
if got != tc.want {
t.Errorf("%q: got %v, want %v", tc.name, got, tc.want)
}
}
}
func TestStructCodec(t *testing.T) {
type oStruct struct {
O int
}
type pStruct struct {
P int
Q int
}
type rStruct struct {
R int
S pStruct
T oStruct
oStruct
}
type uStruct struct {
U int
v int
}
type vStruct struct {
V string `datastore:",noindex"`
}
oStructCodec := &structCodec{
fields: map[string]fieldCodec{
"O": {path: []int{0}},
},
complete: true,
}
pStructCodec := &structCodec{
fields: map[string]fieldCodec{
"P": {path: []int{0}},
"Q": {path: []int{1}},
},
complete: true,
}
rStructCodec := &structCodec{
fields: map[string]fieldCodec{
"R": {path: []int{0}},
"S": {path: []int{1}, structCodec: pStructCodec},
"T": {path: []int{2}, structCodec: oStructCodec},
"O": {path: []int{3, 0}},
},
complete: true,
}
uStructCodec := &structCodec{
fields: map[string]fieldCodec{
"U": {path: []int{0}},
},
complete: true,
}
vStructCodec := &structCodec{
fields: map[string]fieldCodec{
"V": {path: []int{0}, noIndex: true},
},
complete: true,
}
testCases := []struct {
desc string
structValue interface{}
want *structCodec
}{
{
"oStruct",
oStruct{},
oStructCodec,
},
{
"pStruct",
pStruct{},
pStructCodec,
},
{
"rStruct",
rStruct{},
rStructCodec,
},
{
"uStruct",
uStruct{},
uStructCodec,
},
{
"non-basic fields",
struct {
B appengine.BlobKey
K *Key
T time.Time
}{},
&structCodec{
fields: map[string]fieldCodec{
"B": {path: []int{0}},
"K": {path: []int{1}},
"T": {path: []int{2}},
},
complete: true,
},
},
{
"struct tags with ignored embed",
struct {
A int `datastore:"a,noindex"`
B int `datastore:"b"`
C int `datastore:",noindex"`
D int `datastore:""`
E int
I int `datastore:"-"`
J int `datastore:",noindex" json:"j"`
oStruct `datastore:"-"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"a": {path: []int{0}, noIndex: true},
"b": {path: []int{1}},
"C": {path: []int{2}, noIndex: true},
"D": {path: []int{3}},
"E": {path: []int{4}},
"J": {path: []int{6}, noIndex: true},
},
complete: true,
},
},
{
"unexported fields",
struct {
A int
b int
C int `datastore:"x"`
d int `datastore:"Y"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"A": {path: []int{0}},
"x": {path: []int{2}},
},
complete: true,
},
},
{
"nested and embedded structs",
struct {
A int
B int
CC oStruct
DDD rStruct
oStruct
}{},
&structCodec{
fields: map[string]fieldCodec{
"A": {path: []int{0}},
"B": {path: []int{1}},
"CC": {path: []int{2}, structCodec: oStructCodec},
"DDD": {path: []int{3}, structCodec: rStructCodec},
"O": {path: []int{4, 0}},
},
complete: true,
},
},
{
"struct tags with nested and embedded structs",
struct {
A int `datastore:"-"`
B int `datastore:"w"`
C oStruct `datastore:"xx"`
D rStruct `datastore:"y"`
oStruct `datastore:"z"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"w": {path: []int{1}},
"xx": {path: []int{2}, structCodec: oStructCodec},
"y": {path: []int{3}, structCodec: rStructCodec},
"z.O": {path: []int{4, 0}},
},
complete: true,
},
},
{
"unexported nested and embedded structs",
struct {
a int
B int
c uStruct
D uStruct
uStruct
}{},
&structCodec{
fields: map[string]fieldCodec{
"B": {path: []int{1}},
"D": {path: []int{3}, structCodec: uStructCodec},
"U": {path: []int{4, 0}},
},
complete: true,
},
},
{
"noindex nested struct",
struct {
A oStruct `datastore:",noindex"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"A": {path: []int{0}, structCodec: oStructCodec, noIndex: true},
},
complete: true,
},
},
{
"noindex slice",
struct {
A []string `datastore:",noindex"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"A": {path: []int{0}, noIndex: true},
},
hasSlice: true,
complete: true,
},
},
{
"noindex embedded struct slice",
struct {
// vStruct has a single field, V, also with noindex.
A []vStruct `datastore:",noindex"`
}{},
&structCodec{
fields: map[string]fieldCodec{
"A": {path: []int{0}, structCodec: vStructCodec, noIndex: true},
},
hasSlice: true,
complete: true,
},
},
}
for _, tc := range testCases {
got, err := getStructCodec(reflect.TypeOf(tc.structValue))
if err != nil {
t.Errorf("%s: getStructCodec: %v", tc.desc, err)
continue
}
// can't reflect.DeepEqual b/c element order in fields map may differ
if !isEqualStructCodec(got, tc.want) {
t.Errorf("%s\ngot %+v\nwant %+v\n", tc.desc, got, tc.want)
}
}
}
func isEqualStructCodec(got, want *structCodec) bool {
if got.complete != want.complete {
return false
}
if got.hasSlice != want.hasSlice {
return false
}
if len(got.fields) != len(want.fields) {
return false
}
for name, wantF := range want.fields {
gotF := got.fields[name]
if !reflect.DeepEqual(wantF.path, gotF.path) {
return false
}
if wantF.noIndex != gotF.noIndex {
return false
}
if wantF.structCodec != nil {
if gotF.structCodec == nil {
return false
}
if !isEqualStructCodec(gotF.structCodec, wantF.structCodec) {
return false
}
}
}
return true
}
func TestRepeatedPropertyName(t *testing.T) {
good := []interface{}{
struct {
A int `datastore:"-"`
}{},
struct {
A int `datastore:"b"`
B int
}{},
struct {
A int
B int `datastore:"B"`
}{},
struct {
A int `datastore:"B"`
B int `datastore:"-"`
}{},
struct {
A int `datastore:"-"`
B int `datastore:"A"`
}{},
struct {
A int `datastore:"B"`
B int `datastore:"A"`
}{},
struct {
A int `datastore:"B"`
B int `datastore:"C"`
C int `datastore:"A"`
}{},
struct {
A int `datastore:"B"`
B int `datastore:"C"`
C int `datastore:"D"`
}{},
}
bad := []interface{}{
struct {
A int `datastore:"B"`
B int
}{},
struct {
A int
B int `datastore:"A"`
}{},
struct {
A int `datastore:"C"`
B int `datastore:"C"`
}{},
struct {
A int `datastore:"B"`
B int `datastore:"C"`
C int `datastore:"B"`
}{},
}
testGetStructCodec(t, good, bad)
}
func TestFlatteningNestedStructs(t *testing.T) {
type DeepGood struct {
A struct {
B []struct {
C struct {
D int
}
}
}
}
type DeepBad struct {
A struct {
B []struct {
C struct {
D []int
}
}
}
}
type ISay struct {
Tomato int
}
type YouSay struct {
Tomato int
}
type Tweedledee struct {
Dee int `datastore:"D"`
}
type Tweedledum struct {
Dum int `datastore:"D"`
}
good := []interface{}{
struct {
X []struct {
Y string
}
}{},
struct {
X []struct {
Y []byte
}
}{},
struct {
P []int
X struct {
Y []int
}
}{},
struct {
X struct {
Y []int
}
Q []int
}{},
struct {
P []int
X struct {
Y []int
}
Q []int
}{},
struct {
DeepGood
}{},
struct {
DG DeepGood
}{},
struct {
Foo struct {
Z int
} `datastore:"A"`
Bar struct {
Z int
} `datastore:"B"`
}{},
}
bad := []interface{}{
struct {
X []struct {
Y []string
}
}{},
struct {
X []struct {
Y []int
}
}{},
struct {
DeepBad
}{},
struct {
DB DeepBad
}{},
struct {
ISay
YouSay
}{},
struct {
Tweedledee
Tweedledum
}{},
struct {
Foo struct {
Z int
} `datastore:"A"`
Bar struct {
Z int
} `datastore:"A"`
}{},
}
testGetStructCodec(t, good, bad)
}
func testGetStructCodec(t *testing.T, good []interface{}, bad []interface{}) {
for _, x := range good {
if _, err := getStructCodec(reflect.TypeOf(x)); err != nil {
t.Errorf("type %T: got non-nil error (%s), want nil", x, err)
}
}
for _, x := range bad {
if _, err := getStructCodec(reflect.TypeOf(x)); err == nil {
t.Errorf("type %T: got nil error, want non-nil", x)
}
}
}
func TestNilKeyIsStored(t *testing.T) {
x := struct {
K *Key
I int
}{}
p := PropertyList{}
// Save x as properties.
p1, _ := SaveStruct(&x)
p.Load(p1)
// Set x's fields to non-zero.
x.K = &Key{}
x.I = 2
// Load x from properties.
p2, _ := p.Save()
LoadStruct(&x, p2)
// Check that x's fields were set to zero.
if x.K != nil {
t.Errorf("K field was not zero")
}
if x.I != 0 {
t.Errorf("I field was not zero")
}
}
func TestSaveStructOmitEmpty(t *testing.T) {
// Expected props names are sorted alphabetically
expectedPropNamesForSingles := []string{"EmptyValue", "NonEmptyValue", "OmitEmptyWithValue"}
expectedPropNamesForSlices := []string{"NonEmptyValue", "NonEmptyValue", "OmitEmptyWithValue", "OmitEmptyWithValue"}
testOmitted := func(expectedPropNames []string, src interface{}) {
// t.Helper() - this is available from Go version 1.9, but we also support Go versions 1.6, 1.7, 1.8
if props, err := SaveStruct(src); err != nil {
t.Fatal(err)
} else {
// Collect names for reporting if diffs from expected and for easier sorting
actualPropNames := make([]string, len(props))
for i := range props {
actualPropNames[i] = props[i].Name
}
// Sort actuals for comparing with already sorted expected names
sort.Sort(sort.StringSlice(actualPropNames))
if !reflect.DeepEqual(actualPropNames, expectedPropNames) {
t.Errorf("Expected this properties: %v, got: %v", expectedPropNames, actualPropNames)
}
}
}
testOmitted(expectedPropNamesForSingles, &struct {
EmptyValue int
NonEmptyValue int
OmitEmptyNoValue int `datastore:",omitempty"`
OmitEmptyWithValue int `datastore:",omitempty"`
}{
NonEmptyValue: 1,
OmitEmptyWithValue: 2,
})
testOmitted(expectedPropNamesForSlices, &struct {
EmptyValue []int
NonEmptyValue []int
OmitEmptyNoValue []int `datastore:",omitempty"`
OmitEmptyWithValue []int `datastore:",omitempty"`
}{
NonEmptyValue: []int{1, 2},
OmitEmptyWithValue: []int{3, 4},
})
testOmitted(expectedPropNamesForSingles, &struct {
EmptyValue bool
NonEmptyValue bool
OmitEmptyNoValue bool `datastore:",omitempty"`
OmitEmptyWithValue bool `datastore:",omitempty"`
}{
NonEmptyValue: true,
OmitEmptyWithValue: true,
})
testOmitted(expectedPropNamesForSlices, &struct {
EmptyValue []bool
NonEmptyValue []bool
OmitEmptyNoValue []bool `datastore:",omitempty"`
OmitEmptyWithValue []bool `datastore:",omitempty"`
}{
NonEmptyValue: []bool{true, true},
OmitEmptyWithValue: []bool{true, true},
})
testOmitted(expectedPropNamesForSingles, &struct {
EmptyValue string
NonEmptyValue string
OmitEmptyNoValue string `datastore:",omitempty"`
OmitEmptyWithValue string `datastore:",omitempty"`
}{
NonEmptyValue: "s",
OmitEmptyWithValue: "s",
})
testOmitted(expectedPropNamesForSlices, &struct {
EmptyValue []string
NonEmptyValue []string
OmitEmptyNoValue []string `datastore:",omitempty"`
OmitEmptyWithValue []string `datastore:",omitempty"`
}{
NonEmptyValue: []string{"s1", "s2"},
OmitEmptyWithValue: []string{"s3", "s4"},
})
testOmitted(expectedPropNamesForSingles, &struct {
EmptyValue float32
NonEmptyValue float32
OmitEmptyNoValue float32 `datastore:",omitempty"`
OmitEmptyWithValue float32 `datastore:",omitempty"`
}{
NonEmptyValue: 1.1,
OmitEmptyWithValue: 1.2,
})
testOmitted(expectedPropNamesForSlices, &struct {
EmptyValue []float32
NonEmptyValue []float32
OmitEmptyNoValue []float32 `datastore:",omitempty"`
OmitEmptyWithValue []float32 `datastore:",omitempty"`
}{
NonEmptyValue: []float32{1.1, 2.2},
OmitEmptyWithValue: []float32{3.3, 4.4},
})
testOmitted(expectedPropNamesForSingles, &struct {
EmptyValue time.Time
NonEmptyValue time.Time
OmitEmptyNoValue time.Time `datastore:",omitempty"`
OmitEmptyWithValue time.Time `datastore:",omitempty"`
}{
NonEmptyValue: now,
OmitEmptyWithValue: now,
})
testOmitted(expectedPropNamesForSlices, &struct {
EmptyValue []time.Time
NonEmptyValue []time.Time
OmitEmptyNoValue []time.Time `datastore:",omitempty"`
OmitEmptyWithValue []time.Time `datastore:",omitempty"`
}{
NonEmptyValue: []time.Time{now, now},
OmitEmptyWithValue: []time.Time{now, now},
})
}
appengine-1.6.7/datastore/query.go 0000664 0000000 0000000 00000054217 13741174344 0017176 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"encoding/base64"
"errors"
"fmt"
"math"
"reflect"
"strings"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
pb "google.golang.org/appengine/internal/datastore"
)
type operator int
const (
lessThan operator = iota
lessEq
equal
greaterEq
greaterThan
)
var operatorToProto = map[operator]*pb.Query_Filter_Operator{
lessThan: pb.Query_Filter_LESS_THAN.Enum(),
lessEq: pb.Query_Filter_LESS_THAN_OR_EQUAL.Enum(),
equal: pb.Query_Filter_EQUAL.Enum(),
greaterEq: pb.Query_Filter_GREATER_THAN_OR_EQUAL.Enum(),
greaterThan: pb.Query_Filter_GREATER_THAN.Enum(),
}
// filter is a conditional filter on query results.
type filter struct {
FieldName string
Op operator
Value interface{}
}
type sortDirection int
const (
ascending sortDirection = iota
descending
)
var sortDirectionToProto = map[sortDirection]*pb.Query_Order_Direction{
ascending: pb.Query_Order_ASCENDING.Enum(),
descending: pb.Query_Order_DESCENDING.Enum(),
}
// order is a sort order on query results.
type order struct {
FieldName string
Direction sortDirection
}
// NewQuery creates a new Query for a specific entity kind.
//
// An empty kind means to return all entities, including entities created and
// managed by other App Engine features, and is called a kindless query.
// Kindless queries cannot include filters or sort orders on property values.
func NewQuery(kind string) *Query {
return &Query{
kind: kind,
limit: -1,
}
}
// Query represents a datastore query.
type Query struct {
kind string
ancestor *Key
filter []filter
order []order
projection []string
distinct bool
distinctOn []string
keysOnly bool
eventual bool
limit int32
offset int32
count int32
start *pb.CompiledCursor
end *pb.CompiledCursor
err error
}
func (q *Query) clone() *Query {
x := *q
// Copy the contents of the slice-typed fields to a new backing store.
if len(q.filter) > 0 {
x.filter = make([]filter, len(q.filter))
copy(x.filter, q.filter)
}
if len(q.order) > 0 {
x.order = make([]order, len(q.order))
copy(x.order, q.order)
}
return &x
}
// Ancestor returns a derivative query with an ancestor filter.
// The ancestor should not be nil.
func (q *Query) Ancestor(ancestor *Key) *Query {
q = q.clone()
if ancestor == nil {
q.err = errors.New("datastore: nil query ancestor")
return q
}
q.ancestor = ancestor
return q
}
// EventualConsistency returns a derivative query that returns eventually
// consistent results.
// It only has an effect on ancestor queries.
func (q *Query) EventualConsistency() *Query {
q = q.clone()
q.eventual = true
return q
}
// Filter returns a derivative query with a field-based filter.
// The filterStr argument must be a field name followed by optional space,
// followed by an operator, one of ">", "<", ">=", "<=", or "=".
// Fields are compared against the provided value using the operator.
// Multiple filters are AND'ed together.
func (q *Query) Filter(filterStr string, value interface{}) *Query {
q = q.clone()
filterStr = strings.TrimSpace(filterStr)
if len(filterStr) < 1 {
q.err = errors.New("datastore: invalid filter: " + filterStr)
return q
}
f := filter{
FieldName: strings.TrimRight(filterStr, " ><=!"),
Value: value,
}
switch op := strings.TrimSpace(filterStr[len(f.FieldName):]); op {
case "<=":
f.Op = lessEq
case ">=":
f.Op = greaterEq
case "<":
f.Op = lessThan
case ">":
f.Op = greaterThan
case "=":
f.Op = equal
default:
q.err = fmt.Errorf("datastore: invalid operator %q in filter %q", op, filterStr)
return q
}
q.filter = append(q.filter, f)
return q
}
// Order returns a derivative query with a field-based sort order. Orders are
// applied in the order they are added. The default order is ascending; to sort
// in descending order prefix the fieldName with a minus sign (-).
func (q *Query) Order(fieldName string) *Query {
q = q.clone()
fieldName = strings.TrimSpace(fieldName)
o := order{
Direction: ascending,
FieldName: fieldName,
}
if strings.HasPrefix(fieldName, "-") {
o.Direction = descending
o.FieldName = strings.TrimSpace(fieldName[1:])
} else if strings.HasPrefix(fieldName, "+") {
q.err = fmt.Errorf("datastore: invalid order: %q", fieldName)
return q
}
if len(o.FieldName) == 0 {
q.err = errors.New("datastore: empty order")
return q
}
q.order = append(q.order, o)
return q
}
// Project returns a derivative query that yields only the given fields. It
// cannot be used with KeysOnly.
func (q *Query) Project(fieldNames ...string) *Query {
q = q.clone()
q.projection = append([]string(nil), fieldNames...)
return q
}
// Distinct returns a derivative query that yields de-duplicated entities with
// respect to the set of projected fields. It is only used for projection
// queries. Distinct cannot be used with DistinctOn.
func (q *Query) Distinct() *Query {
q = q.clone()
q.distinct = true
return q
}
// DistinctOn returns a derivative query that yields de-duplicated entities with
// respect to the set of the specified fields. It is only used for projection
// queries. The field list should be a subset of the projected field list.
// DistinctOn cannot be used with Distinct.
func (q *Query) DistinctOn(fieldNames ...string) *Query {
q = q.clone()
q.distinctOn = fieldNames
return q
}
// KeysOnly returns a derivative query that yields only keys, not keys and
// entities. It cannot be used with projection queries.
func (q *Query) KeysOnly() *Query {
q = q.clone()
q.keysOnly = true
return q
}
// Limit returns a derivative query that has a limit on the number of results
// returned. A negative value means unlimited.
func (q *Query) Limit(limit int) *Query {
q = q.clone()
if limit < math.MinInt32 || limit > math.MaxInt32 {
q.err = errors.New("datastore: query limit overflow")
return q
}
q.limit = int32(limit)
return q
}
// Offset returns a derivative query that has an offset of how many keys to
// skip over before returning results. A negative value is invalid.
func (q *Query) Offset(offset int) *Query {
q = q.clone()
if offset < 0 {
q.err = errors.New("datastore: negative query offset")
return q
}
if offset > math.MaxInt32 {
q.err = errors.New("datastore: query offset overflow")
return q
}
q.offset = int32(offset)
return q
}
// BatchSize returns a derivative query to fetch the supplied number of results
// at once. This value should be greater than zero, and equal to or less than
// the Limit.
func (q *Query) BatchSize(size int) *Query {
q = q.clone()
if size <= 0 || size > math.MaxInt32 {
q.err = errors.New("datastore: query batch size overflow")
return q
}
q.count = int32(size)
return q
}
// Start returns a derivative query with the given start point.
func (q *Query) Start(c Cursor) *Query {
q = q.clone()
if c.cc == nil {
q.err = errors.New("datastore: invalid cursor")
return q
}
q.start = c.cc
return q
}
// End returns a derivative query with the given end point.
func (q *Query) End(c Cursor) *Query {
q = q.clone()
if c.cc == nil {
q.err = errors.New("datastore: invalid cursor")
return q
}
q.end = c.cc
return q
}
// toProto converts the query to a protocol buffer.
func (q *Query) toProto(dst *pb.Query, appID string) error {
if len(q.projection) != 0 && q.keysOnly {
return errors.New("datastore: query cannot both project and be keys-only")
}
if len(q.distinctOn) != 0 && q.distinct {
return errors.New("datastore: query cannot be both distinct and distinct-on")
}
dst.Reset()
dst.App = proto.String(appID)
if q.kind != "" {
dst.Kind = proto.String(q.kind)
}
if q.ancestor != nil {
dst.Ancestor = keyToProto(appID, q.ancestor)
if q.eventual {
dst.Strong = proto.Bool(false)
}
}
if q.projection != nil {
dst.PropertyName = q.projection
if len(q.distinctOn) != 0 {
dst.GroupByPropertyName = q.distinctOn
}
if q.distinct {
dst.GroupByPropertyName = q.projection
}
}
if q.keysOnly {
dst.KeysOnly = proto.Bool(true)
dst.RequirePerfectPlan = proto.Bool(true)
}
for _, qf := range q.filter {
if qf.FieldName == "" {
return errors.New("datastore: empty query filter field name")
}
p, errStr := valueToProto(appID, qf.FieldName, reflect.ValueOf(qf.Value), false)
if errStr != "" {
return errors.New("datastore: bad query filter value type: " + errStr)
}
xf := &pb.Query_Filter{
Op: operatorToProto[qf.Op],
Property: []*pb.Property{p},
}
if xf.Op == nil {
return errors.New("datastore: unknown query filter operator")
}
dst.Filter = append(dst.Filter, xf)
}
for _, qo := range q.order {
if qo.FieldName == "" {
return errors.New("datastore: empty query order field name")
}
xo := &pb.Query_Order{
Property: proto.String(qo.FieldName),
Direction: sortDirectionToProto[qo.Direction],
}
if xo.Direction == nil {
return errors.New("datastore: unknown query order direction")
}
dst.Order = append(dst.Order, xo)
}
if q.limit >= 0 {
dst.Limit = proto.Int32(q.limit)
}
if q.offset != 0 {
dst.Offset = proto.Int32(q.offset)
}
if q.count != 0 {
dst.Count = proto.Int32(q.count)
}
dst.CompiledCursor = q.start
dst.EndCompiledCursor = q.end
dst.Compile = proto.Bool(true)
return nil
}
// Count returns the number of results for the query.
//
// The running time and number of API calls made by Count scale linearly with
// the sum of the query's offset and limit. Unless the result count is
// expected to be small, it is best to specify a limit; otherwise Count will
// continue until it finishes counting or the provided context expires.
func (q *Query) Count(c context.Context) (int, error) {
// Check that the query is well-formed.
if q.err != nil {
return 0, q.err
}
// Run a copy of the query, with keysOnly true (if we're not a projection,
// since the two are incompatible), and an adjusted offset. We also set the
// limit to zero, as we don't want any actual entity data, just the number
// of skipped results.
newQ := q.clone()
newQ.keysOnly = len(newQ.projection) == 0
newQ.limit = 0
if q.limit < 0 {
// If the original query was unlimited, set the new query's offset to maximum.
newQ.offset = math.MaxInt32
} else {
newQ.offset = q.offset + q.limit
if newQ.offset < 0 {
// Do the best we can, in the presence of overflow.
newQ.offset = math.MaxInt32
}
}
req := &pb.Query{}
if err := newQ.toProto(req, internal.FullyQualifiedAppID(c)); err != nil {
return 0, err
}
res := &pb.QueryResult{}
if err := internal.Call(c, "datastore_v3", "RunQuery", req, res); err != nil {
return 0, err
}
// n is the count we will return. For example, suppose that our original
// query had an offset of 4 and a limit of 2008: the count will be 2008,
// provided that there are at least 2012 matching entities. However, the
// RPCs will only skip 1000 results at a time. The RPC sequence is:
// call RunQuery with (offset, limit) = (2012, 0) // 2012 == newQ.offset
// response has (skippedResults, moreResults) = (1000, true)
// n += 1000 // n == 1000
// call Next with (offset, limit) = (1012, 0) // 1012 == newQ.offset - n
// response has (skippedResults, moreResults) = (1000, true)
// n += 1000 // n == 2000
// call Next with (offset, limit) = (12, 0) // 12 == newQ.offset - n
// response has (skippedResults, moreResults) = (12, false)
// n += 12 // n == 2012
// // exit the loop
// n -= 4 // n == 2008
var n int32
for {
// The QueryResult should have no actual entity data, just skipped results.
if len(res.Result) != 0 {
return 0, errors.New("datastore: internal error: Count request returned too much data")
}
n += res.GetSkippedResults()
if !res.GetMoreResults() {
break
}
if err := callNext(c, res, newQ.offset-n, q.count); err != nil {
return 0, err
}
}
n -= q.offset
if n < 0 {
// If the offset was greater than the number of matching entities,
// return 0 instead of negative.
n = 0
}
return int(n), nil
}
// callNext issues a datastore_v3/Next RPC to advance a cursor, such as that
// returned by a query with more results.
func callNext(c context.Context, res *pb.QueryResult, offset, count int32) error {
if res.Cursor == nil {
return errors.New("datastore: internal error: server did not return a cursor")
}
req := &pb.NextRequest{
Cursor: res.Cursor,
}
if count >= 0 {
req.Count = proto.Int32(count)
}
if offset != 0 {
req.Offset = proto.Int32(offset)
}
if res.CompiledCursor != nil {
req.Compile = proto.Bool(true)
}
res.Reset()
return internal.Call(c, "datastore_v3", "Next", req, res)
}
// GetAll runs the query in the given context and returns all keys that match
// that query, as well as appending the values to dst.
//
// dst must have type *[]S or *[]*S or *[]P, for some struct type S or some non-
// interface, non-pointer type P such that P or *P implements PropertyLoadSaver.
//
// As a special case, *PropertyList is an invalid type for dst, even though a
// PropertyList is a slice of structs. It is treated as invalid to avoid being
// mistakenly passed when *[]PropertyList was intended.
//
// The keys returned by GetAll will be in a 1-1 correspondence with the entities
// added to dst.
//
// If q is a ``keys-only'' query, GetAll ignores dst and only returns the keys.
//
// The running time and number of API calls made by GetAll scale linearly with
// the sum of the query's offset and limit. Unless the result count is
// expected to be small, it is best to specify a limit; otherwise GetAll will
// continue until it finishes collecting results or the provided context
// expires.
func (q *Query) GetAll(c context.Context, dst interface{}) ([]*Key, error) {
var (
dv reflect.Value
mat multiArgType
elemType reflect.Type
errFieldMismatch error
)
if !q.keysOnly {
dv = reflect.ValueOf(dst)
if dv.Kind() != reflect.Ptr || dv.IsNil() {
return nil, ErrInvalidEntityType
}
dv = dv.Elem()
mat, elemType = checkMultiArg(dv)
if mat == multiArgTypeInvalid || mat == multiArgTypeInterface {
return nil, ErrInvalidEntityType
}
}
var keys []*Key
for t := q.Run(c); ; {
k, e, err := t.next()
if err == Done {
break
}
if err != nil {
return keys, err
}
if !q.keysOnly {
ev := reflect.New(elemType)
if elemType.Kind() == reflect.Map {
// This is a special case. The zero values of a map type are
// not immediately useful; they have to be make'd.
//
// Funcs and channels are similar, in that a zero value is not useful,
// but even a freshly make'd channel isn't useful: there's no fixed
// channel buffer size that is always going to be large enough, and
// there's no goroutine to drain the other end. Theoretically, these
// types could be supported, for example by sniffing for a constructor
// method or requiring prior registration, but for now it's not a
// frequent enough concern to be worth it. Programmers can work around
// it by explicitly using Iterator.Next instead of the Query.GetAll
// convenience method.
x := reflect.MakeMap(elemType)
ev.Elem().Set(x)
}
if err = loadEntity(ev.Interface(), e); err != nil {
if _, ok := err.(*ErrFieldMismatch); ok {
// We continue loading entities even in the face of field mismatch errors.
// If we encounter any other error, that other error is returned. Otherwise,
// an ErrFieldMismatch is returned.
errFieldMismatch = err
} else {
return keys, err
}
}
if mat != multiArgTypeStructPtr {
ev = ev.Elem()
}
dv.Set(reflect.Append(dv, ev))
}
keys = append(keys, k)
}
return keys, errFieldMismatch
}
// Run runs the query in the given context.
func (q *Query) Run(c context.Context) *Iterator {
if q.err != nil {
return &Iterator{err: q.err}
}
t := &Iterator{
c: c,
limit: q.limit,
count: q.count,
q: q,
prevCC: q.start,
}
var req pb.Query
if err := q.toProto(&req, internal.FullyQualifiedAppID(c)); err != nil {
t.err = err
return t
}
if err := internal.Call(c, "datastore_v3", "RunQuery", &req, &t.res); err != nil {
t.err = err
return t
}
offset := q.offset - t.res.GetSkippedResults()
var count int32
if t.count > 0 && (t.limit < 0 || t.count < t.limit) {
count = t.count
} else {
count = t.limit
}
for offset > 0 && t.res.GetMoreResults() {
t.prevCC = t.res.CompiledCursor
if err := callNext(t.c, &t.res, offset, count); err != nil {
t.err = err
break
}
skip := t.res.GetSkippedResults()
if skip < 0 {
t.err = errors.New("datastore: internal error: negative number of skipped_results")
break
}
offset -= skip
}
if offset < 0 {
t.err = errors.New("datastore: internal error: query offset was overshot")
}
return t
}
// Iterator is the result of running a query.
type Iterator struct {
c context.Context
err error
// res is the result of the most recent RunQuery or Next API call.
res pb.QueryResult
// i is how many elements of res.Result we have iterated over.
i int
// limit is the limit on the number of results this iterator should return.
// A negative value means unlimited.
limit int32
// count is the number of results this iterator should fetch at once. This
// should be equal to or greater than zero.
count int32
// q is the original query which yielded this iterator.
q *Query
// prevCC is the compiled cursor that marks the end of the previous batch
// of results.
prevCC *pb.CompiledCursor
}
// Done is returned when a query iteration has completed.
var Done = errors.New("datastore: query has no more results")
// Next returns the key of the next result. When there are no more results,
// Done is returned as the error.
//
// If the query is not keys only and dst is non-nil, it also loads the entity
// stored for that key into the struct pointer or PropertyLoadSaver dst, with
// the same semantics and possible errors as for the Get function.
func (t *Iterator) Next(dst interface{}) (*Key, error) {
k, e, err := t.next()
if err != nil {
return nil, err
}
if dst != nil && !t.q.keysOnly {
err = loadEntity(dst, e)
}
return k, err
}
func (t *Iterator) next() (*Key, *pb.EntityProto, error) {
if t.err != nil {
return nil, nil, t.err
}
// Issue datastore_v3/Next RPCs as necessary.
for t.i == len(t.res.Result) {
if !t.res.GetMoreResults() {
t.err = Done
return nil, nil, t.err
}
t.prevCC = t.res.CompiledCursor
var count int32
if t.count > 0 && (t.limit < 0 || t.count < t.limit) {
count = t.count
} else {
count = t.limit
}
if err := callNext(t.c, &t.res, 0, count); err != nil {
t.err = err
return nil, nil, t.err
}
if t.res.GetSkippedResults() != 0 {
t.err = errors.New("datastore: internal error: iterator has skipped results")
return nil, nil, t.err
}
t.i = 0
if t.limit >= 0 {
t.limit -= int32(len(t.res.Result))
if t.limit < 0 {
t.err = errors.New("datastore: internal error: query returned more results than the limit")
return nil, nil, t.err
}
}
}
// Extract the key from the t.i'th element of t.res.Result.
e := t.res.Result[t.i]
t.i++
if e.Key == nil {
return nil, nil, errors.New("datastore: internal error: server did not return a key")
}
k, err := protoToKey(e.Key)
if err != nil || k.Incomplete() {
return nil, nil, errors.New("datastore: internal error: server returned an invalid key")
}
return k, e, nil
}
// Cursor returns a cursor for the iterator's current location.
func (t *Iterator) Cursor() (Cursor, error) {
if t.err != nil && t.err != Done {
return Cursor{}, t.err
}
// If we are at either end of the current batch of results,
// return the compiled cursor at that end.
skipped := t.res.GetSkippedResults()
if t.i == 0 && skipped == 0 {
if t.prevCC == nil {
// A nil pointer (of type *pb.CompiledCursor) means no constraint:
// passing it as the end cursor of a new query means unlimited results
// (glossing over the integer limit parameter for now).
// A non-nil pointer to an empty pb.CompiledCursor means the start:
// passing it as the end cursor of a new query means 0 results.
// If prevCC was nil, then the original query had no start cursor, but
// Iterator.Cursor should return "the start" instead of unlimited.
return Cursor{&zeroCC}, nil
}
return Cursor{t.prevCC}, nil
}
if t.i == len(t.res.Result) {
return Cursor{t.res.CompiledCursor}, nil
}
// Otherwise, re-run the query offset to this iterator's position, starting from
// the most recent compiled cursor. This is done on a best-effort basis, as it
// is racy; if a concurrent process has added or removed entities, then the
// cursor returned may be inconsistent.
q := t.q.clone()
q.start = t.prevCC
q.offset = skipped + int32(t.i)
q.limit = 0
q.keysOnly = len(q.projection) == 0
t1 := q.Run(t.c)
_, _, err := t1.next()
if err != Done {
if err == nil {
err = fmt.Errorf("datastore: internal error: zero-limit query did not have zero results")
}
return Cursor{}, err
}
return Cursor{t1.res.CompiledCursor}, nil
}
var zeroCC pb.CompiledCursor
// Cursor is an iterator's position. It can be converted to and from an opaque
// string. A cursor can be used from different HTTP requests, but only with a
// query with the same kind, ancestor, filter and order constraints.
type Cursor struct {
cc *pb.CompiledCursor
}
// String returns a base-64 string representation of a cursor.
func (c Cursor) String() string {
if c.cc == nil {
return ""
}
b, err := proto.Marshal(c.cc)
if err != nil {
// The only way to construct a Cursor with a non-nil cc field is to
// unmarshal from the byte representation. We panic if the unmarshal
// succeeds but the marshaling of the unchanged protobuf value fails.
panic(fmt.Sprintf("datastore: internal error: malformed cursor: %v", err))
}
return strings.TrimRight(base64.URLEncoding.EncodeToString(b), "=")
}
// Decode decodes a cursor from its base-64 string representation.
func DecodeCursor(s string) (Cursor, error) {
if s == "" {
return Cursor{&zeroCC}, nil
}
if n := len(s) % 4; n != 0 {
s += strings.Repeat("=", 4-n)
}
b, err := base64.URLEncoding.DecodeString(s)
if err != nil {
return Cursor{}, err
}
cc := &pb.CompiledCursor{}
if err := proto.Unmarshal(b, cc); err != nil {
return Cursor{}, err
}
return Cursor{cc}, nil
}
appengine-1.6.7/datastore/query_test.go 0000664 0000000 0000000 00000034565 13741174344 0020241 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"errors"
"fmt"
"reflect"
"strings"
"testing"
"github.com/golang/protobuf/proto"
"google.golang.org/appengine/internal"
"google.golang.org/appengine/internal/aetesting"
pb "google.golang.org/appengine/internal/datastore"
)
var (
path1 = &pb.Path{
Element: []*pb.Path_Element{
{
Type: proto.String("Gopher"),
Id: proto.Int64(6),
},
},
}
path2 = &pb.Path{
Element: []*pb.Path_Element{
{
Type: proto.String("Gopher"),
Id: proto.Int64(6),
},
{
Type: proto.String("Gopher"),
Id: proto.Int64(8),
},
},
}
)
func fakeRunQuery(in *pb.Query, out *pb.QueryResult) error {
expectedIn := &pb.Query{
App: proto.String("dev~fake-app"),
Kind: proto.String("Gopher"),
Compile: proto.Bool(true),
}
if !proto.Equal(in, expectedIn) {
return fmt.Errorf("unsupported argument: got %v want %v", in, expectedIn)
}
*out = pb.QueryResult{
Result: []*pb.EntityProto{
{
Key: &pb.Reference{
App: proto.String("s~test-app"),
Path: path1,
},
EntityGroup: path1,
Property: []*pb.Property{
{
Meaning: pb.Property_TEXT.Enum(),
Name: proto.String("Name"),
Value: &pb.PropertyValue{
StringValue: proto.String("George"),
},
},
{
Name: proto.String("Height"),
Value: &pb.PropertyValue{
Int64Value: proto.Int64(32),
},
},
},
},
{
Key: &pb.Reference{
App: proto.String("s~test-app"),
Path: path2,
},
EntityGroup: path1, // ancestor is George
Property: []*pb.Property{
{
Meaning: pb.Property_TEXT.Enum(),
Name: proto.String("Name"),
Value: &pb.PropertyValue{
StringValue: proto.String("Rufus"),
},
},
// No height for Rufus.
},
},
},
MoreResults: proto.Bool(false),
}
return nil
}
type StructThatImplementsPLS struct{}
func (StructThatImplementsPLS) Load(p []Property) error { return nil }
func (StructThatImplementsPLS) Save() ([]Property, error) { return nil, nil }
var _ PropertyLoadSaver = StructThatImplementsPLS{}
type StructPtrThatImplementsPLS struct{}
func (*StructPtrThatImplementsPLS) Load(p []Property) error { return nil }
func (*StructPtrThatImplementsPLS) Save() ([]Property, error) { return nil, nil }
var _ PropertyLoadSaver = &StructPtrThatImplementsPLS{}
type PropertyMap map[string]Property
func (m PropertyMap) Load(props []Property) error {
for _, p := range props {
if p.Multiple {
return errors.New("PropertyMap does not support multiple properties")
}
m[p.Name] = p
}
return nil
}
func (m PropertyMap) Save() ([]Property, error) {
props := make([]Property, 0, len(m))
for _, p := range m {
if p.Multiple {
return nil, errors.New("PropertyMap does not support multiple properties")
}
props = append(props, p)
}
return props, nil
}
var _ PropertyLoadSaver = PropertyMap{}
type Gopher struct {
Name string
Height int
}
// typeOfEmptyInterface is the type of interface{}, but we can't use
// reflect.TypeOf((interface{})(nil)) directly because TypeOf takes an
// interface{}.
var typeOfEmptyInterface = reflect.TypeOf((*interface{})(nil)).Elem()
func TestCheckMultiArg(t *testing.T) {
testCases := []struct {
v interface{}
mat multiArgType
elemType reflect.Type
}{
// Invalid cases.
{nil, multiArgTypeInvalid, nil},
{Gopher{}, multiArgTypeInvalid, nil},
{&Gopher{}, multiArgTypeInvalid, nil},
{PropertyList{}, multiArgTypeInvalid, nil}, // This is a special case.
{PropertyMap{}, multiArgTypeInvalid, nil},
{[]*PropertyList(nil), multiArgTypeInvalid, nil},
{[]*PropertyMap(nil), multiArgTypeInvalid, nil},
{[]**Gopher(nil), multiArgTypeInvalid, nil},
{[]*interface{}(nil), multiArgTypeInvalid, nil},
// Valid cases.
{
[]PropertyList(nil),
multiArgTypePropertyLoadSaver,
reflect.TypeOf(PropertyList{}),
},
{
[]PropertyMap(nil),
multiArgTypePropertyLoadSaver,
reflect.TypeOf(PropertyMap{}),
},
{
[]StructThatImplementsPLS(nil),
multiArgTypePropertyLoadSaver,
reflect.TypeOf(StructThatImplementsPLS{}),
},
{
[]StructPtrThatImplementsPLS(nil),
multiArgTypePropertyLoadSaver,
reflect.TypeOf(StructPtrThatImplementsPLS{}),
},
{
[]Gopher(nil),
multiArgTypeStruct,
reflect.TypeOf(Gopher{}),
},
{
[]*Gopher(nil),
multiArgTypeStructPtr,
reflect.TypeOf(Gopher{}),
},
{
[]interface{}(nil),
multiArgTypeInterface,
typeOfEmptyInterface,
},
}
for _, tc := range testCases {
mat, elemType := checkMultiArg(reflect.ValueOf(tc.v))
if mat != tc.mat || elemType != tc.elemType {
t.Errorf("checkMultiArg(%T): got %v, %v want %v, %v",
tc.v, mat, elemType, tc.mat, tc.elemType)
}
}
}
func TestSimpleQuery(t *testing.T) {
struct1 := Gopher{Name: "George", Height: 32}
struct2 := Gopher{Name: "Rufus"}
pList1 := PropertyList{
{
Name: "Name",
Value: "George",
},
{
Name: "Height",
Value: int64(32),
},
}
pList2 := PropertyList{
{
Name: "Name",
Value: "Rufus",
},
}
pMap1 := PropertyMap{
"Name": Property{
Name: "Name",
Value: "George",
},
"Height": Property{
Name: "Height",
Value: int64(32),
},
}
pMap2 := PropertyMap{
"Name": Property{
Name: "Name",
Value: "Rufus",
},
}
testCases := []struct {
dst interface{}
want interface{}
}{
// The destination must have type *[]P, *[]S or *[]*S, for some non-interface
// type P such that *P implements PropertyLoadSaver, or for some struct type S.
{new([]Gopher), &[]Gopher{struct1, struct2}},
{new([]*Gopher), &[]*Gopher{&struct1, &struct2}},
{new([]PropertyList), &[]PropertyList{pList1, pList2}},
{new([]PropertyMap), &[]PropertyMap{pMap1, pMap2}},
// Any other destination type is invalid.
{0, nil},
{Gopher{}, nil},
{PropertyList{}, nil},
{PropertyMap{}, nil},
{[]int{}, nil},
{[]Gopher{}, nil},
{[]PropertyList{}, nil},
{new(int), nil},
{new(Gopher), nil},
{new(PropertyList), nil}, // This is a special case.
{new(PropertyMap), nil},
{new([]int), nil},
{new([]map[int]int), nil},
{new([]map[string]Property), nil},
{new([]map[string]interface{}), nil},
{new([]*int), nil},
{new([]*map[int]int), nil},
{new([]*map[string]Property), nil},
{new([]*map[string]interface{}), nil},
{new([]**Gopher), nil},
{new([]*PropertyList), nil},
{new([]*PropertyMap), nil},
}
for _, tc := range testCases {
nCall := 0
c := aetesting.FakeSingleContext(t, "datastore_v3", "RunQuery", func(in *pb.Query, out *pb.QueryResult) error {
nCall++
return fakeRunQuery(in, out)
})
c = internal.WithAppIDOverride(c, "dev~fake-app")
var (
expectedErr error
expectedNCall int
)
if tc.want == nil {
expectedErr = ErrInvalidEntityType
} else {
expectedNCall = 1
}
keys, err := NewQuery("Gopher").GetAll(c, tc.dst)
if err != expectedErr {
t.Errorf("dst type %T: got error [%v], want [%v]", tc.dst, err, expectedErr)
continue
}
if nCall != expectedNCall {
t.Errorf("dst type %T: Context.Call was called an incorrect number of times: got %d want %d", tc.dst, nCall, expectedNCall)
continue
}
if err != nil {
continue
}
key1 := NewKey(c, "Gopher", "", 6, nil)
expectedKeys := []*Key{
key1,
NewKey(c, "Gopher", "", 8, key1),
}
if l1, l2 := len(keys), len(expectedKeys); l1 != l2 {
t.Errorf("dst type %T: got %d keys, want %d keys", tc.dst, l1, l2)
continue
}
for i, key := range keys {
if key.AppID() != "s~test-app" {
t.Errorf(`dst type %T: Key #%d's AppID = %q, want "s~test-app"`, tc.dst, i, key.AppID())
continue
}
if !keysEqual(key, expectedKeys[i]) {
t.Errorf("dst type %T: got key #%d %v, want %v", tc.dst, i, key, expectedKeys[i])
continue
}
}
if !reflect.DeepEqual(tc.dst, tc.want) {
t.Errorf("dst type %T: Entities got %+v, want %+v", tc.dst, tc.dst, tc.want)
continue
}
}
}
// keysEqual is like (*Key).Equal, but ignores the App ID.
func keysEqual(a, b *Key) bool {
for a != nil && b != nil {
if a.Kind() != b.Kind() || a.StringID() != b.StringID() || a.IntID() != b.IntID() {
return false
}
a, b = a.Parent(), b.Parent()
}
return a == b
}
func TestQueriesAreImmutable(t *testing.T) {
// Test that deriving q2 from q1 does not modify q1.
q0 := NewQuery("foo")
q1 := NewQuery("foo")
q2 := q1.Offset(2)
if !reflect.DeepEqual(q0, q1) {
t.Errorf("q0 and q1 were not equal")
}
if reflect.DeepEqual(q1, q2) {
t.Errorf("q1 and q2 were equal")
}
// Test that deriving from q4 twice does not conflict, even though
// q4 has a long list of order clauses. This tests that the arrays
// backed by a query's slice of orders are not shared.
f := func() *Query {
q := NewQuery("bar")
// 47 is an ugly number that is unlikely to be near a re-allocation
// point in repeated append calls. For example, it's not near a power
// of 2 or a multiple of 10.
for i := 0; i < 47; i++ {
q = q.Order(fmt.Sprintf("x%d", i))
}
return q
}
q3 := f().Order("y")
q4 := f()
q5 := q4.Order("y")
q6 := q4.Order("z")
if !reflect.DeepEqual(q3, q5) {
t.Errorf("q3 and q5 were not equal")
}
if reflect.DeepEqual(q5, q6) {
t.Errorf("q5 and q6 were equal")
}
}
func TestFilterParser(t *testing.T) {
testCases := []struct {
filterStr string
wantOK bool
wantFieldName string
wantOp operator
}{
// Supported ops.
{"x<", true, "x", lessThan},
{"x <", true, "x", lessThan},
{"x <", true, "x", lessThan},
{" x < ", true, "x", lessThan},
{"x <=", true, "x", lessEq},
{"x =", true, "x", equal},
{"x >=", true, "x", greaterEq},
{"x >", true, "x", greaterThan},
{"in >", true, "in", greaterThan},
{"in>", true, "in", greaterThan},
// Valid but (currently) unsupported ops.
{"x!=", false, "", 0},
{"x !=", false, "", 0},
{" x != ", false, "", 0},
{"x IN", false, "", 0},
{"x in", false, "", 0},
// Invalid ops.
{"x EQ", false, "", 0},
{"x lt", false, "", 0},
{"x <>", false, "", 0},
{"x >>", false, "", 0},
{"x ==", false, "", 0},
{"x =<", false, "", 0},
{"x =>", false, "", 0},
{"x !", false, "", 0},
{"x ", false, "", 0},
{"x", false, "", 0},
}
for _, tc := range testCases {
q := NewQuery("foo").Filter(tc.filterStr, 42)
if ok := q.err == nil; ok != tc.wantOK {
t.Errorf("%q: ok=%t, want %t", tc.filterStr, ok, tc.wantOK)
continue
}
if !tc.wantOK {
continue
}
if len(q.filter) != 1 {
t.Errorf("%q: len=%d, want %d", tc.filterStr, len(q.filter), 1)
continue
}
got, want := q.filter[0], filter{tc.wantFieldName, tc.wantOp, 42}
if got != want {
t.Errorf("%q: got %v, want %v", tc.filterStr, got, want)
continue
}
}
}
func TestQueryToProto(t *testing.T) {
// The context is required to make Keys for the test cases.
var got *pb.Query
NoErr := errors.New("No error")
c := aetesting.FakeSingleContext(t, "datastore_v3", "RunQuery", func(in *pb.Query, out *pb.QueryResult) error {
got = in
return NoErr // return a non-nil error so Run doesn't keep going.
})
c = internal.WithAppIDOverride(c, "dev~fake-app")
testCases := []struct {
desc string
query *Query
want *pb.Query
err string
}{
{
desc: "empty",
query: NewQuery(""),
want: &pb.Query{},
},
{
desc: "standard query",
query: NewQuery("kind").Order("-I").Filter("I >", 17).Filter("U =", "Dave").Limit(7).Offset(42).BatchSize(5),
want: &pb.Query{
Kind: proto.String("kind"),
Filter: []*pb.Query_Filter{
{
Op: pb.Query_Filter_GREATER_THAN.Enum(),
Property: []*pb.Property{
{
Name: proto.String("I"),
Value: &pb.PropertyValue{Int64Value: proto.Int64(17)},
Multiple: proto.Bool(false),
},
},
},
{
Op: pb.Query_Filter_EQUAL.Enum(),
Property: []*pb.Property{
{
Name: proto.String("U"),
Value: &pb.PropertyValue{StringValue: proto.String("Dave")},
Multiple: proto.Bool(false),
},
},
},
},
Order: []*pb.Query_Order{
{
Property: proto.String("I"),
Direction: pb.Query_Order_DESCENDING.Enum(),
},
},
Limit: proto.Int32(7),
Offset: proto.Int32(42),
Count: proto.Int32(5),
},
},
{
desc: "ancestor",
query: NewQuery("").Ancestor(NewKey(c, "kind", "Mummy", 0, nil)),
want: &pb.Query{
Ancestor: &pb.Reference{
App: proto.String("dev~fake-app"),
Path: &pb.Path{
Element: []*pb.Path_Element{{Type: proto.String("kind"), Name: proto.String("Mummy")}},
},
},
},
},
{
desc: "projection",
query: NewQuery("").Project("A", "B"),
want: &pb.Query{
PropertyName: []string{"A", "B"},
},
},
{
desc: "projection with distinct",
query: NewQuery("").Project("A", "B").Distinct(),
want: &pb.Query{
PropertyName: []string{"A", "B"},
GroupByPropertyName: []string{"A", "B"},
},
},
{
desc: "distinct on",
query: NewQuery("").Project("A", "B").DistinctOn("A"),
want: &pb.Query{
PropertyName: []string{"A", "B"},
GroupByPropertyName: []string{"A"},
},
},
{
desc: "keys only",
query: NewQuery("").KeysOnly(),
want: &pb.Query{
KeysOnly: proto.Bool(true),
RequirePerfectPlan: proto.Bool(true),
},
},
{
desc: "empty filter",
query: NewQuery("kind").Filter("=", 17),
err: "empty query filter field nam",
},
{
desc: "bad filter type",
query: NewQuery("kind").Filter("M =", map[string]bool{}),
err: "bad query filter value type",
},
{
desc: "bad filter operator",
query: NewQuery("kind").Filter("I <<=", 17),
err: `invalid operator "<<=" in filter "I <<="`,
},
{
desc: "empty order",
query: NewQuery("kind").Order(""),
err: "empty order",
},
{
desc: "bad order direction",
query: NewQuery("kind").Order("+I"),
err: `invalid order: "+I`,
},
}
for _, tt := range testCases {
got = nil
if _, err := tt.query.Run(c).Next(nil); err != NoErr {
if tt.err == "" || !strings.Contains(err.Error(), tt.err) {
t.Errorf("%s: error %v, want %q", tt.desc, err, tt.err)
}
continue
}
if tt.err != "" {
t.Errorf("%s: no error, want %q", tt.desc, tt.err)
continue
}
// Fields that are common to all protos.
tt.want.App = proto.String("dev~fake-app")
tt.want.Compile = proto.Bool(true)
if !proto.Equal(got, tt.want) {
t.Errorf("%s:\ngot %v\nwant %v", tt.desc, got, tt.want)
}
}
}
appengine-1.6.7/datastore/save.go 0000664 0000000 0000000 00000022312 13741174344 0016756 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"errors"
"fmt"
"math"
"reflect"
"time"
"github.com/golang/protobuf/proto"
"google.golang.org/appengine"
pb "google.golang.org/appengine/internal/datastore"
)
func toUnixMicro(t time.Time) int64 {
// We cannot use t.UnixNano() / 1e3 because we want to handle times more than
// 2^63 nanoseconds (which is about 292 years) away from 1970, and those cannot
// be represented in the numerator of a single int64 divide.
return t.Unix()*1e6 + int64(t.Nanosecond()/1e3)
}
func fromUnixMicro(t int64) time.Time {
return time.Unix(t/1e6, (t%1e6)*1e3).UTC()
}
var (
minTime = time.Unix(int64(math.MinInt64)/1e6, (int64(math.MinInt64)%1e6)*1e3)
maxTime = time.Unix(int64(math.MaxInt64)/1e6, (int64(math.MaxInt64)%1e6)*1e3)
)
// valueToProto converts a named value to a newly allocated Property.
// The returned error string is empty on success.
func valueToProto(defaultAppID, name string, v reflect.Value, multiple bool) (p *pb.Property, errStr string) {
var (
pv pb.PropertyValue
unsupported bool
)
switch v.Kind() {
case reflect.Invalid:
// No-op.
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
pv.Int64Value = proto.Int64(v.Int())
case reflect.Bool:
pv.BooleanValue = proto.Bool(v.Bool())
case reflect.String:
pv.StringValue = proto.String(v.String())
case reflect.Float32, reflect.Float64:
pv.DoubleValue = proto.Float64(v.Float())
case reflect.Ptr:
if k, ok := v.Interface().(*Key); ok {
if k != nil {
pv.Referencevalue = keyToReferenceValue(defaultAppID, k)
}
} else {
unsupported = true
}
case reflect.Struct:
switch t := v.Interface().(type) {
case time.Time:
if t.Before(minTime) || t.After(maxTime) {
return nil, "time value out of range"
}
pv.Int64Value = proto.Int64(toUnixMicro(t))
case appengine.GeoPoint:
if !t.Valid() {
return nil, "invalid GeoPoint value"
}
// NOTE: Strangely, latitude maps to X, longitude to Y.
pv.Pointvalue = &pb.PropertyValue_PointValue{X: &t.Lat, Y: &t.Lng}
default:
unsupported = true
}
case reflect.Slice:
if b, ok := v.Interface().([]byte); ok {
pv.StringValue = proto.String(string(b))
} else {
// nvToProto should already catch slice values.
// If we get here, we have a slice of slice values.
unsupported = true
}
default:
unsupported = true
}
if unsupported {
return nil, "unsupported datastore value type: " + v.Type().String()
}
p = &pb.Property{
Name: proto.String(name),
Value: &pv,
Multiple: proto.Bool(multiple),
}
if v.IsValid() {
switch v.Interface().(type) {
case []byte:
p.Meaning = pb.Property_BLOB.Enum()
case ByteString:
p.Meaning = pb.Property_BYTESTRING.Enum()
case appengine.BlobKey:
p.Meaning = pb.Property_BLOBKEY.Enum()
case time.Time:
p.Meaning = pb.Property_GD_WHEN.Enum()
case appengine.GeoPoint:
p.Meaning = pb.Property_GEORSS_POINT.Enum()
}
}
return p, ""
}
type saveOpts struct {
noIndex bool
multiple bool
omitEmpty bool
}
// saveEntity saves an EntityProto into a PropertyLoadSaver or struct pointer.
func saveEntity(defaultAppID string, key *Key, src interface{}) (*pb.EntityProto, error) {
var err error
var props []Property
if e, ok := src.(PropertyLoadSaver); ok {
props, err = e.Save()
} else {
props, err = SaveStruct(src)
}
if err != nil {
return nil, err
}
return propertiesToProto(defaultAppID, key, props)
}
func saveStructProperty(props *[]Property, name string, opts saveOpts, v reflect.Value) error {
if opts.omitEmpty && isEmptyValue(v) {
return nil
}
p := Property{
Name: name,
NoIndex: opts.noIndex,
Multiple: opts.multiple,
}
switch x := v.Interface().(type) {
case *Key:
p.Value = x
case time.Time:
p.Value = x
case appengine.BlobKey:
p.Value = x
case appengine.GeoPoint:
p.Value = x
case ByteString:
p.Value = x
default:
switch v.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
p.Value = v.Int()
case reflect.Bool:
p.Value = v.Bool()
case reflect.String:
p.Value = v.String()
case reflect.Float32, reflect.Float64:
p.Value = v.Float()
case reflect.Slice:
if v.Type().Elem().Kind() == reflect.Uint8 {
p.NoIndex = true
p.Value = v.Bytes()
}
case reflect.Struct:
if !v.CanAddr() {
return fmt.Errorf("datastore: unsupported struct field: value is unaddressable")
}
sub, err := newStructPLS(v.Addr().Interface())
if err != nil {
return fmt.Errorf("datastore: unsupported struct field: %v", err)
}
return sub.save(props, name+".", opts)
}
}
if p.Value == nil {
return fmt.Errorf("datastore: unsupported struct field type: %v", v.Type())
}
*props = append(*props, p)
return nil
}
func (s structPLS) Save() ([]Property, error) {
var props []Property
if err := s.save(&props, "", saveOpts{}); err != nil {
return nil, err
}
return props, nil
}
func (s structPLS) save(props *[]Property, prefix string, opts saveOpts) error {
for name, f := range s.codec.fields {
name = prefix + name
v := s.v.FieldByIndex(f.path)
if !v.IsValid() || !v.CanSet() {
continue
}
var opts1 saveOpts
opts1.noIndex = opts.noIndex || f.noIndex
opts1.multiple = opts.multiple
opts1.omitEmpty = f.omitEmpty // don't propagate
// For slice fields that aren't []byte, save each element.
if v.Kind() == reflect.Slice && v.Type().Elem().Kind() != reflect.Uint8 {
opts1.multiple = true
for j := 0; j < v.Len(); j++ {
if err := saveStructProperty(props, name, opts1, v.Index(j)); err != nil {
return err
}
}
continue
}
// Otherwise, save the field itself.
if err := saveStructProperty(props, name, opts1, v); err != nil {
return err
}
}
return nil
}
func propertiesToProto(defaultAppID string, key *Key, props []Property) (*pb.EntityProto, error) {
e := &pb.EntityProto{
Key: keyToProto(defaultAppID, key),
}
if key.parent == nil {
e.EntityGroup = &pb.Path{}
} else {
e.EntityGroup = keyToProto(defaultAppID, key.root()).Path
}
prevMultiple := make(map[string]bool)
for _, p := range props {
if pm, ok := prevMultiple[p.Name]; ok {
if !pm || !p.Multiple {
return nil, fmt.Errorf("datastore: multiple Properties with Name %q, but Multiple is false", p.Name)
}
} else {
prevMultiple[p.Name] = p.Multiple
}
x := &pb.Property{
Name: proto.String(p.Name),
Value: new(pb.PropertyValue),
Multiple: proto.Bool(p.Multiple),
}
switch v := p.Value.(type) {
case int64:
x.Value.Int64Value = proto.Int64(v)
case bool:
x.Value.BooleanValue = proto.Bool(v)
case string:
x.Value.StringValue = proto.String(v)
if p.NoIndex {
x.Meaning = pb.Property_TEXT.Enum()
}
case float64:
x.Value.DoubleValue = proto.Float64(v)
case *Key:
if v != nil {
x.Value.Referencevalue = keyToReferenceValue(defaultAppID, v)
}
case time.Time:
if v.Before(minTime) || v.After(maxTime) {
return nil, fmt.Errorf("datastore: time value out of range")
}
x.Value.Int64Value = proto.Int64(toUnixMicro(v))
x.Meaning = pb.Property_GD_WHEN.Enum()
case appengine.BlobKey:
x.Value.StringValue = proto.String(string(v))
x.Meaning = pb.Property_BLOBKEY.Enum()
case appengine.GeoPoint:
if !v.Valid() {
return nil, fmt.Errorf("datastore: invalid GeoPoint value")
}
// NOTE: Strangely, latitude maps to X, longitude to Y.
x.Value.Pointvalue = &pb.PropertyValue_PointValue{X: &v.Lat, Y: &v.Lng}
x.Meaning = pb.Property_GEORSS_POINT.Enum()
case []byte:
x.Value.StringValue = proto.String(string(v))
x.Meaning = pb.Property_BLOB.Enum()
if !p.NoIndex {
return nil, fmt.Errorf("datastore: cannot index a []byte valued Property with Name %q", p.Name)
}
case ByteString:
x.Value.StringValue = proto.String(string(v))
x.Meaning = pb.Property_BYTESTRING.Enum()
default:
if p.Value != nil {
return nil, fmt.Errorf("datastore: invalid Value type for a Property with Name %q", p.Name)
}
}
if p.NoIndex {
e.RawProperty = append(e.RawProperty, x)
} else {
e.Property = append(e.Property, x)
if len(e.Property) > maxIndexedProperties {
return nil, errors.New("datastore: too many indexed properties")
}
}
}
return e, nil
}
// isEmptyValue is taken from the encoding/json package in the standard library.
func isEmptyValue(v reflect.Value) bool {
switch v.Kind() {
case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
// TODO(performance): Only reflect.String needed, other property types are not supported (copy/paste from json package)
return v.Len() == 0
case reflect.Bool:
return !v.Bool()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return v.Int() == 0
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
// TODO(performance): Uint* are unsupported property types - should be removed (copy/paste from json package)
return v.Uint() == 0
case reflect.Float32, reflect.Float64:
return v.Float() == 0
case reflect.Interface, reflect.Ptr:
return v.IsNil()
case reflect.Struct:
switch x := v.Interface().(type) {
case time.Time:
return x.IsZero()
}
}
return false
}
appengine-1.6.7/datastore/time_test.go 0000664 0000000 0000000 00000003677 13741174344 0020032 0 ustar 00root root 0000000 0000000 // Copyright 2012 Google Inc. All Rights Reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"testing"
"time"
)
func TestUnixMicro(t *testing.T) {
// Test that all these time.Time values survive a round trip to unix micros.
testCases := []time.Time{
{},
time.Date(2, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(23, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(234, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(1000, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(1600, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(1700, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(1800, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(1900, 1, 1, 0, 0, 0, 0, time.UTC),
time.Unix(-1e6, -1000),
time.Unix(-1e6, 0),
time.Unix(-1e6, +1000),
time.Unix(-60, -1000),
time.Unix(-60, 0),
time.Unix(-60, +1000),
time.Unix(-1, -1000),
time.Unix(-1, 0),
time.Unix(-1, +1000),
time.Unix(0, -3000),
time.Unix(0, -2000),
time.Unix(0, -1000),
time.Unix(0, 0),
time.Unix(0, +1000),
time.Unix(0, +2000),
time.Unix(+60, -1000),
time.Unix(+60, 0),
time.Unix(+60, +1000),
time.Unix(+1e6, -1000),
time.Unix(+1e6, 0),
time.Unix(+1e6, +1000),
time.Date(1999, 12, 31, 23, 59, 59, 999000, time.UTC),
time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC),
time.Date(2006, 1, 2, 15, 4, 5, 678000, time.UTC),
time.Date(2009, 11, 10, 23, 0, 0, 0, time.UTC),
time.Date(3456, 1, 1, 0, 0, 0, 0, time.UTC),
}
for _, tc := range testCases {
got := fromUnixMicro(toUnixMicro(tc))
if !got.Equal(tc) {
t.Errorf("got %q, want %q", got, tc)
}
}
// Test that a time.Time that isn't an integral number of microseconds
// is not perfectly reconstructed after a round trip.
t0 := time.Unix(0, 123)
t1 := fromUnixMicro(toUnixMicro(t0))
if t1.Nanosecond()%1000 != 0 || t0.Nanosecond()%1000 == 0 {
t.Errorf("quantization to µs: got %q with %d ns, started with %d ns", t1, t1.Nanosecond(), t0.Nanosecond())
}
}
appengine-1.6.7/datastore/transaction.go 0000664 0000000 0000000 00000006732 13741174344 0020355 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package datastore
import (
"errors"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
pb "google.golang.org/appengine/internal/datastore"
)
func init() {
internal.RegisterTransactionSetter(func(x *pb.Query, t *pb.Transaction) {
x.Transaction = t
})
internal.RegisterTransactionSetter(func(x *pb.GetRequest, t *pb.Transaction) {
x.Transaction = t
})
internal.RegisterTransactionSetter(func(x *pb.PutRequest, t *pb.Transaction) {
x.Transaction = t
})
internal.RegisterTransactionSetter(func(x *pb.DeleteRequest, t *pb.Transaction) {
x.Transaction = t
})
}
// ErrConcurrentTransaction is returned when a transaction is rolled back due
// to a conflict with a concurrent transaction.
var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction")
// RunInTransaction runs f in a transaction. It calls f with a transaction
// context tc that f should use for all App Engine operations.
//
// If f returns nil, RunInTransaction attempts to commit the transaction,
// returning nil if it succeeds. If the commit fails due to a conflicting
// transaction, RunInTransaction retries f, each time with a new transaction
// context. It gives up and returns ErrConcurrentTransaction after three
// failed attempts. The number of attempts can be configured by specifying
// TransactionOptions.Attempts.
//
// If f returns non-nil, then any datastore changes will not be applied and
// RunInTransaction returns that same error. The function f is not retried.
//
// Note that when f returns, the transaction is not yet committed. Calling code
// must be careful not to assume that any of f's changes have been committed
// until RunInTransaction returns nil.
//
// Since f may be called multiple times, f should usually be idempotent.
// datastore.Get is not idempotent when unmarshaling slice fields.
//
// Nested transactions are not supported; c may not be a transaction context.
func RunInTransaction(c context.Context, f func(tc context.Context) error, opts *TransactionOptions) error {
xg := false
if opts != nil {
xg = opts.XG
}
readOnly := false
if opts != nil {
readOnly = opts.ReadOnly
}
attempts := 3
if opts != nil && opts.Attempts > 0 {
attempts = opts.Attempts
}
var t *pb.Transaction
var err error
for i := 0; i < attempts; i++ {
if t, err = internal.RunTransactionOnce(c, f, xg, readOnly, t); err != internal.ErrConcurrentTransaction {
return err
}
}
return ErrConcurrentTransaction
}
// TransactionOptions are the options for running a transaction.
type TransactionOptions struct {
// XG is whether the transaction can cross multiple entity groups. In
// comparison, a single group transaction is one where all datastore keys
// used have the same root key. Note that cross group transactions do not
// have the same behavior as single group transactions. In particular, it
// is much more likely to see partially applied transactions in different
// entity groups, in global queries.
// It is valid to set XG to true even if the transaction is within a
// single entity group.
XG bool
// Attempts controls the number of retries to perform when commits fail
// due to a conflicting transaction. If omitted, it defaults to 3.
Attempts int
// ReadOnly controls whether the transaction is a read only transaction.
// Read only transactions are potentially more efficient.
ReadOnly bool
}
appengine-1.6.7/delay/ 0000775 0000000 0000000 00000000000 13741174344 0014601 5 ustar 00root root 0000000 0000000 appengine-1.6.7/delay/delay.go 0000664 0000000 0000000 00000027275 13741174344 0016243 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
/*
Package delay provides a way to execute code outside the scope of a
user request by using the taskqueue API.
To declare a function that may be executed later, call Func
in a top-level assignment context, passing it an arbitrary string key
and a function whose first argument is of type context.Context.
The key is used to look up the function so it can be called later.
var laterFunc = delay.Func("key", myFunc)
It is also possible to use a function literal.
var laterFunc = delay.Func("key", func(c context.Context, x string) {
// ...
})
To call a function, invoke its Call method.
laterFunc.Call(c, "something")
A function may be called any number of times. If the function has any
return arguments, and the last one is of type error, the function may
return a non-nil error to signal that the function should be retried.
The arguments to functions may be of any type that is encodable by the gob
package. If an argument is of interface type, it is the client's responsibility
to register with the gob package whatever concrete type may be passed for that
argument; see http://golang.org/pkg/gob/#Register for details.
Any errors during initialization or execution of a function will be
logged to the application logs. Error logs that occur during initialization will
be associated with the request that invoked the Call method.
The state of a function invocation that has not yet successfully
executed is preserved by combining the file name in which it is declared
with the string key that was passed to the Func function. Updating an app
with pending function invocations should safe as long as the relevant
functions have the (filename, key) combination preserved. The filename is
parsed according to these rules:
* Paths in package main are shortened to just the file name (github.com/foo/foo.go -> foo.go)
* Paths are stripped to just package paths (/go/src/github.com/foo/bar.go -> github.com/foo/bar.go)
* Module versions are stripped (/go/pkg/mod/github.com/foo/bar@v0.0.0-20181026220418-f595d03440dc/baz.go -> github.com/foo/bar/baz.go)
There is some inherent risk of pending function invocations being lost during
an update that contains large changes. For example, switching from using GOPATH
to go.mod is a large change that may inadvertently cause file paths to change.
The delay package uses the Task Queue API to create tasks that call the
reserved application path "/_ah/queue/go/delay".
This path must not be marked as "login: required" in app.yaml;
it must be marked as "login: admin" or have no access restriction.
*/
package delay // import "google.golang.org/appengine/delay"
import (
"bytes"
stdctx "context"
"encoding/gob"
"errors"
"fmt"
"go/build"
stdlog "log"
"net/http"
"path/filepath"
"reflect"
"regexp"
"runtime"
"strings"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/internal"
"google.golang.org/appengine/log"
"google.golang.org/appengine/taskqueue"
)
// Function represents a function that may have a delayed invocation.
type Function struct {
fv reflect.Value // Kind() == reflect.Func
key string
err error // any error during initialization
}
const (
// The HTTP path for invocations.
path = "/_ah/queue/go/delay"
// Use the default queue.
queue = ""
)
type contextKey int
var (
// registry of all delayed functions
funcs = make(map[string]*Function)
// precomputed types
errorType = reflect.TypeOf((*error)(nil)).Elem()
// errors
errFirstArg = errors.New("first argument must be context.Context")
errOutsideDelayFunc = errors.New("request headers are only available inside a delay.Func")
// context keys
headersContextKey contextKey = 0
stdContextType = reflect.TypeOf((*stdctx.Context)(nil)).Elem()
netContextType = reflect.TypeOf((*context.Context)(nil)).Elem()
)
func isContext(t reflect.Type) bool {
return t == stdContextType || t == netContextType
}
var modVersionPat = regexp.MustCompile("@v[^/]+")
// fileKey finds a stable representation of the caller's file path.
// For calls from package main: strip all leading path entries, leaving just the filename.
// For calls from anywhere else, strip $GOPATH/src, leaving just the package path and file path.
func fileKey(file string) (string, error) {
if !internal.IsSecondGen() {
return file, nil
}
// If the caller is in the same Dir as mainPath, then strip everything but the file name.
if filepath.Dir(file) == internal.MainPath {
return filepath.Base(file), nil
}
// If the path contains "gopath/src/", which is what the builder uses for
// apps which don't use go modules, strip everything up to and including src.
// Or, if the path starts with /tmp/staging, then we're importing a package
// from the app's module (and we must be using go modules), and we have a
// path like /tmp/staging1234/srv/... so strip everything up to and
// including the first /srv/.
// And be sure to look at the GOPATH, for local development.
s := string(filepath.Separator)
for _, s := range []string{filepath.Join("gopath", "src") + s, s + "srv" + s, filepath.Join(build.Default.GOPATH, "src") + s} {
if idx := strings.Index(file, s); idx > 0 {
return file[idx+len(s):], nil
}
}
// Finally, if that all fails then we must be using go modules, and the file is a module,
// so the path looks like /go/pkg/mod/github.com/foo/bar@v0.0.0-20181026220418-f595d03440dc/baz.go
// So... remove everything up to and including mod, plus the @.... version string.
m := "/mod/"
if idx := strings.Index(file, m); idx > 0 {
file = file[idx+len(m):]
} else {
return file, fmt.Errorf("fileKey: unknown file path format for %q", file)
}
return modVersionPat.ReplaceAllString(file, ""), nil
}
// Func declares a new Function. The second argument must be a function with a
// first argument of type context.Context.
// This function must be called at program initialization time. That means it
// must be called in a global variable declaration or from an init function.
// This restriction is necessary because the instance that delays a function
// call may not be the one that executes it. Only the code executed at program
// initialization time is guaranteed to have been run by an instance before it
// receives a request.
func Func(key string, i interface{}) *Function {
f := &Function{fv: reflect.ValueOf(i)}
// Derive unique, somewhat stable key for this func.
_, file, _, _ := runtime.Caller(1)
fk, err := fileKey(file)
if err != nil {
// Not fatal, but log the error
stdlog.Printf("delay: %v", err)
}
f.key = fk + ":" + key
t := f.fv.Type()
if t.Kind() != reflect.Func {
f.err = errors.New("not a function")
return f
}
if t.NumIn() == 0 || !isContext(t.In(0)) {
f.err = errFirstArg
return f
}
// Register the function's arguments with the gob package.
// This is required because they are marshaled inside a []interface{}.
// gob.Register only expects to be called during initialization;
// that's fine because this function expects the same.
for i := 0; i < t.NumIn(); i++ {
// Only concrete types may be registered. If the argument has
// interface type, the client is resposible for registering the
// concrete types it will hold.
if t.In(i).Kind() == reflect.Interface {
continue
}
gob.Register(reflect.Zero(t.In(i)).Interface())
}
if old := funcs[f.key]; old != nil {
old.err = fmt.Errorf("multiple functions registered for %s in %s", key, file)
}
funcs[f.key] = f
return f
}
type invocation struct {
Key string
Args []interface{}
}
// Call invokes a delayed function.
// err := f.Call(c, ...)
// is equivalent to
// t, _ := f.Task(...)
// _, err := taskqueue.Add(c, t, "")
func (f *Function) Call(c context.Context, args ...interface{}) error {
t, err := f.Task(args...)
if err != nil {
return err
}
_, err = taskqueueAdder(c, t, queue)
return err
}
// Task creates a Task that will invoke the function.
// Its parameters may be tweaked before adding it to a queue.
// Users should not modify the Path or Payload fields of the returned Task.
func (f *Function) Task(args ...interface{}) (*taskqueue.Task, error) {
if f.err != nil {
return nil, fmt.Errorf("delay: func is invalid: %v", f.err)
}
nArgs := len(args) + 1 // +1 for the context.Context
ft := f.fv.Type()
minArgs := ft.NumIn()
if ft.IsVariadic() {
minArgs--
}
if nArgs < minArgs {
return nil, fmt.Errorf("delay: too few arguments to func: %d < %d", nArgs, minArgs)
}
if !ft.IsVariadic() && nArgs > minArgs {
return nil, fmt.Errorf("delay: too many arguments to func: %d > %d", nArgs, minArgs)
}
// Check arg types.
for i := 1; i < nArgs; i++ {
at := reflect.TypeOf(args[i-1])
var dt reflect.Type
if i < minArgs {
// not a variadic arg
dt = ft.In(i)
} else {
// a variadic arg
dt = ft.In(minArgs).Elem()
}
// nil arguments won't have a type, so they need special handling.
if at == nil {
// nil interface
switch dt.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice:
continue // may be nil
}
return nil, fmt.Errorf("delay: argument %d has wrong type: %v is not nilable", i, dt)
}
switch at.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Ptr, reflect.Slice:
av := reflect.ValueOf(args[i-1])
if av.IsNil() {
// nil value in interface; not supported by gob, so we replace it
// with a nil interface value
args[i-1] = nil
}
}
if !at.AssignableTo(dt) {
return nil, fmt.Errorf("delay: argument %d has wrong type: %v is not assignable to %v", i, at, dt)
}
}
inv := invocation{
Key: f.key,
Args: args,
}
buf := new(bytes.Buffer)
if err := gob.NewEncoder(buf).Encode(inv); err != nil {
return nil, fmt.Errorf("delay: gob encoding failed: %v", err)
}
return &taskqueue.Task{
Path: path,
Payload: buf.Bytes(),
}, nil
}
// Request returns the special task-queue HTTP request headers for the current
// task queue handler. Returns an error if called from outside a delay.Func.
func RequestHeaders(c context.Context) (*taskqueue.RequestHeaders, error) {
if ret, ok := c.Value(headersContextKey).(*taskqueue.RequestHeaders); ok {
return ret, nil
}
return nil, errOutsideDelayFunc
}
var taskqueueAdder = taskqueue.Add // for testing
func init() {
http.HandleFunc(path, func(w http.ResponseWriter, req *http.Request) {
runFunc(appengine.NewContext(req), w, req)
})
}
func runFunc(c context.Context, w http.ResponseWriter, req *http.Request) {
defer req.Body.Close()
c = context.WithValue(c, headersContextKey, taskqueue.ParseRequestHeaders(req.Header))
var inv invocation
if err := gob.NewDecoder(req.Body).Decode(&inv); err != nil {
log.Errorf(c, "delay: failed decoding task payload: %v", err)
log.Warningf(c, "delay: dropping task")
return
}
f := funcs[inv.Key]
if f == nil {
log.Errorf(c, "delay: no func with key %q found", inv.Key)
log.Warningf(c, "delay: dropping task")
return
}
ft := f.fv.Type()
in := []reflect.Value{reflect.ValueOf(c)}
for _, arg := range inv.Args {
var v reflect.Value
if arg != nil {
v = reflect.ValueOf(arg)
} else {
// Task was passed a nil argument, so we must construct
// the zero value for the argument here.
n := len(in) // we're constructing the nth argument
var at reflect.Type
if !ft.IsVariadic() || n < ft.NumIn()-1 {
at = ft.In(n)
} else {
at = ft.In(ft.NumIn() - 1).Elem()
}
v = reflect.Zero(at)
}
in = append(in, v)
}
out := f.fv.Call(in)
if n := ft.NumOut(); n > 0 && ft.Out(n-1) == errorType {
if errv := out[n-1]; !errv.IsNil() {
log.Errorf(c, "delay: func failed (will retry): %v", errv.Interface())
w.WriteHeader(http.StatusInternalServerError)
return
}
}
}
appengine-1.6.7/delay/delay_test.go 0000664 0000000 0000000 00000034335 13741174344 0017275 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
package delay
import (
"bytes"
stdctx "context"
"encoding/gob"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"testing"
"github.com/golang/protobuf/proto"
"golang.org/x/net/context"
"google.golang.org/appengine/internal"
"google.golang.org/appengine/taskqueue"
)
type CustomType struct {
N int
}
type CustomInterface interface {
N() int
}
type CustomImpl int
func (c CustomImpl) N() int { return int(c) }
// CustomImpl needs to be registered with gob.
func init() {
gob.Register(CustomImpl(0))
}
var (
invalidFunc = Func("invalid", func() {})
regFuncRuns = 0
regFuncMsg = ""
regFunc = Func("reg", func(c context.Context, arg string) {
regFuncRuns++
regFuncMsg = arg
})
custFuncTally = 0
custFunc = Func("cust", func(c context.Context, ct *CustomType, ci CustomInterface) {
a, b := 2, 3
if ct != nil {
a = ct.N
}
if ci != nil {
b = ci.N()
}
custFuncTally += a + b
})
anotherCustFunc = Func("cust2", func(c context.Context, n int, ct *CustomType, ci CustomInterface) {
})
varFuncMsg = ""
varFunc = Func("variadic", func(c context.Context, format string, args ...int) {
// convert []int to []interface{} for fmt.Sprintf.
as := make([]interface{}, len(args))
for i, a := range args {
as[i] = a
}
varFuncMsg = fmt.Sprintf(format, as...)
})
errFuncRuns = 0
errFuncErr = errors.New("error!")
errFunc = Func("err", func(c context.Context) error {
errFuncRuns++
if errFuncRuns == 1 {
return nil
}
return errFuncErr
})
dupeWhich = 0
dupe1Func = Func("dupe", func(c context.Context) {
if dupeWhich == 0 {
dupeWhich = 1
}
})
dupe2Func = Func("dupe", func(c context.Context) {
if dupeWhich == 0 {
dupeWhich = 2
}
})
reqFuncRuns = 0
reqFuncHeaders *taskqueue.RequestHeaders
reqFuncErr error
reqFunc = Func("req", func(c context.Context) {
reqFuncRuns++
reqFuncHeaders, reqFuncErr = RequestHeaders(c)
})
stdCtxRuns = 0
stdCtxFunc = Func("stdctx", func(c stdctx.Context) {
stdCtxRuns++
})
)
type fakeContext struct {
ctx context.Context
logging [][]interface{}
}
func newFakeContext() *fakeContext {
f := new(fakeContext)
f.ctx = internal.WithCallOverride(context.Background(), f.call)
f.ctx = internal.WithLogOverride(f.ctx, f.logf)
return f
}
func (f *fakeContext) call(ctx context.Context, service, method string, in, out proto.Message) error {
panic("should never be called")
}
var logLevels = map[int64]string{1: "INFO", 3: "ERROR"}
func (f *fakeContext) logf(level int64, format string, args ...interface{}) {
f.logging = append(f.logging, append([]interface{}{logLevels[level], format}, args...))
}
func TestInvalidFunction(t *testing.T) {
c := newFakeContext()
if got, want := invalidFunc.Call(c.ctx), fmt.Errorf("delay: func is invalid: %s", errFirstArg); got.Error() != want.Error() {
t.Errorf("Incorrect error: got %q, want %q", got, want)
}
}
func TestVariadicFunctionArguments(t *testing.T) {
// Check the argument type validation for variadic functions.
c := newFakeContext()
calls := 0
taskqueueAdder = func(c context.Context, t *taskqueue.Task, _ string) (*taskqueue.Task, error) {
calls++
return t, nil
}
varFunc.Call(c.ctx, "hi")
varFunc.Call(c.ctx, "%d", 12)
varFunc.Call(c.ctx, "%d %d %d", 3, 1, 4)
if calls != 3 {
t.Errorf("Got %d calls to taskqueueAdder, want 3", calls)
}
if got, want := varFunc.Call(c.ctx, "%d %s", 12, "a string is bad"), errors.New("delay: argument 3 has wrong type: string is not assignable to int"); got.Error() != want.Error() {
t.Errorf("Incorrect error: got %q, want %q", got, want)
}
}
func TestBadArguments(t *testing.T) {
// Try running regFunc with different sets of inappropriate arguments.
c := newFakeContext()
tests := []struct {
args []interface{} // all except context
wantErr string
}{
{
args: nil,
wantErr: "delay: too few arguments to func: 1 < 2",
},
{
args: []interface{}{"lala", 53},
wantErr: "delay: too many arguments to func: 3 > 2",
},
{
args: []interface{}{53},
wantErr: "delay: argument 1 has wrong type: int is not assignable to string",
},
}
for i, tc := range tests {
got := regFunc.Call(c.ctx, tc.args...)
if got.Error() != tc.wantErr {
t.Errorf("Call %v: got %q, want %q", i, got, tc.wantErr)
}
}
}
func TestRunningFunction(t *testing.T) {
c := newFakeContext()
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
regFuncRuns, regFuncMsg = 0, "" // reset state
const msg = "Why, hello!"
regFunc.Call(c.ctx, msg)
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if regFuncRuns != 1 {
t.Errorf("regFuncRuns: got %d, want 1", regFuncRuns)
}
if regFuncMsg != msg {
t.Errorf("regFuncMsg: got %q, want %q", regFuncMsg, msg)
}
}
func TestCustomType(t *testing.T) {
c := newFakeContext()
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
custFuncTally = 0 // reset state
custFunc.Call(c.ctx, &CustomType{N: 11}, CustomImpl(13))
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if custFuncTally != 24 {
t.Errorf("custFuncTally = %d, want 24", custFuncTally)
}
// Try the same, but with nil values; one is a nil pointer (and thus a non-nil interface value),
// and the other is a nil interface value.
custFuncTally = 0 // reset state
custFunc.Call(c.ctx, (*CustomType)(nil), nil)
// Simulate the Task Queue service.
req, err = http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw = httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if custFuncTally != 5 {
t.Errorf("custFuncTally = %d, want 5", custFuncTally)
}
}
func TestRunningVariadic(t *testing.T) {
c := newFakeContext()
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
varFuncMsg = "" // reset state
varFunc.Call(c.ctx, "Amiga %d has %d KB RAM", 500, 512)
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
const expected = "Amiga 500 has 512 KB RAM"
if varFuncMsg != expected {
t.Errorf("varFuncMsg = %q, want %q", varFuncMsg, expected)
}
}
func TestErrorFunction(t *testing.T) {
c := newFakeContext()
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
errFunc.Call(c.ctx)
// Simulate the Task Queue service.
// The first call should succeed; the second call should fail.
{
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
}
{
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if rw.Code != http.StatusInternalServerError {
t.Errorf("Got status code %d, want %d", rw.Code, http.StatusInternalServerError)
}
wantLogging := [][]interface{}{
{"ERROR", "delay: func failed (will retry): %v", errFuncErr},
}
if !reflect.DeepEqual(c.logging, wantLogging) {
t.Errorf("Incorrect logging: got %+v, want %+v", c.logging, wantLogging)
}
}
}
func TestDuplicateFunction(t *testing.T) {
c := newFakeContext()
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
if err := dupe1Func.Call(c.ctx); err == nil {
t.Error("dupe1Func.Call did not return error")
}
if task != nil {
t.Error("dupe1Func.Call posted a task")
}
if err := dupe2Func.Call(c.ctx); err != nil {
t.Errorf("dupe2Func.Call error: %v", err)
}
if task == nil {
t.Fatalf("dupe2Func.Call did not post a task")
}
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if dupeWhich == 1 {
t.Error("dupe2Func.Call used old registered function")
} else if dupeWhich != 2 {
t.Errorf("dupeWhich = %d; want 2", dupeWhich)
}
}
func TestGetRequestHeadersFromContext(t *testing.T) {
c := newFakeContext()
// Outside a delay.Func should return an error.
headers, err := RequestHeaders(c.ctx)
if headers != nil {
t.Errorf("RequestHeaders outside Func, got %v, want nil", headers)
}
if err != errOutsideDelayFunc {
t.Errorf("RequestHeaders outside Func err, got %v, want %v", err, errOutsideDelayFunc)
}
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
reqFunc.Call(c.ctx)
reqFuncRuns, reqFuncHeaders = 0, nil // reset state
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
req.Header.Set("x-appengine-taskname", "foobar")
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if reqFuncRuns != 1 {
t.Errorf("reqFuncRuns: got %d, want 1", reqFuncRuns)
}
if reqFuncHeaders.TaskName != "foobar" {
t.Errorf("reqFuncHeaders.TaskName: got %v, want 'foobar'", reqFuncHeaders.TaskName)
}
if reqFuncErr != nil {
t.Errorf("reqFuncErr: got %v, want nil", reqFuncErr)
}
}
func TestStandardContext(t *testing.T) {
// Fake out the adding of a task.
var task *taskqueue.Task
taskqueueAdder = func(_ context.Context, tk *taskqueue.Task, queue string) (*taskqueue.Task, error) {
if queue != "" {
t.Errorf(`Got queue %q, expected ""`, queue)
}
task = tk
return tk, nil
}
c := newFakeContext()
stdCtxRuns = 0 // reset state
if err := stdCtxFunc.Call(c.ctx); err != nil {
t.Fatal("Function.Call:", err)
}
// Simulate the Task Queue service.
req, err := http.NewRequest("POST", path, bytes.NewBuffer(task.Payload))
if err != nil {
t.Fatalf("Failed making http.Request: %v", err)
}
rw := httptest.NewRecorder()
runFunc(c.ctx, rw, req)
if stdCtxRuns != 1 {
t.Errorf("stdCtxRuns: got %d, want 1", stdCtxRuns)
}
}
func TestFileKey(t *testing.T) {
const firstGenTest = 0
tests := []struct {
mainPath string
file string
want string
}{
// first-gen
{
"",
filepath.FromSlash("srv/foo.go"),
filepath.FromSlash("srv/foo.go"),
},
// gopath
{
filepath.FromSlash("/tmp/staging1234/srv/"),
filepath.FromSlash("/tmp/staging1234/srv/foo.go"),
"foo.go",
},
{
filepath.FromSlash("/tmp/staging1234/srv/_gopath/src/example.com/foo"),
filepath.FromSlash("/tmp/staging1234/srv/_gopath/src/example.com/foo/foo.go"),
"foo.go",
},
{
filepath.FromSlash("/tmp/staging2234/srv/_gopath/src/example.com/foo"),
filepath.FromSlash("/tmp/staging2234/srv/_gopath/src/example.com/foo/bar/bar.go"),
filepath.FromSlash("example.com/foo/bar/bar.go"),
},
{
filepath.FromSlash("/tmp/staging3234/srv/_gopath/src/example.com/foo"),
filepath.FromSlash("/tmp/staging3234/srv/_gopath/src/example.com/bar/main.go"),
filepath.FromSlash("example.com/bar/main.go"),
},
{
filepath.FromSlash("/tmp/staging3234/srv/gopath/src/example.com/foo"),
filepath.FromSlash("/tmp/staging3234/srv/gopath/src/example.com/bar/main.go"),
filepath.FromSlash("example.com/bar/main.go"),
},
{
filepath.FromSlash(""),
filepath.FromSlash("/tmp/staging3234/srv/gopath/src/example.com/bar/main.go"),
filepath.FromSlash("example.com/bar/main.go"),
},
// go mod, same package
{
filepath.FromSlash("/tmp/staging3234/srv"),
filepath.FromSlash("/tmp/staging3234/srv/main.go"),
"main.go",
},
{
filepath.FromSlash("/tmp/staging3234/srv"),
filepath.FromSlash("/tmp/staging3234/srv/bar/main.go"),
filepath.FromSlash("bar/main.go"),
},
{
filepath.FromSlash("/tmp/staging3234/srv/cmd"),
filepath.FromSlash("/tmp/staging3234/srv/cmd/main.go"),
"main.go",
},
{
filepath.FromSlash("/tmp/staging3234/srv/cmd"),
filepath.FromSlash("/tmp/staging3234/srv/bar/main.go"),
filepath.FromSlash("bar/main.go"),
},
{
filepath.FromSlash(""),
filepath.FromSlash("/tmp/staging3234/srv/bar/main.go"),
filepath.FromSlash("bar/main.go"),
},
// go mod, other package
{
filepath.FromSlash("/tmp/staging3234/srv"),
filepath.FromSlash("/go/pkg/mod/github.com/foo/bar@v0.0.0-20181026220418-f595d03440dc/baz.go"),
filepath.FromSlash("github.com/foo/bar/baz.go"),
},
}
for i, tc := range tests {
if i > firstGenTest {
os.Setenv("GAE_ENV", "standard")
}
internal.MainPath = tc.mainPath
got, err := fileKey(tc.file)
if err != nil {
t.Errorf("Unexpected error, call %v, file %q: %v", i, tc.file, err)
continue
}
if got != tc.want {
t.Errorf("Call %v, file %q: got %q, want %q", i, tc.file, got, tc.want)
}
}
}
appengine-1.6.7/demos/ 0000775 0000000 0000000 00000000000 13741174344 0014612 5 ustar 00root root 0000000 0000000 appengine-1.6.7/demos/guestbook/ 0000775 0000000 0000000 00000000000 13741174344 0016614 5 ustar 00root root 0000000 0000000 appengine-1.6.7/demos/guestbook/app.yaml 0000664 0000000 0000000 00000000473 13741174344 0020264 0 ustar 00root root 0000000 0000000 # Demo application for App Engine "flexible environment".
runtime: go
vm: true
api_version: go1
handlers:
# Favicon. Without this, the browser hits this once per page view.
- url: /favicon.ico
static_files: favicon.ico
upload: favicon.ico
# Main app. All the real work is here.
- url: /.*
script: _go_app
appengine-1.6.7/demos/guestbook/favicon.ico 0000664 0000000 0000000 00000002176 13741174344 0020743 0 ustar 00root root 0000000 0000000 h ( l2Ƨɰĭ«¨tI k(GfI @bǿϿ̿ǽ¸õɲϐI @I ĚƛΧi(tF;ùPrlPI @n;٪pӦaŜN£w}HZѸė^t30 I kǰѺƮ{#ȯůʱ rξ˼Ç,ͧh˻˱ j0@˸;ǼͶ vѯzķb +̎4ʘKϸµȻy @јLwI @xC`hxb` ƎSϙS` y5lߐI m q: t appengine-1.6.7/demos/guestbook/guestbook.go 0000664 0000000 0000000 00000005470 13741174344 0021153 0 ustar 00root root 0000000 0000000 // Copyright 2011 Google Inc. All rights reserved.
// Use of this source code is governed by the Apache 2.0
// license that can be found in the LICENSE file.
// This example only works on App Engine "flexible environment".
// +build !appengine
package main
import (
"html/template"
"net/http"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/log"
"google.golang.org/appengine/user"
)
var initTime time.Time
type Greeting struct {
Author string
Content string
Date time.Time
}
func main() {
http.HandleFunc("/", handleMainPage)
http.HandleFunc("/sign", handleSign)
appengine.Main()
}
// guestbookKey returns the key used for all guestbook entries.
func guestbookKey(ctx context.Context) *datastore.Key {
// The string "default_guestbook" here could be varied to have multiple guestbooks.
return datastore.NewKey(ctx, "Guestbook", "default_guestbook", 0, nil)
}
var tpl = template.Must(template.ParseGlob("templates/*.html"))
func handleMainPage(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
http.Error(w, "GET requests only", http.StatusMethodNotAllowed)
return
}
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
ctx := appengine.NewContext(r)
tic := time.Now()
q := datastore.NewQuery("Greeting").Ancestor(guestbookKey(ctx)).Order("-Date").Limit(10)
var gg []*Greeting
if _, err := q.GetAll(ctx, &gg); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
log.Errorf(ctx, "GetAll: %v", err)
return
}
log.Infof(ctx, "Datastore lookup took %s", time.Since(tic).String())
log.Infof(ctx, "Rendering %d greetings", len(gg))
var email, logout, login string
if u := user.Current(ctx); u != nil {
logout, _ = user.LogoutURL(ctx, "/")
email = u.Email
} else {
login, _ = user.LoginURL(ctx, "/")
}
data := struct {
Greetings []*Greeting
Login, Logout, Email string
}{
Greetings: gg,
Login: login,
Logout: logout,
Email: email,
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
if err := tpl.ExecuteTemplate(w, "guestbook.html", data); err != nil {
log.Errorf(ctx, "%v", err)
}
}
func handleSign(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" {
http.Error(w, "POST requests only", http.StatusMethodNotAllowed)
return
}
ctx := appengine.NewContext(r)
g := &Greeting{
Content: r.FormValue("content"),
Date: time.Now(),
}
if u := user.Current(ctx); u != nil {
g.Author = u.String()
}
key := datastore.NewIncompleteKey(ctx, "Greeting", guestbookKey(ctx))
if _, err := datastore.Put(ctx, key, g); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Redirect with 303 which causes the subsequent request to use GET.
http.Redirect(w, r, "/", http.StatusSeeOther)
}
appengine-1.6.7/demos/guestbook/index.yaml 0000664 0000000 0000000 00000000134 13741174344 0020605 0 ustar 00root root 0000000 0000000 indexes:
- kind: Greeting
ancestor: yes
properties:
- name: Date
direction: desc
appengine-1.6.7/demos/guestbook/templates/ 0000775 0000000 0000000 00000000000 13741174344 0020612 5 ustar 00root root 0000000 0000000 appengine-1.6.7/demos/guestbook/templates/guestbook.html 0000664 0000000 0000000 00000001316 13741174344 0023503 0 ustar 00root root 0000000 0000000
Guestbook Demo
{{with .Email}}You are currently logged in as {{.}}.{{end}}
{{with .Login}}Sign in{{end}}
{{with .Logout}}Sign out{{end}}
{{range .Greetings }}
{{with .Author}}{{.}}{{else}}An anonymous person{{end}}
on {{.Date.Format "3:04pm, Mon 2 Jan"}}
wrote
{{.Content}}
{{end}}
appengine-1.6.7/demos/helloworld/ 0000775 0000000 0000000 00000000000 13741174344 0016765 5 ustar 00root root 0000000 0000000 appengine-1.6.7/demos/helloworld/app.yaml 0000664 0000000 0000000 00000000224 13741174344 0020427 0 ustar 00root root 0000000 0000000 runtime: go
api_version: go1
vm: true
handlers:
- url: /favicon.ico
static_files: favicon.ico
upload: favicon.ico
- url: /.*
script: _go_app
appengine-1.6.7/demos/helloworld/favicon.ico 0000664 0000000 0000000 00000002176 13741174344 0021114 0 ustar 00root root 0000000 0000000 h ( S E M ~' Oi/Rǰ[ժ~KJs O 3 J6 N{( e q<ȵh WҟZɘQkW aɽG\Si> I kբhǓ?{W ´yǽлk
Zy / ¤~k`¢pxǪ}: K S ( IîԭsǀϴźK` }# `ȷ̺ɷo
. S QȅĨk 6 Q}Α5ɃIƳøʭp
% [ MuԝSɏ6N j +_!Sp9O X Hqޭtk N <