Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge Development #79

Merged
merged 32 commits into from
Jan 15, 2018
Merged

Merge Development #79

merged 32 commits into from
Jan 15, 2018

Conversation

domodwyer
Copy link

@domodwyer domodwyer commented Jan 15, 2018

Includes:

Thanks to:


Throughput overview

Select throughput has increased by ~600 requests/second with slightly increased variance:

x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68815499fc7f8fbb5931d643b8fd7804a80-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

Note: latencies are approximations calculated from grouped data

domodwyer and others added 30 commits August 8, 2017 11:00
* master:
  Add contribution guidelines.
  Add more changes to README.
* master:
  Run integration tests against 3.2.16 (#24)
Create a new method to drop all the indexes of a collection
in a single call
fix [#484](https://github.com/go-mgo/mgo/issues/484)

Annotate connections with metadata provided by the
connecting client.

informations send:

{
 "aplication": {         // optional
   "name": "myAppName"
 }
 "driver": {
    "name": "mgo",
    "version": "v2"
  },
  "os": {
    "type": runtime.GOOS,
    "architecture": runtime.GOARCH
  }
}

to set "application.name", add `appname` param in options
of string connection URI,
for example : "mongodb://localhost:27017?appname=myAppName"
* docs: elaborate on what appName does

* readme: add appName to changes
Fix #30.

Thanks to @feliixx for the time and effort.
* Stop all db instances after tests (go-mgo#462)

If all tests pass, the builds for mongo earlier than 2.6 are still failing.
Running a clean up fixes the issue.

* fixing int64 type failing when getting indexes and trying to type them

* requested changes relating to case statement and panic

* Update README.md to credit @mapete94.

* tests: ensure indexed int64 fields do not cause a panic in Indexes()

See:
* #23
* https://github.com/go-mgo/mgo/issues/475
* go-mgo#476
- Allow specifying the default collation for the collection when creating it.
- Add some documentation to query.Collation() method.

fix #29
* test against MongoDB 3.4.x

* tests: use listIndexes to assert index state for 3.4+

* make test pass against v3.4.x

  - skip `TestViewWithCollation` because of SERVER-31049,
    cf: https://jira.mongodb.org/browse/SERVER-31049

  - add versionAtLeast() method in init.js script to better
    detect server version

fixes #31
* Adds missing collation feature description (by @feliixx).
* Adds missing 3.4 tests description (by @feliixx).
* Adds BSON constants description (by @bozaro).
* Adds UTC time.Time unmarshalling (by @gazoon).
readme: add missing feature descriptions / credit
* Fix GetBSON() method usage

Original issue
---

You can't use type with custom GetBSON() method mixed with structure field type and structure field reference type.

For example, you can't create custom GetBSON() for Bar type:

```
struct Foo {
	a  Bar
	b *Bar
}
```

Type implementation (`func (t Bar) GetBSON()` ) would crash on `Foo.b = nil` value encoding.

Reference implementation (`func (t *Bar) GetBSON()` ) would not call on `Foo.a` value encoding.

After this change
---

For type implementation  `func (t Bar) GetBSON()` would not call on `Foo.b = nil` value encoding.
In this case `nil` value would be seariazied as `nil` BSON value.

For reference implementation `func (t *Bar) GetBSON()` would call even on `Foo.a` value encoding.

* Minor refactoring
This change remove full BSON decoding on:

 - parsing to `bson.Raw` and `bson.DocElem` fields;
 - skipping unused BSON fields.
* readme: credit @bozaro and @idy

* readme: add @idy to contributor list
* Add proper DN construction

* Added openssl check to test

* Addressed code review comments

* Changes to RDNformatting, and test

* type/value fields to tests

* Changes to RDN formatting

* Corrected comment in getRFC2253NameString

* Corrected comment in getRFC2253NameString

* Changes to login and rdn formatting

* Changed escaping of #
use memory pooling to reuse bulkActions and
  avoid some allocations
* test against 3.6

* update go and mongodb version

  - use last minor version for each major serie of
    mongodb
  - use travis 'go' param instead of eval "$(gimme $GO)"
    as it fails to install correctly last minor version
    ( 1.8.x notation for example)

* test fixes on 3.2.17

also re-enable TestFindIterSnapshot as it was fixed
a long time ago

* fix X509 test

fix TestAuthX509CredRDNConstruction test: need to
create an user with {"username": subject} before
trying to login

* Fix auth test on 3.6-rc3

Make sure that "rs3/127.0.0.1/40031" is elected
at primary.
Create user before running 'addShard' command as
it requires root access
Also add a retry mechanism to make sure that shard
are correctly added
cf https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/

* implement OP_MSG wire protocole

require maxWireVersion >= 6 on server side, and
`experimental=opmsg` in the connection string

  - get `MaxMessageSizeBytes` and `MaxWriteBatchSize`
    from handshake
  - allow unacknowledged writes with `moreToCome` flag
  - split bulk operations in batch of `maxWriteBatchSize`
  - add 'experimental' param in URL. To enable an experimental
    feature, add `experimental=featureName` in the connection
    URL

flush logout directly

Previously, `flushLogout()` was called at the beginning
of each Query to the database.
To avoid these useless calls, we flush logout directly
when `Logout()` or `LogoutAll()` is called

* re-enable TestViewWithCollation

SERVER-31049 is fixed in 3.4.10, so re-enable
it

* refactor memory pooling

use the same pool for send and received
messages.
Slices are returned to the pool without being
resized.

Default allocation size might need to be updated
(currently 256, no benchmarks available yet)

* update to 3.6.0 stable
* master:
  Merge Development (#57)
  Revert "do not lock while writing to a socket (#52)"
  do not lock while writing to a socket (#52)
  Merge Development (#48)
  Merge development (#39)

# Conflicts:
#	.travis.yml
#	cluster_test.go
#	harness/daemons/.env
#	harness/mongojs/init.js
#	session.go
#	session_internal_test.go
#	session_test.go
* Revert "MongoDB 3.6: implement the new wire protocol (#61)"

This reverts commit 90c056c.

* test against 3.6

* update go and mongodb version

  - use last minor version for each major serie of
    mongodb
  - use travis 'go' param instead of eval "$(gimme $GO)"
    as it fails to install correctly last minor version
    ( 1.8.x notation for example)

* test fixes on 3.2.17

also re-enable TestFindIterSnapshot as it was fixed
a long time ago

* fix X509 test

fix TestAuthX509CredRDNConstruction test: need to
create an user with {"username": subject} before
trying to login

* Fix auth test on 3.6-rc3

Make sure that "rs3/127.0.0.1/40031" is elected
at primary.
Create user before running 'addShard' command as
it requires root access
Also add a retry mechanism to make sure that shard
are correctly added
cf https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/

* update to 3.6.0 stable

* tests: cherry pick missing 3.6+ support changes
@domodwyer domodwyer merged commit 896bbb8 into master Jan 15, 2018
- MONGODB=x86_64-ubuntu1404-3.0.15
- MONGODB=x86_64-ubuntu1404-3.2.17
- MONGODB=x86_64-ubuntu1404-3.4.10
- MONGODB=x86_64-ubuntu1404-3.6.0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we upgrade to 3.6.2 while we are at it ?

libi pushed a commit to libi/mgo that referenced this pull request Dec 1, 2022
Includes:

* Reduced memory in bulk operations (globalsign#56)
* Native x509 authentication (globalsign#55)
* Better connection recovery (globalsign#69)
* Example usage (globalsign#75 and globalsign#78)

Thanks to:

* @bachue
* @csucu 
* @feliixx


---
[Throughput overview](https://user-images.githubusercontent.com/9275968/34954403-3d3253dc-fa18-11e7-8eef-0f2b0f21edc3.png)

Select throughput has increased by ~600 requests/second with slightly increased variance:
```
x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

```

* [insert-latency.txt](https://github.com/globalsign/mgo/files/1632474/insert-latency.txt)
* [insert-throughput.txt](https://github.com/globalsign/mgo/files/1632475/insert-throughput.txt)
* [select-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632476/select-zipfian-latency.txt)
* [select-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632477/select-zipfian-throughput.txt)
* [update-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632478/update-zipfian-latency.txt)
* [update-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632479/update-zipfian-throughput.txt)

Note: latencies are approximations calculated from grouped data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants