Add support for aggregation event hooks

[aarobc]

I'm attempting to set up an aggregation endpoint that would select the first point from every day within the location.

An example of a document:

{
    "_id": ObjectID("59ab0a1950d0b5000183f88e"),
    "accuracy": 10,
    "velocity": 0,
    "heading": 0,
    "altitude": 1372,
    "created": ISODate("2017-08-03T23:14:47.000Z"),
    "date": "8/3/2017",
    "location": [
        0,
        0
    ]
}

I set up my settings.py like so:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],


    }
}

first = {
    'datasource': {
        'aggregation': {
            'pipeline': [
               { "$match":{
                    "location": {
                        "$near": {
                            "$geometry": {
                                'type': "Point",
                                'coordinates': ["$lng", "$lat"]
                            },
                            "$maxDistance": "$radius"
                        }
                    }
                }},
                {"$group": {
                    "datee": "$date"
                }}

            ]
        }
    }
}

But testing in the browser retrieves no results. example: http://localhost:5000/first?aggregate={%22$lat%22:0,%22$lng%22:0%22$radius%22:12} in the browser gives:

<resource>
<mapped>
<_id>
<type>objectid</type>
</_id>
</mapped>
</resource>

What am I doing wrong? (Also Doesn't it seem odd you can only have one aggregation for the entire application as per the SCHEMA_ENDPOINT parameter?)

[sergekir]

Looks like #1025

I'm working on this problem

[aarobc]

Thank you for being so helpful. I've attempted to use the approach you've linked to, but I'm still having the same problem. You can see the simplified and revised approach below:

MONGO_HOST="mongo"
MONGO_PORT=27017
MONGO_DBNAME="history"
ALLOW_UNKNOWN=True
PAGINATION_LIMIT=100
PAGINATION_DEFAULT=100
SCHEMA_ENDPOINT="first"

X_DOMAINS='*'

DOMAIN = {
    'mapped': {
        'allowed_filters' : ['*'],
        'sorting': True,
        'resource_methods': ['GET', 'POST'],
    }
}

first = {
    'datasource': {
        'source': 'mapped',
        'aggregation': {
            'pipeline': [
                {"$match": "$where$"}
            ]
        }
    }
}

the endpoint I'm hitting is:
http://localhost:5000/first?aggregate={"$where":{"altitude":1372}}
url encoded, that looks like:
http://localhost:5000/first?aggregate={%22$where%22:{%22altitude%22:1372}}

At this point I'm just wanting to see that using aggregation can select anything at all from the database, but I'm having the same problem.

Is this a configuration problem? Would you be able to point out where my misunderstanding is, or direct me to more comprehensive examples than are provided in the docs?

[Amedeo91]

This is how the aggregation work:

def parse_aggregation_stage(d, key, value):
    for st_key, st_value in d.items():
        if isinstance(st_value, dict):
            parse_aggregation_stage(st_value, key, value)
        if key == st_value:
            d[st_key] = value

response = {}
documents = []
req = parse_request(resource)

req_pipeline = copy.deepcopy(pipeline)
if req.aggregation:
    try:
        query = json.loads(req.aggregation)
    except ValueError:
        abort(400, description='Aggregation query could not be parsed.')

    for key, value in query.items():
        if key[0] != '$':
            pass
        for stage in req_pipeline:
            parse_aggregation_stage(stage, key, value)

[aarobc]

@Amedeo91 That really doesn't clarify anything for me. Would you be able to give a corrected example of aggregation configured correctly using my example?

[Amedeo91]

That is the thing: I am not sure that Eve is able to parse dictionary to create an entire step of the aggregation pipeline!!

[sergekir]

The solution can be additional Event Hooks: before_aggregation and after_aggregation.

before_aggregation will access the pipeline for aggregation and can dynamically change it.

Is it relevant issue? Need some help?

[nrchandan]

Event hooks for aggregation sounds great. How can I find out if this is in the roadmap?

[nicolaiarocci]

I just merged this feature to master.

[nrchandan]

Thanks Nicola! Would you recommend using the development version in a new yet-to-be released project? I am tempted to use it just for this feature, but I might have a workaround for the stable version as well.

PS. My untested workaround is to use a pass a match criteria to the aggregate function and require the web application pass the filter criteria (user ID) as an aggregate query parameter. I can use the pre-request event hook to ensure that the user ID provided matches the user ID of the current user. A little dirty, but it should work.

[nicolaiarocci]

A lot of people are using the development branch, I'd say it is (hopefully) stable enough. Also, I plan on releasing v0.8 next month.

[nrchandan]

I switched over to the dev branch. I am able to use the new pre-aggregate event hooks. Looks great, thank you!

I was also trying to understand your commit for this change. It looks very neat. Just two lines of code to enable the aggregate hook mechanism, but good amount of tests and documentation. Very well crafted.