filter on a field for a set of values (OR between values)

[apelliciari]

Hi,

i don't know if is the right place to ask, because it's not an issue, just a question.

I'm using django-filter with Django-REST-Framework to build APIs for my app (great job by the way to both of you!)

I would like to know if there's a way to get this filter working

http://example.com/api/products?category=clothing,shoes

to get all products with category "clothing" OR category "shoes".

I've tried with

http://example.com/api/products?category=clothing&category=shoes

but as expected it does an AND between the filters and retrieve only the shoes products (i suppose because it's the last filter).

[hanifvirani]

I have the exact same question.

[Flowerowl]

I have the exact same question.too

[guialante]

I have the exact question too

[Flowerowl]

i have an idea: overwrite MultipleChoiceFilter:

from django.forms.fields import MultipleChoiceField
from django_filters.filters import MultipleChoiceFilter
...
class MultipleField(MultipleChoiceField):
def valid_value(self, value):
return True

class MultipleFilter(MultipleChoiceFilter):
field_class = MultipleField

class xxxFilter(django.filters.FilterSet):
object_id = MultipleFilter(look_up='in')
class Meta(object):
model = yyy
fields = ['object_id']

[ddevlin]

Update 2015-04-17: As @TankOs pointed out below a change in django-filter required this solution to be adjusted, I'm correcting it here in case anyone uses this code without reading further. @kmwenja also posted a nice solution below that includes sanitisation and type coercion (this solution assumes you're filtering on string fields).

@apelliciari, I have the same use case as you (DRF, need comma separated OR filter), this is my solution:

from django_filters import Filter
from django_filters.fields import Lookup

from .models import Product

class ListFilter(Filter):
    def filter(self, qs, value):
        value_list = value.split(u',')
        return super(ListFilter, self).filter(qs, Lookup(value_list, 'in'))

class ProductFilterSet(django_filters.FilterSet):
    category = ListFilter(name='categories__slug')

    class Meta:
        model = Product
        fields = ['category']

To anybody else, YMMV outside use with Django REST Framework

[xordoquy]

For the record, http://example.com/api/products?category=clothing&category=shoes does work provided you use MultipleChoiceFilter which does an OR between the two.

[jdotjdot]

The MultipleChoiceFilter hasn't even been working for me using that style.

[tiagoboldt]

+1. @ddevlin solution works perfectly, I suggest adding it to the lib.

[josh-t]

Thank you @ddevlin!

[carltongibson]

Hmmm. This is curious. I'm looking at the source code for MultipleChoiceFilter and it's definitely OR-ing — so it should work.

Anyone fancy chucking a PR together with a failing test? And if so, @ddevlin's fix in a separate commit. ;)

[carltongibson]

Can anyone paste an example of how you're setting up the filter? Basically the ?category=clothing&category=shoes with MultipleChoiceFilter should work but this issue seems too frequent (both here and on Stack Overflow) for it to be just nothing. (It may be that it's a documentation issue.)

... and retrieve only the shoes products (i suppose because it's the last filter).

This sounds like the QueryDict is having __getItem__ — or equivalent — called rather than getList somewhere. (That would explain it at any rate.) If I can see the set up then I can step through and confirm it either way.

(There's a separate issue about the ?a=1,2,3 syntax. That is on my list to come back to at a later date.)

[carltongibson]

Also — can you confirm DRF and django-filter versions?

Thanks!

[ddevlin]

Hi @carltongibson, I don't believe this is a django-filter issue.

When DRF instantiates the FilterSet, it passes request.QUERY_PARAMS as the data argument (see this line). If the url was http://example.com/api/products?category=clothing&category=shoes then the QueryDict passed to the django-filter FilterSet won't contain the first query parameter as the second one will have overwritten it (in this example, if I put a breakpoint in the FilterSet __init__ the data argument is django.http.request.QueryDict({u'category': u'shoes'})).

DRF's request.QUERY_PARAMS is just an alias for .GET in the Django request that DRF's request is wrapping (see here) so this deduplication of query parameters happens to the request before DRF is involved. I think this makes the ?category=clothing&category=shoes style impossible to support.

[carltongibson]

But it should work like this, no...?

In [1]: from django.http.request import QueryDict

In [2]: qd = QueryDict('category=clothing&category=shoes')
---------------------------------------------------------------------------
ImproperlyConfigured                      Traceback (most recent call last)

[Hmmm. Accursed DJANGO_SETTINGS_MODULE Lets skip that shall we...]

In [3]: qd = QueryDict(u'category=clothing&category=shoes', encoding='utf8')

In [4]: qd
Out[4]: <QueryDict: {u'category': [u'clothing', u'shoes']}>

In [5]: qd["category"]
Out[5]: u'shoes'

In [6]: qd.get("category")
Out[6]: u'shoes'

In [7]: qd.getlist("category")
Out[7]: [u'clothing', u'shoes']

(This is more or less the example from the docs)

So assuming getlist is called in all the right places, it should work.

(Thanks for coming back!)

[ddevlin]

I spoke too soon, I wasn't aware of request.GET.getlist('var') 😄

[carltongibson]

if I put a breakpoint in the FilterSet __init__ the data argument is django.http.request.QueryDict({u'category': u'shoes'})

Is that an actual, "I put a breakpoint there, and that was the value"? — If so that's wrong. We should be getting the equivalent to my:

 Out[4]: <QueryDict: {u'category': [u'clothing', u'shoes']}>

[ddevlin]

Actual, but it appears to be a side effect of how werkzeug's debugger represents dicts, when I use IPython I get the same <QueryDict: {u'category': [u'clothing', u'shoes']}> as you do.

[BrickXu]

Specify widget=SelectMultiple works well for my project.

[TankOs]

Just wanted to let you guys know that a change in django-filter requires an adjusted @ddevlin solution. ;-)

from django_filters import Filter
from django_filters.fields import Lookup

class ListFilter( Filter ):
  def filter( self, qs, value ):
    return super( ListFilter, self ).filter( qs, Lookup( value.split( u"," ), "in") )

[tuky]

i think, most people cannot use MultipleChoiceFilter directly, because they want to OR-filter on fields that are not a MultipleChoiceField, e.g. id: http://example.com/api/products?id=2&id=3. Right now, this can only be done with the solution in #137 (comment)

[kmwenja]

Does @ddevlin solution work for any type of filter (ie Number, Char)?

[carltongibson]

You might have to adjust types, but it should do (more or less)

[kmwenja]

@carltongibson so does that mean inheriting from different filter types or actually typing the split values?

[carltongibson]

Yeah, you'll have to implement the filter yourself (but it's only small). split will give you a list of strings, if you're filtering on, say, an IntegerField you'll need to covert the value.

[kmwenja]

@carltongibson you were right, here's my implementation:

class ListFilter(CharFilter):

    def sanitize(self, value_list):
        """
        remove empty items in case of ?number=1,,2
        """
        return [v for v in value_list if v != u'']

    def customize(self, value):
        return value

    def filter(self, qs, value):
        multiple_vals = value.split(u",")
        multiple_vals = self.sanitize(multiple_vals)
        multiple_vals = map(self.customize, multiple_vals)
        actual_filter = django_filters.fields.Lookup(multiple_vals, 'in')
        return super(ListFilter, self).filter(qs, actual_filter)

So for filtering integers:

class ListIntegerFilter(ListFilter):

    def customize(self, value):
        return int(value)

[zoidyzoidzoid]

(There's a separate issue about the ?a=1,2,3 syntax. That is on my list to come back to at a later date.)

Sorry to post this on this issue, but I really can't find the issue. @carltongibson . Is it closed?

[carltongibson]

Hey @zoidbergwill — No problem — and "Not exactly".

The are two options if you want to support the 1,2,3 syntax — a custom filter(as per the examples in this thread) or (I'm coming round to thinking) a custom widget that implements value_from_datadict to return the list you're after.

(For this latter suggestion see my similar comment on #187)

If there's anything specific I'm happy for an issue on the 1,2,3 syntax to be opened — but I'm inclined to see it as a documentation issue — I'm very keen to encourage users to implement the project specific filters and widgets they need.

(I really must sit down and give the docs the love they need.)

[zoidyzoidzoid]

I ended up using a custom simple CommaSeparatedValueFilter to get the behaviour I was looking for.

It's not a difficult filter to write, but for some reason I expected it to work out the box. I guess it is more of a documentation issue.

While reading these docs and seeing how lists worked in the queries when using the FilterSet class directly in the tests, I thought it'd work, until I looked through the code properly.

[tiagoboldt]

this is a recurring question and a requirement for many. I'm all in favor of having an implementation available in the framework.

[carltongibson]

@tiagoboldt Are you talking about the Or filtering or the 1,2,3 syntax?

The Or filtering works — this ticket is only still open because I haven't had the time to sit down and go over every corner of it to make sure there's nothing lurking.

As for supporting 1,2,3 query param syntax, I'll grant it needs documenting but a custom widget here is a simple thing.

I'm all in favor of having an implementation available in the framework.

PRs welcome.

[tiagoboldt]

@carltongibson 1,2,3 syntax. I've been using django-rest for some time and every project the need for such custom filter appears.

[visigoth]

i just ran across this issue in a project i'm working on. i used some of the above code to come up with a similar solution that works across ManyToMany relationships. that is, i have a Foo model with bars=model.ManyToManyField(Bar). when i tried the code from @ddevlin and @kmwenja , i ran into the fact that Relationship fields do not support Lookup. however, you can use Q expressions to achieve the desired filter expression. Here's my replacement code:

class ListFilter(django_filters.Filter):
  def __init__(self, filter_value=lambda x: x, **kwargs):
    super(ListFilter, self).__init__(**kwargs)
    self.filter_value_fn = filter_value

  def sanitize(self, value_list):
    return [v for v in value_list if v != u'']

  def filter(self, qs, value):
    values = value.split(u",")
    values = self.sanitize(values)
    values = map(self.filter_value_fn, values)
    f = Q()
    for v in values:
      kwargs = {self.name: v}
      f = f|Q(**kwargs)
    return qs.filter(f)

In the FilterSet, i created a ListFilter with the relationship syntax, e.g. entries = ListFilter(name='entries__id', filter_value=lambda x: int(x)). I'm not too familiar with the innards of django_filters, but i oddly did not have to add it to the fields list in order to have my filter run.

[carltongibson]

I'm going to close this as it's not a bug as it stands. We'll see if #259 makes sense for adding the 1,2,3 list syntax support as a custom widget but it should be easy enough to add in user land code if you need it in the meantime.

[andybak]

I'm a little confused as to why this is closed. Maybe it's a documentation issue but MultipleChoiceFilter doesn't seem to behave as expected as Tuky is states at #137 (comment)

I spent a good hour or so on this before I found this issue thread so at the very least the docs seem a nice big warning :-)

[carltongibson]

What's the bug? Where's the failing test case demonstrating it? No one has been able to pin down the supposed issue.

[rohitkeshav]

I am sorry for asking, but what is that works now? for filtering things such as ?a=1,2,3

[tiagoboldt]

From my understanding, it's:
http://URL/something/?field=value1,value2,value3
The expected query in django would be:
something.objects.filter(Q(field=value1) | Q(field=value2) | Q(field=value3))

I totally agree with the need for a native filter as such, as it has been a need I've had in every project I've used django-rest.

[carltongibson]

Yep. ?f=1&f=2&f=3 has always worked — it just requires the right widget. The new CSV stuff adds support for ?f=1,2,3 as a single value.

Check the docs, have a play. Any issues open a new ticket.