-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UDFs that filter on url should also filter on src #418
Comments
@SinghGursimran want this one since we're stuck in a holding pattern on the Python side of things until I sort out the Scala UDF -> Python UDF linkage? |
@ruebot Shall I add a new function to incorporate src and dest OR accommodate this within the same function using an extra argument? |
Based on the chat @lintool and I where having in Slack this morning, it'd be amending the current functions. I think we could just do this with try cases (oh, I don't know what the proper Scala term is for it 😆 ) for |
Ok.... |
@SinghGursimran If it helps to see an actual use case/test case, this is how it popped up: https://gist.github.com/ruebot/60b5f848252284b7f380e3d5006d7135 I tried to run the |
We are currently unable to run a number of DataFrame filters on
.imageLinks()
andwebgraph()
because they havesrc
and/ordest
columns instead ofurl
. The DataFrame filters should be able to filter on those columns as well.keepUrlsDF
keepDomainsDF
discardUrlsDF
discardDomainsDF
discardUrlPatternsDF
keepUrlPatternsDF
The text was updated successfully, but these errors were encountered: