Add subqueries, attribute modifiers & interactive mode #27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The currently solution requires us to track both the tokenizer's "previous" Token as well as each Token's preceding Token. When we reach an opening paren, we check if it's preceding token is IN, and if so, the next block is read as a Subquery. This is just a string representing another input, so it'll be tokenized, parsed, and evalulated as an individual query. If the subquery includes parens, this method of reading breaks, since we stop reading as soon as we reach a closing paren. So, we'll have to count opening parens and only exit when the count is down to 0.
Addresses FIXMEs for parsing parens in subqueries and quotes/backticks.
Subqueries were missing an identifier; resolved by adding the final "word" to the query string before breaking (in Tokenizer.readQuery).
Subquery implementation
=======================
- Evaluates subqueries as individual queries on parse step. Currently
lets you do something like:
SELECT
all
FROM
~/Desktop
WHERE
name IN (
SELECT
name
FROM
$GOPATH/src
WHERE
name = %.go) AND
size > 2mb
Some points for discussion:
- No support for SELECTing and comparing more than one attribute in a
subquery yet, I'm not really sure _how_ to implement this (or even
if it's necessary).
- Also implemented boilerplate for superquery/subquery references (as
discussed in #4#issuecomment-302104954). Currently runs **terribly**
though, not really sure if there's a way to improve performance. Also,
haven't added support for substituting directory/file names yet.
IN operator (non-subquery)
==========================
- Haven't decided if I'm going to keep this. Currently only works for `name`.
Accepts a comma-separated list of strings and checks if filename is in list.
- Example:
SELECT
all
FROM
.
WHERE
name IN [main.go, query.go]
Project layout changes
======================
- Introduce individual packages for tokenizer and parser;
- Move compare package to query.
Unit tests
==========
- Adds tests for tokenizer and compare functions.
- Some tokenizer tests are currently failing (reflect.DeepEqual is saying
identical strings don't match, might be a type issue (interface{} vs.
string)).
Bug fixes
=========
- Minor: Fix less-than-equal raw text (was >= before, doesn't really change
anything, since we're not using the raw text for anything).
- Rename Transformation -> Modifier. - Update PerformTransformations (renamed to ApplyModifiers) to run on all attributes at once. - Each modifier function receives access to the key as well as path and os.FileInfo, so we're able to run our operation based on the key instead of type (i.e. format(time, ...) vs format(size, ...) shouldn't be a problem anymore). Note that nothing here is final. I plan to merge these changes with the updated parser structure, which means we'll a large portion of this will have to be rewritten to fit the new structure.
* Adds excluder interface that can be used to determine file paths to exclude from output * Implements excluder in regexExclude to detect if a filepath should be excluded
Further simplify Exlcuder.buildRegex, make test cases more concise
* feature/subquery: Minor clean-up Use regex to detect exclusions instead of string.Contains (#21) Fix failing tests; track tokens with slice instead of linked list Add prelim. implementation of subqueries; update package structure Fix bug with missing ending identifier in parsed subqueries Replace stack implementation with Lane (oleiade/lane) Minor tokenizer refactor Add subquery tokenizer
Currently supports 'iso' and 'unix' modes.
- Add support for attribute modifiers in WHERE clause. The Condition struct holds any and all attribute modifiers used in that condition. - Offset all "transformation" work to transform package. This package is split into two parts: format and parse. Parse is use to parse input values (i.e. WHERE clause attributes) and format is used to format output values (i.e. SELECT clause attributes). - Currently support modifiers: FORMAT for time (iso/unix), size (kb/mb/gb), UPPER and LOWER (for name), and FULLPATH (also for name). - Note that attribute modifiers are not fully implemented for subqueries (both evaluated and unevaluated).
Use filepath.Clean on user given paths before processing
Now supports the following cases: 1) SELECT all FROM . WHERE UPPER(name) IN (SELECT name FROM foo) 2) SELECT all FROM . WHERE LOWER(name) IN [foo, BAR, BaZ] Decided to make the parsers for these as abstract as possible; ended up using reflect _quite_ heavily (mainly for working with "interfaced" slices/maps). From my brief testing, it's working as it should, but this is probably slightly unsafe. Also removed tokenizer.readList in favour of parsing everything related to conditions/attributes in parser.parseNextCond. Note: Still no support for queries with super/sub-relationships (even without attribute modifiers). Going to try to get this squashed next.
- Rename Condition.Comparator -> Condition.Operator. - Change type of ConditionNode.Type to *TokenType (was TokenType). - Clean directory filepaths when parsing source list instead of during traversal (exclusions are normalized as well now). - Handle aliased exclusions (throws error, since it'd make no sense to give excluded directories an alias). - Also removes a ton of dead code (notably size parsing, which was moved to the transform package).
Removes `go vet` cmd from Travis pre-script, since was getting a different result locally, will take a look into this.
Recursive modifiers (e.g. `FULLPATH(UPPER(name))`) weren't being applied since we were passing c.Value each time instead of the previously-altered value (so only the innermost modifier was having an effect).
- Invoked with -interactive. - No arrow key support yet (for moving fwd/back in query, accessing previous queries, etc). Not sure how to go about implementing this right now. - Slight change in structure: flag/input parsing moved to cmd/fsql, while top-level package (fsql) handles running the query and printing output. - Adds new package with channel-based shell prompt. Allows for quote-less AND escape-less queries. End queries with single semicolon.
Layout must be set according to 2006-01-02T15:04:05.999999-07:00, some sample queries: $ fsql SELECT all FROM . WHERE format(time, 2006-01-02) > 2017-05-29 $ fsql SELECT name, format(time, "2016-01-02 15:04:05") FROM .
Subqueries, attribute modifiers & interactive mode!
* master: Add Homebrew installation instructions Bump version Add note for selecting from dirs that begin with hypen
- Add Travis badge to README - Organize .gitignore - Bump version to 0.2.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #4, #17 (#3, #15).
This PR introduces 3 main features:
Subqueries
See Implement subqueries #4 for a detailed breakdown. No support for referencing values from superqueries yet.
$ fsql "SELECT all FROM . WHERE name IN (SELECT name FROM ../foo)"Attribute modifiers
See Attribute modifiers #17 for a detailed breakdown. Started in Allow attribute transformations in SELECT clause #23.
$ fsql "SELECT UPPER(name) FROM . WHERE name REGEXP .*\.go"$ fsql "SELECT all FROM . WHERE FORMAT(time, ISO) > 2006-01-02T15:04:05Z"Interactive mode
Invoked with the
-interactiveflag.Also includes a slight refactor to the project structure. I've included a little before/after highlighting package changes below.
Before:
Now:
Additionally adds a Travis build step with some unit tests.