Skip to content

Commit

Permalink
cmd: improve --match/--include/--exclude patterns parsing, improve re…
Browse files Browse the repository at this point in the history 
…adme
  • Loading branch information
@tdewolff
tdewolff committed Oct 6, 2023
1 parent 38f0e88 commit 76935f3
Show file tree
Hide file tree
Showing 4 changed files with 128 additions and 89 deletions.
114 changes: 65 additions & 49 deletions cmd/minify/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,59 +77,70 @@ which will output
```

## Usage
Usage: minify [options] [input]

Usage: minify [options] inputs...

Options:
-a, --all Minify all files, including hidden files and files in hidden directories
-b, --bundle Bundle files by concatenation into a single file
--cpuprofile string Export CPU profile
--css-precision int Number of significant digits to preserve in numbers, 0 is all
--exclude string Filename exclusion pattern, excludes files from being processed
-h, --help Show usage
--html-keep-comments Preserve all comments
--html-keep-conditional-comments Preserve all IE conditional comments
--html-keep-default-attrvals Preserve default attribute values
--html-keep-document-tags Preserve html, head and body tags
--html-keep-end-tags Preserve all end tags
--html-keep-quotes Preserve quotes around attribute values
--html-keep-whitespace Preserve whitespace characters but still collapse multiple into one
--include string Filename inclusion pattern, includes files previously excluded
--js-keep-var-names Preserve original variable names
--js-precision int Number of significant digits to preserve in numbers, 0 is all
--js-version int ECMAScript version to toggle supported optimizations (e.g. 2019, 2020), by default 0 is the latest version
--json-keep-numbers Preserve original numbers instead of minifying them
--json-precision int Number of significant digits to preserve in numbers, 0 is all
-l, --list List all accepted filetypes
--match string Filename matching pattern, only matching files are processed
--memprofile string Export memory profile
--mime string Mimetype (eg. text/css), optional for input filenames, has precedence over --type
-o, --output string Output file or directory (must have trailing slash), leave blank to use stdout
-p, --preserve strings[=mode,ownership,timestamps] Preserve options (mode, ownership, timestamps, links, all)
-q, --quiet Quiet mode to suppress all output
-r, --recursive Recursively minify directories
--svg-keep-comments Preserve all comments
--svg-precision int Number of significant digits to preserve in numbers, 0 is all
-s, --sync Copy all files to destination directory and minify when filetype matches
--type string Filetype (eg. css), optional for input filenames
--url string URL of file to enable URL minification
-v, --verbose count Verbose mode, set twice for more verbosity
--version Version
-w, --watch Watch files and minify upon changes
--xml-keep-whitespace Preserve whitespace characters but still collapse multiple into one

Input:
Files or directories, leave blank to use stdin. Specify --mime or --type to use stdin and stdout.

-a, --all Minify all files, including hidden files and files in hidden
directories
-b, --bundle Bundle files by concatenation into a single file
--css-precision int Number of significant digits to preserve in numbers, 0 is all
--exclude []string Path exclusion pattern, excludes paths from being processed
--ext map[string]string
Filename extension mapping to filetype (eg. css or text/css)
-h, --help Help
--html-keep-comments Preserve all comments
--html-keep-conditional-comments
Preserve all IE conditional comments
--html-keep-default-attrvals
Preserve default attribute values
--html-keep-document-tags
Preserve html, head and body tags
--html-keep-end-tags Preserve all end tags
--html-keep-quotes Preserve quotes around attribute values
--html-keep-whitespace Preserve whitespace characters but still collapse multiple into one
--include []string Path inclusion pattern, includes paths previously excluded
--js-keep-var-names Preserve original variable names
--js-precision int Number of significant digits to preserve in numbers, 0 is all
--js-version int ECMAScript version to toggle supported optimizations (e.g. 2019,
2020), by default 0 is the latest version
--json-keep-numbers Preserve original numbers instead of minifying them
--json-precision int Number of significant digits to preserve in numbers, 0 is all
-l, --list List all accepted filetypes
--match []string Filename matching pattern, only matching filenames are processed
--mime string Mimetype (eg. text/css), optional for input filenames (DEPRECATED, use --type)
-o, --output string Output file or directory, leave blank to use stdout
-p, --preserve []string Preserve options (mode, ownership, timestamps, links, all)
-q, --quiet Quiet mode to suppress all output
-r, --recursive Recursively minify directories
-s, --sync Copy all files to destination directory and minify when filetype
matches
--svg-keep-comments Preserve all comments
--svg-precision int Number of significant digits to preserve in numbers, 0 is all
--type string Filetype (eg. css or text/css), optional when specifying inputs
--url string URL of file to enable URL minification
-v, --verbose Verbose mode, set twice for more verbosity
--version Version
-w, --watch Watch files and minify upon changes
--xml-keep-whitespace Preserve whitespace characters but still collapse multiple into one

Arguments:
inputs Input files or directories, leave blank to use stdin

### Types

css text/css
htm text/html
html text/html
js application/javascript
json application/json
svg image/svg+xml
xml text/xml
Default extension mapping to mimetype (and thus minifier). Use `--ext` to add more mappings, see below for an example.

css text/css
htm text/html
html text/html
js application/javascript
json application/json
mjs application/javascript
rss application/rss+xml
svg image/svg+xml
webmanifest application/manifest+json
xhtml application/xhtml-xml
xml text/xml

## Examples
Minify **index.html** to **index-min.html**:
Expand Down Expand Up @@ -192,6 +203,11 @@ or
$ minify -r -o out/ --ext {scss:text/css xjs:js} src/
```

#### Matching and include/exclude patterns
The patterns for `--match`, `--include`, and `--exclude` can be either a glob or a regular expression. To use the latter, prefix the pattern with `~` (if you want to use a glob starting with `~`, escape the tilde `\~...`). Match only matches the base filename, while include/exclude match the full path. Be aware of bash expansion of glob patterns, which requires you to quote the pattern or escape asterisks.

Match will filters all files by the given pattern, eg. `--match '*.css'` will only minify CSS files. The `--include` and `--exclude` options allow to add or remove certain files or directories and is interpreted in the order given. For example, `minify -rvo out/ --exclude 'src/*/**' --include 'src/foo/**' src/` will minify the directory `src/`, except for `src/*/...` where `*` is not `foo`.

### Concatenate
When multiple inputs are given and the output is either standard output or a single file, it will concatenate the files together if you use the bundle option.

Expand Down
92 changes: 60 additions & 32 deletions cmd/minify/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ import (
"os"
"os/signal"
"path/filepath"
"reflect"
"regexp"
"runtime"
"sort"
Expand Down Expand Up @@ -74,15 +73,40 @@ var (
oldmimetype string
)

type Matches struct {
matches *[]string
}

func (scanner Matches) Scan(s []string) (int, error) {
n := 0
for _, item := range s {
if strings.HasPrefix(item, "-") {
break
}
*scanner.matches = append(*scanner.matches, item)
n++
}
return n, nil
}

func (typenamer Matches) TypeName() string {
return "[]string"
}

type Includes struct {
filters *[]string
}

func (scanner Includes) Scan(s []string) (int, error) {
v := ""
n, err := argp.ScanVar(reflect.ValueOf(&v).Elem(), s)
*scanner.filters = append(*scanner.filters, "+"+v)
return n, err
n := 0
for _, item := range s {
if strings.HasPrefix(item, "-") {
break
}
*scanner.filters = append(*scanner.filters, "+"+item)
n++
}
return n, nil
}

func (typenamer Includes) TypeName() string {
Expand All @@ -94,10 +118,15 @@ type Excludes struct {
}

func (scanner Excludes) Scan(s []string) (int, error) {
v := ""
n, err := argp.ScanVar(reflect.ValueOf(&v).Elem(), s)
*scanner.filters = append(*scanner.filters, "-"+v)
return n, err
n := 0
for _, item := range s {
if strings.HasPrefix(item, "-") {
break
}
*scanner.filters = append(*scanner.filters, "-"+item)
n++
}
return n, nil
}

func (typenamer Excludes) TypeName() string {
Expand Down Expand Up @@ -152,10 +181,10 @@ func run() int {
f.AddRest(&inputs, "inputs", "Input files or directories, leave blank to use stdin")
f.AddOpt(&output, "o", "output", nil, "Output file or directory, leave blank to use stdout")
f.AddOpt(&oldmimetype, "", "mime", nil, "Mimetype (eg. text/css), optional for input filenames (DEPRECATED, use --type)")
f.AddOpt(&mimetype, "", "type", nil, "Filetype (eg. css or text/css), optional for input filenames")
f.AddOpt(&matches, "", "match", nil, "Filename matching pattern, only matching files are processed")
f.AddOpt(Includes{&filters}, "", "include", nil, "Filename inclusion pattern, includes files previously excluded")
f.AddOpt(Excludes{&filters}, "", "exclude", nil, "Filename exclusion pattern, excludes files from being processed")
f.AddOpt(&mimetype, "", "type", nil, "Filetype (eg. css or text/css), optional when specifying inputs")
f.AddOpt(Matches{&matches}, "", "match", nil, "Filename matching pattern, only matching filenames are processed")
f.AddOpt(Includes{&filters}, "", "include", nil, "Path inclusion pattern, includes paths previously excluded")
f.AddOpt(Excludes{&filters}, "", "exclude", nil, "Path exclusion pattern, excludes paths from being processed")
f.AddOpt(&extensions, "", "ext", nil, "Filename extension mapping to filetype (eg. css or text/css)")
f.AddOpt(&recursive, "r", "recursive", false, "Recursively minify directories")
f.AddOpt(&hidden, "a", "all", false, "Minify all files, including hidden files and files in hidden directories")
Expand Down Expand Up @@ -213,7 +242,7 @@ func run() int {
}
sort.Strings(keys)
for _, k := range keys {
fmt.Println(k + strings.Repeat(" ", n-len(k)+1) + extMap[k])
fmt.Println(k + strings.Repeat(" ", n-len(k)+2) + extMap[k])
}
}
return 0
Expand Down Expand Up @@ -545,29 +574,27 @@ func minifyWorker(chanTasks <-chan Task, chanFails chan<- int) {
chanFails <- fails
}

// compilePattern returns *regexp.Regexp or glob.Glob
func compilePattern(pattern string) (*regexp.Regexp, error) {
// compile regexp or glob pattern
re, err := regexp.Compile(pattern)
if err != nil {
// glob wildcards to regexp
pattern = regexp.QuoteMeta(pattern)
pattern = strings.ReplaceAll(pattern, `\*`, `.+`)
pattern += "$"

var err2 error
if re, err2 = regexp.Compile(pattern); err2 != nil {
Error.Println(err)
return nil, err
if len(pattern) == 0 || pattern[0] != '~' {
if strings.HasPrefix(pattern, `\~`) {
pattern = pattern[1:]
}
pattern = regexp.QuoteMeta(pattern)
pattern = strings.ReplaceAll(pattern, `\*\*`, `.*`)
pattern = strings.ReplaceAll(pattern, `\*`, fmt.Sprintf(`[^%c]*`, filepath.Separator))
pattern = strings.ReplaceAll(pattern, `\?`, fmt.Sprintf(`[^%c]?`, filepath.Separator))
pattern = "^" + pattern + "$"
}
return re, nil
return regexp.Compile(pattern)
}

func fileFilter(filename string) bool {
if 0 < len(matches) {
match := false
base := filepath.Base(filename)
for _, re := range matchesRegexp {
if re.MatchString(filename) {
if re.MatchString(base) {
match = true
break
}
Expand All @@ -576,12 +603,13 @@ func fileFilter(filename string) bool {
return false
}
}
match := true
for i, re := range filtersRegexp {
if re.MatchString(filename) {
return filters[i][0] == '+'
match = filters[i][0] == '+'
}
}
return true
return match
}

func fileMatches(filename string) bool {
Expand Down Expand Up @@ -632,7 +660,7 @@ func createTasks(fsys fs.FS, inputs []string, output string) ([]Task, []string,
}
tasks = append(tasks, task)
} else if info.Mode().IsRegular() {
valid := fileFilter(info.Name()) // don't filter mimetype
valid := fileFilter(input) // don't filter mimetype
if valid || sync {
task, err := NewTask(root, input, output, !valid)
if err != nil {
Expand Down Expand Up @@ -683,7 +711,7 @@ func createTasks(fsys fs.FS, inputs []string, output string) ([]Task, []string,
}
tasks = append(tasks, task)
} else if d.Type().IsRegular() {
valid := fileMatches(d.Name())
valid := fileMatches(input)
if valid || sync {
task, err := NewTask(root, input, output, !valid)
if err != nil {
Expand Down
3 changes: 1 addition & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ require (
github.com/dustin/go-humanize v1.0.1
github.com/fsnotify/fsnotify v1.6.0
github.com/matryer/try v0.0.0-20161228173917-9ac251b645a2
github.com/spf13/pflag v1.0.5
github.com/tdewolff/argp v0.0.0-20231006010547-9c5fed0deeda
github.com/tdewolff/argp v0.0.0-20231006153134-9e8ae9690730
github.com/tdewolff/parse/v2 v2.6.8
github.com/tdewolff/test v1.0.9
)
Expand Down
8 changes: 2 additions & 6 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,8 @@ github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/matryer/try v0.0.0-20161228173917-9ac251b645a2 h1:JAEbJn3j/FrhdWA9jW8B5ajsLIjeuEHLi8xE4fk997o=
github.com/matryer/try v0.0.0-20161228173917-9ac251b645a2/go.mod h1:0KeJpeMD6o+O4hW7qJOT7vyQPKrWmj26uf5wMc/IiIs=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/tdewolff/argp v0.0.0-20230622205231-8a4234db046f h1:l5DhJGDkk4qI60fmBURYxC9cr7hgEtrrc4P1WJKp38E=
github.com/tdewolff/argp v0.0.0-20230622205231-8a4234db046f/go.mod h1:fF+gnKbmf3iMG+ErLiF+orMU/InyZIEnKVVigUjfriw=
github.com/tdewolff/argp v0.0.0-20231006010547-9c5fed0deeda h1:6CfJwZHxYOIpEYRiqU7z34yWc99balM7xxSFYz/ke8U=
github.com/tdewolff/argp v0.0.0-20231006010547-9c5fed0deeda/go.mod h1:fF+gnKbmf3iMG+ErLiF+orMU/InyZIEnKVVigUjfriw=
github.com/tdewolff/argp v0.0.0-20231006153134-9e8ae9690730 h1:+ovScTG2is8AhPgzJCs9g+jo2R1l5Mq6o6np9FSkCfQ=
github.com/tdewolff/argp v0.0.0-20231006153134-9e8ae9690730/go.mod h1:fF+gnKbmf3iMG+ErLiF+orMU/InyZIEnKVVigUjfriw=
github.com/tdewolff/parse/v2 v2.6.8 h1:mhNZXYCx//xG7Yq2e/kVLNZw4YfYmeHbhx+Zc0OvFMA=
github.com/tdewolff/parse/v2 v2.6.8/go.mod h1:XHDhaU6IBgsryfdnpzUXBlT6leW/l25yrFBTEb4eIyM=
github.com/tdewolff/test v1.0.6/go.mod h1:6DAvZliBAAnD7rhVgwaM7DE5/d9NMOAJ09SqYqeK4QE=
Expand Down

0 comments on commit 76935f3

Please sign in to comment.