Welcome to How do I Go.
You can find this book on-line at: TODO
It is licensed under the Creative Commons Attribution-Non Commercial-Share Alike 3.0 license.
This is a FAQ which deals specifically with questions of the form "How do I…?" or "Why does…?"
The inspiration for this documentation is perlfaq. While many sections of perlfaq are answered adequately in other places such as Effective Go and the Go FAQ, the example code in sections 4, 5 and 6 of perlfaq are not as well represented, or at least not in a central place. This document is an attempt to rectify that underrepresentation.
Wherever possible the Go Programming Language Specification is referenced. The acronym GPLS is used to refer to it.
First, if you have never read anything about how floating point numbers are represented in a machine, stop now, go to a search engine and search for "what every programmer should know about floating point". For bonus fun reading, read about an Excel 2007 bug: http://www.joelonsoftware.com/items/2007/09/26b.html or Eric Lippert’s explaination which is a great read: http://blogs.msdn.com/b/ericlippert/archive/2005/01/20/fun-with-floating-point-arithmetic-part-five.aspx
When doing any math, always be aware of what types are being used.
Integer math results in integers. e.g.
fmt.Println(3/9) 0
Constants in Go may be typed or untyped. Untyped constants are converted to appropriate type when needed. GPLS Constants
When printing floating point numbers, always be aware of what precision is used.
If you are using 32bit floats, you will hit rounding errors much sooner than 64bit. Literals default to 64bit. (This is not really true. The spec says floating point constants are represented with a mantissa of at least 256 bits. At runtime values are converted and 64bit math operations are used. This is also not really true. The floating point units on many CPUs use 80bit IEEE operations.)
e.g.
fmt.Printf("%.20f\n",float32(3)/9)
0.33333334326744079590
fmt.Printf("%.20f\n",float64(3)/9)
0.33333333333333331483
fmt.Printf("%.20f\n",3.0/9)
0.33333333333333331483
It does work. The numbers may not be what you think. See the previous question.
1.0 minus nine tenths minus one tenth is represented as a number very close to zero, but not zero.
e.g.
fmt.Println(int(1.0-0.9-0.1))
constant -8.45304e-132 truncated to integer
A floating point constant which is zero is convertible.
e.g.
fmt.Println(int(5.0-1.0-4.0)) 0
strconv.Atoi doesn’t know about literal language prefixes.
e.g.
mine := int64(0600)
claim, _ := strconv.Atoi("0600")
fmt.Println(mine == claim)
false
strconv.Atoi is a convenience method for ParseInt(s, 10, 0). Where 10 and 0 are base and bit size. Use base 8 for converting octal.
e.g.
mine := int64(0600)
claim, _ := strconv.ParseInt("0600", 8, 0)
fmt.Println(mine == claim)
true
A side note: The bit size parameter does not change the return type of ParseInt (read the spec and you’ll know that it wouldn’t). It causes ParseInt to return a value out of range error.
Look at the math package.
e.g.
import "math" ... fmt.Println(math.Ciel(1.6)) fmt.Println(math.Floor(1.6)) fmt.Println(math.Round(1.6))
See also, the above question about int().
Combining these two questions we have some interesting results:
e.g.
fmt.Println(int(math.Floor(1.0-0.9-0.1))) -1
In the question about int() we said that int() cannot truncate and that the floating representation of 1.0-0.9-0.1 is a tiny negative number. Calling Floor on that tiny negative number floors it to -1.
Calling Ciel on that tiny negative number ciellings it to 0.
fmt.Println(int(math.Ciel(1.0-0.9-0.1))) 0
Trig functions exist and suffer from the same limitations of floating point numbers as everything else. (Why isn’t math.Sin(math.Pi) == 0? Why isn’t math.Cos(math.Pi/2) == 0?) See the above question about int().
fmt.Println(math.Cos(math.Pi/2)) 6.123233995736757e-17 fmt.Println(math.Sin(math.Pi)) 1.2246467991473515e-16 fmt.Println(math.Cos(0)) 1 fmt.Println(math.Sin(math.Pi/2)) 1
For the same reason, trig identities will not always be exact. No really, go read the "What Every Programmer Should Know About Floating-Point Arithmetic" publication right now.
a30 := math.Pi/6
if (math.Sin(a30) == math.Sqrt(3)/2) { // is equal
fmt.Println("sin(30)==√3/2")
}
i := 0.01
if (math.Tan(i) == math.Sin(i)/math.Cos(i)) { // is not equal for all values i
fmt.Println("Identity not held.")
}
You may be tempted to compare using some epsilon value and this works for some cases, but really, go read WEPSKAFPA.
You can use literals with the 0 prefix for octal, the 0x prefix for hexadecimal. http://golang.org/ref/spec#Integer_literals
You can use strconv.ParseInt to parse hexadecimal and octal. https://www.golang.org/pkg/strconv/#ParseInt
deadbeef := "deadbeef" fmt.Println(strconv.ParseUint(deadbeef, 16, 32)) 3735928559 <nil>
hex := fmt.Sprintf("%x", uint(3735928559)) // hex is "deadbeef"
deadbeef := "33653337357" fmt.Println(strconv.ParseUint(deadbeef, 8, 32)) 3735928559 <nil>
Note the use of unsigned integer parsing. If you want to treat your parsed number as signed you will need to cast it.
deadbeef := "33653337357" val,_:=strconv.ParseUint(deadbeef, 8, 32) fmt.Println(int(val)) -559038737
Go isn’t a functional programming language. There is no generic map function. Use a for loop.
results := make([]int, len(array))
for i:=0; i< len(array); i++ {
results[i] = somefunc(array[i])
}
The "math/rand" package generates pseudo-random numbers. For cryptographic work use the crypto/rand package.
You can seed the "math/rand" package by calling rand.Seed(). A common seed which will not repeat without clock manipulation is time.Now().UnixNano().
r := rand.New(rand.NewSource(time.Now().UnixNano())) fmt.Println(r.Int())
Note that the playground at http://play.golang.org may not have the time that you think it is. time.Now() returns 2009-11-10 23:00:00 +0000 UTC as I write this. It is conceivable that this return value may change in the future.
Time.YearDay() returns the day of the year for a time.
e.g.
fmt.Println(time.Now()) fmt.Println(time.Now().YearDay()) 2009-11-10 23:00:00 +0000 UTC 314
The second return value of ISOWeek() is the week of the year.
e.g.
fmt.Println(time.Now()) fmt.Println(time.Now().ISOWeek()) 2009-11-10 23:00:00 +0000 UTC 2009 46
Do some math on the result of Year().
e.g.
fmt.Println("This is century", time.Now().Year() / 100 + 1)
fmt.Println("This is millenium", time.Now().Year() / 1000 + 1)
The time package supports math functions on times and returns durations. There are functions such as Before, After, and Equal to test which time instances are before, after, or equal to the compared time.
e.g.
var now = time.Now()
var date1, _ = time.Parse(someParsableDateString)
if date1.Before(now) {
fmt.Println("The date was in the past")
}
fmt.Println(date1.Sub(now))
The date was in the past
62425h0m0s
The time package has a Parse function which returns a new time value. That time value has a Unix function.
e.g.
var date1, _ = time.Parse(someParsableDateString) var seconds = date1.Unix()
A duckduckgo search yields this playground post: http://play.golang.org/p/ocYFWY7kpo
Use a calendar library if you want to catch all the edge cases. Dates and times are always more complex than you may think. I’m not aware of anything like Noda Time for golang, but if you really want to get it right, then you will need to write or use something like it.
Or…
Use time.AddDate(0,0,-1)
Before considering some of these items, it will be useful to understand what kind of strings Go uses. This blog post is a good explaination: https://blog.golang.org/strings
There is no one way. Your needs may be as simple as comparing input to a list of possible values, or as complex as matching a grammar or consulting a third party service.
It depends on what you mean by unescape. What is escaped? utf8? c-strings, go strings, other?
In Go, you use a loop.
Many other languages suggest a regular expression.
It is up to the reader to decide which is better and why.
You don’t.
That said, the %v format specifier does the job nicely in an fmt.Sprintf call.
msg := fmt.Sprintf("now its: %v", time.Now())
In Go, you use a loop and a simple state machine.
Many other languages suggest a regular expression.
In Go, you use a loop.
Many other languages have a generic reverse function.
In Go, you use a loop.
Many other languages suggest a regular expression.
Create a buffered string output and count width of characters yourself.
Strings are slices of runes. Runes are utf8 characters. You can slice a string.
s := "Just another gopher" j := s[0:4] g := s[13:19] fmt.Println(g, "in the hole!")
In go, strings are immutable. This is very familiar if you come from python, C#, or Java, but may be unfamiliar if you come from C or ruby.
s := "Just another gopher" s[0:3] = "yes" // cannot assign to s[0:3]
To change characters, create a new string.
s := "Just another gopher" s2 := "yet" + s[4:] fmt.Println(s2)
Find the Nth yourself.
package main
import (
"bytes"
"fmt"
"strings"
)
func main() {
const src = `Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec a diam lectus. Sed sit amet ipsum mauris. Maecenas congue ligula ac quam viverra nec consectetur ante hendrerit. Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ut gravida lorem. Ut turpis felis, pulvinar a semper sed, adipiscing id dolor. Pellentesque auctor nisi id magna consequat sagittis. Curabitur dapibus enim sit amet elit pharetra tincidunt feugiat nisl imperdiet. Ut convallis libero in urna ultrices accumsan. Donec sed odio eros. Donec viverra mi quis quam pulvinar at malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In rutrum accumsan ultricies. Mauris vitae nisi at sem facilisis semper ac in est.`
count_sit := 0
count_Donec := 0
nth_sit := 3
nth_Donec := 2
var b bytes.Buffer
for _,word:=range strings.Split(src, " ") {
wrote:=false
if word=="sit" {
count_sit++
if count_sit == nth_sit {
b.WriteString("SIT!")
wrote=true
}
}
if word == "Donec" {
count_Donec++
if count_Donec == nth_Donec { b.WriteString("ZOMGDONEC"); wrote = true }
}
if !wrote {
b.WriteString(word)
}
b.WriteString(" ")
}
fmt.Println(b.String())
}
Optimizing the garbage created by caling strings.Split is an exercise left to the reader.
Note that other languages use a regular expression to do this operation. Go can too.
nth := 3
count := 0
src := "One fish two fish red fish blue fish"
r := regexp.MustCompile(`(\w+)\s+fish\b`)
for _, m := range r.FindAllStringSubmatch(src, -1) {
count++
if (count == nth) {
fmt.Printf("The third fish is a %s one.\n", m[0])
}
}
If you want to count the number of occurences of a single character in a string, you can simply loop and keep a counter:
s := "ThisXlineXhasXsomeXx'sXinXit"
count := 0
for _, e := range s {
if e == 'X' {
count++
}
}
fmt.Println(count, "Xs")
Or you could use strings.Count
s := "ThisXlineXhasXsomeXx'sXinXit"
fmt.Println(strings.Count(s, "X"), "Xs")
Note that strings.Count is the preferred method and it works with more than a single character.
Call strings.Title.
s := "gone with the wind's director"
fmt.Println(strings.Title(s))
Use the encoding/csv package. It is not just for comma separated files and it handles commas inside of quoted fields correctly.
in := `first_name,last_name,username
"Rob","Pike",rob
Ken,Thompson,ken
"Robert","Griesemer","gri"
"jay,r",wren,"fuk,ka"
`
r := csv.NewReader(strings.NewReader(in))
for {
record, err := r.Read()
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
fmt.Println(record)
}
Use strings.TrimSpace.
s := " some user may have entered this "
fmt.Println(strings.TrimSpace(s))
See also, all of the Trim* functions in the strings package.
The fmt package is short for format.
fmt.Printf("%-10s%-10s%10s %06.2f %06d\n","jay","raymond","wren", 20.0/3, 123)
fmt.Printf("%-[5]*[1]s%-[5]*[2]s%[5]*[3]s %0[6]*.[7]*[4]f %0[6]*[8]d\n","jay","raymond","wren", 20.0/3, 10, 6, 2, 123)
If you know the columns which contain the data, then you can slice the string.
// column := line[startcolumn:end]
line := "root 28550 2 0 2015 ? 00:00:04 [btrfs-worker]"
fmt.Println(line[9:14],"#")
You can use strings.Split if the columns are separated by a delimiter as long as the delimiter is not part of the data.
line := strings.Split("root 28550 2 0 2015 ? 00:00:04 [btrfs-worker]", "\t")
fmt.Printf("user: %s\tpid:%s\n",line[0],line[1])
If you want to work with comma separated values, then use encoding/csv and See above.
Write soundex yourself, or use a library. A quick searches shows this implementation.
Use text/template and wrap the variables in the templating annotations.
If you can not use a template language you can use strings.Replacer
replacer := strings.NewReplacer("foo", "Fred", "bar", "Barney")
fmt.Println(replacer.Replace("Say Hello to foo and bar"))
Go doesn’t have HERE documents, but it does have multiline strings by using backticks.
s := `I do not need to heredoc when I can backtick`
Rob Pike has an excellent blog post oh The Go Blog titled "Arrays, slices (and strings): The mechanics of 'append'. That should be background reading for any serious go developer. I’ll briefly answer the question here.
Slices wrap arrays underneath and expose all sorts of nice things on top of them. You almost always want to use a slice. If you are unsure, then you almost certainly want to use a slice. An array is a contiguous piece of memory. The length of an array is part of its type. In Go, [16]byte and [24]byte are two different types. A slice is a view into an array.
That is the incomplete answer. For the complete answer read the specification and the above blog post.
I apologize for this answer being too verbatim like perlfaq. Perlfaq is just too well written to not copy it.
Use a map. When you think the words "unique" or "duplicated" think "map keys".
If you don’t care about order of the elements, you could just create the map then extract the keys. It’s not important how you create that has.
a := []int{4, 1, 2, 3, 3, 4, 4, 5 }
m := make(map[int]bool)
for _, i := range a {
m[i] = true
}
newa := make([]int, 0, len(m))
for k, _ := range m {
newa := append(newa, k)
}
a = newa
Hearing the word "in" is an /in/dictation that you probably should have used a map, not a slice or array, to store your data. Maps are designed to answer this question quickly and efficiently. Arrays and slices aren’t.
If you want to know an element is in a slice or array, iterate the array and look for yourself.
a := []int{4, 1, 2, 3, 3, 4, 4, 5 }
var has_six, has_five bool
for _, i := range a {
if i == 6 {
has_six = true
}
if i == 5 {
has_five = true
}
}
fmt.Println("has six:", has_six)
fmt.Println("has five:", has_five)
Like this: (assuming that each element is unique in a given array)
a := []int{1, 2, 3, 4, 5 }
b := []int{4, 5, 6, 7, 2 }
union := make([]int, 0, len(a)+len(b))
intersection:= make([]int, 0, len(a))
difference := make([]int, 0, len(a))
count := make(map[int]int, len(a))
for _, i := range a {
count[i]++
}
for _, i := range b {
count[i]++
}
for k, _ := range count {
union = append(union, k)
if count[k] > 1 {
intersection = append(intersection, k)
} else {
difference = append(difference, k)
}
}
fmt.Println("union:", union)
fmt.Println("intersection:", intersection)
fmt.Println("difference:", difference)
What does equal mean? (sorry) Same elements? Same elements and same order? Same keys? Same keys and same values?
reflect has a DeepEquals function.
a := []int{1, 2, 3, 4, 5 }
b := []int{4, 5, 6, 7, 2 }
c := []int{1, 2, 3, 4, 5 }
fmt.Println("a", a)
fmt.Println("b", b)
fmt.Println("c", c)
fmt.Println("a equals b?", reflect.DeepEqual(a, b))
fmt.Println("b equals c?", reflect.DeepEqual(c, b))
fmt.Println("a equals c?", reflect.DeepEqual(c, a))
Works for maps too. For a different definition of equals, write the comparison yourself.
Iterate the slice yourself.
b := []int{5, 9, 17, 2 }
var found, position int
for i, e := range b {
if e > 5 {
found = e
position = i
break
}
}
fmt.Println("found", found, "at position", position)
Go slices do not have a fixed size. You do not need linked lists if you want to add or remove items. Do your own homework.
If you want to cycle through an slice endlessly, you can increment the index modulo the number of elements in the slice.
b := []int{5, 9, 17, 2 }
var i int
for {
fmt.Println(b[i % 4])
if i >= 20 {
break
}
i++
}
Choose a shuffling algorithm, for example Fisher-Yates:
func fisher_yates_shuffle(deck []int) {
r := rand.New(rand.NewSource(time.Now().UnixNano()))
i := len(deck)
for (i--) {
j := r.Intn(i+1)
deck[i], deck[j] = deck[j], deck[i]
}
}
...
deck := make([]int, 52)
for i in range(deck) {
deck[i] = i
}
fisher_yates_shuffle(deck)
fmt.Println(deck)
Use a for loop and range.
for i := range lines { lines[i] = strings.Replace(lines[i], "coffee", "decaf", -1) }
There is no map function. Go’s lack of generics make writing one require using reflection. Idiomatic Go uses for loops. If you would like to see a generic implementation, Rob Pike wrote one to see how hard it would be. It is available at https://github.com/robpike/filter
If you want to do the same thing to modify the values of a map, range returns a key and value on each iteration.
for k, v := range people_ages { people_ages[k] = people_ages[k] + 1 // Another year older. }
Use the rand.Intn function.
b := []int{5, 9, 17, 2 }
item := b[rand.Intn(len(b))]
The sort package https://golang.org/pkg/sort/ defines an interface named Interface which must be implemented to use sort.Sort. It also includes some common implementations of that interface for []int, []float64, and []string.
func main() {
a := []int{64, 29, 32, 69, 42, 115, 40, 41, 11}
//sort.Ints is shorthand for sort.Sort(sort.IntSlice(a))
sort.Ints(a)
fmt.Println(a)
}
// [11 29 32 40 41 42 64 69 115]
The Reverse function for an interface effectively inverses the response of Less.
func main() {
a := []int{64, 29, 32, 69, 42, 115, 40, 41, 11}
sort.Sort(sort.Reverse(sort.IntSlice(a)))
fmt.Println(a)
}
Implementing the three method interface is easy.
func main() {
a := []string{"my", "mommy", "told", "me", "~not", "!to", "{swear}"}
sort.Sort(TildeFirst(a))
fmt.Println(a)
}
type TildeFirst []string
func(t TildeFirst) Len() int {
return len(t)
}
func (t TildeFirst) Less(i, j int) bool {
if strings.HasPrefix(t[i],"~") || strings.HasPrefix(t[i],"{") {
return true
}
if strings.HasPrefix(t[j],"~") || strings.HasPrefix(t[j],"{") {
return false
}
return t[i] < t[j]
}
func (t TildeFirst) Swap(i, j int) {
t[i], t[j] = t[j], t[i]
}
Use a default int type such at byte, int or int64 and bit mask operations or a third party package such as "github.com/golang-collections/go-datastructures/bitarray"
The Go Blog has a great post Go Slices: usage and internals which goes into depth on arrays and slices.
A Tour of Go has a page on Nil slices.
A nil slice acts like an empty slice so you can take its length, append to it, and things work. Note that this is not the case for a map.
The range clause of a for statement returns each key and value. Use a for loop to process an entire map.
for k, v := range mymap {
...
}
Filtering is done the same way, using an if statement inside of the for loop.
For example, to only do something with values when keys start with the string "now:"
for k, v := range some_string_keyed_map {
if strings.HasPrefix(s, "now:") {
doSomething(v)
}
}
Use a for loop. Also, know what you want by "merge". Do you want to merge map a into map b, merge map b into a, or merge a and b into a new map c?
Also, what do you do when a key exists in each map? does one have preference over another? One does not simply merge two maps. More specificity is required.
// Merging a into b, overwriting b's keys if a has matching keys.
for k, v := range a {
b[k] = v
}
A new map can be created with with the make() built-in and we can specify its capacity to prevent any extra allocations. Although if a and b have colliding keys, there will be overallocation by that number of keys.
c := make(map[string]string, len(a)+len(b))
for k, v := range a {
c[k] = v
}
for k, v := range b {
c[k] = v
}
The Go Blog has a great post Go Maps in Action with things every Go programmer should know about maps.
Mostly what you would expect. Maps iteration order with range is not defined and cannot be relied upon (it is in fact random). You can delete items and if they have not been iterated yet, they will not be. You can add items and they will either be iterated or not.
See the language specification section on For statements with range clause
You could iterate the map looking for the value you want but if you are going to do it more than once you should…
Create a reverse or inverted map.
byval := make(map[string]string, len(my_map))
for k, v := range my_map {
byval[v] = k
}
If you know you have multiple keys with the same value you could invert into a map of slice.
byval := make(map[string][]string, len(my_map))
for k, v := range my_map {
byval[v] = append(byval[v], k)
}
Built a slice of the keys and sort that slice.
keys := make([]string, 0, len(my_map))
for k, _ := range my_map {
keys = append(keys, k)
}
sort.Strings(keys)
Now iterate that sorted keys slice instead of the map when using range.
To sort by value, invert the map (see previous question) and then do the same thing, filling a slice of values (now keys) and sorting it.
You may also want to use a slice of key value pairs an implement the sort.Interface. There are solutions to this same question on stackoverflow. https://stackoverflow.com/questions/18695346/how-to-sort-a-mapstringint-by-its-values
Go does not have a built in sorted map and does not have the ability to create new generic data structures outside of the built-ins. This is not the Go way, yet?
That said, if you feel that you must do this, there are third party collection libraries, none of them are widely used as gophers tend to prefer to stay in the gopher tunnel of using built in slices and map. You may also consider writing your own with the types you need.
You cannot reset the for-range, but you can restart the entire iteration.
The simplest way to probably to use goto, even though it is considered harmful.
m := map[string]int{"a": 1, "b": 2, "reset":-1, "c": 3, "d": 4}
reset := false
reset:
for k, v := range m {
if k == "a" {
m["e"] = 5
}
if k == "b" {
delete(m, "b")
}
if k == "reset" && !reset {
reset = true
fmt.Println("resetting")
goto reset
}
fmt.Println(v)
}
You can combine the keys into a single slice and then use the above answer for removing duplicates, but that answer says to use a map, so you may as well combine the two maps into a third map.
c := make(map[Whatever]bool)
for k := range a {
c[k] = true
}
for k := range b {
c[k] = true
}
Keep a corresponding slice of elements.
See the above question about keeping a sorted map.
Compose struct instances, maps and slices, or use literals.
data := struct {
Name string
Description string
tags []string
topicWeights map[string]int
}{
"Alice's manifesto",
"A statement for an awesome life",
[]string{"life", "awesome", "manifesto"},
map[string]int{"life": 10, "awesome": 10, "lame": 0},
}
for example you have a
type User struct { Name string json:"name" }
and I want to make it into
type User struct { Name string json:"name" yaml:"bar" }
In the same way that a []int can use conversion to become a IntSlice because
type IntSlice []int
If your struct types are identical, other than the tags, then they can use conversion to become another type.
rephrase: Let’s say I have two struct types that are the same except they have different JSON tags (used by different APIs for example). If I have a slice of one type and I want to convert it to a slice of the other type without doing a linear amount of copying, is that a thing that is possible / advisable with Go?
type type1 struct {
Foo string `json:"foo1"`
Bar int `json:"bar1"`
}
type type2 struct {
Foo string `json:"foo2"`
Bar int `json:"bar2"`
}
func main() {
t1 := type1{Foo: "abc", Bar: 123}
t2 := type2(t1)
json.NewEncoder(os.Stdout).Encode(t1)
json.NewEncoder(os.Stdout).Encode(t2)
}
This only works if they are exactly the same. Order must match.
Ignoring the tags was added in Go 1.8: golang/go#16085
Use a marshal format to a file or use a database. For many simple things the standard library gob encoder can be used to write an object graph to a file. JSON may also be a nice format if a human needs to read or write the file.
The %v and %#v fmt formatters are very useful.
Sometimes, serializing to json can be useful, even moreso with json.MarshalIndent
json.MarshalIndent(my_thing, "", " ")
The bufio.Reader type has a ReadLine() method, but don’t use it. As its documentation says, it is low level. Use ReadString('\n') instead.
Use a scanner. (from golang docs)
func main() {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
fmt.Println(scanner.Text()) // Println will add back the final '\n'
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
}
The basic operation of doing this is to read or seek to the location of the file to be updated and then writing what is to be inserted or changed and then writing the remainder of the file.
See above for reading a file line at a time and use a counter.
The easy concept is to count the number of lines in a file (see above) and then print that number minus N to a new file.
The real question may be how to do it on a large file without doing a lot of copying. One trick is to read the file backwards. Unforntunately, there is no Scanner which can operate on a Reader in reverse. One would need to read some chunk size at a time in reverse, counting the lines numbers and then when the point T is found, use (*File).Truncate(T).
Open the source file for reading, open the destination file for writing. Use io.Copy(dst, src) to copy the bytes.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
io.Copy(dst, src)
It depends… If the file is text, use Scanner to find the point and write only what you want written.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
scanner := bufio.NewScanner(src)
for scanner.Scan() {
line := scanner.Text()
if line == `line I'm looking for` {
break
}
io.WriteString(scanner.Text())
io.WriteString("\n")
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
If the file is binary, use io.Copy and a LimitedReader.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
l := io.LimitedReader{R:src, N:2048}
io.Copy(dst, l)
Scanner buffers each line it reads. If you are reading a mixed format file, for example PDF which looks kind of like plain text, but can have large binary parts throughout it, then you may go beyond Scanner’s default 64kB token size.
You could set Scanner’s buffer using the Buffer(buf []byte , max int) method.
scanner := bufio.NewScanner(src)
buf := &bytes.Buffer{}
scanner.Buffer(buf, 256 * 1024) // max 256kB token size.
for scanner.Scan() {
...
Or, Read chunks. include the last few bytes of the previous buffer when searching for "it". I wrote something called UntilReader which is a bit like LimitedReader only instead of stopping when it has read N bytes, it stops when it has found matching bytes.
You can use it like this:
...
r := &UntilReader{R: src, S:[]byte("needle")}
io.Copy(dst, r)
Everything in the src Reader will be copied to the dst Writer up to and including the bytes of the string "needle".
Here is the implementation of UntilReader:
type UntilReader struct {
R io.Reader
S []byte
b []byte
}
func (u *UntilReader) Read(p []byte) (n int, err error) {
if len(u.S) == 0 {
return 0, io.EOF
}
// The underlying reader could do anything so use ReadFull to fill p.
n, err = io.ReadFull(u.R, p)
if err == io.ErrUnexpectedEOF {
// ReadFull returns ErrUnexpected, but lets make that EOF.
err = io.EOF
}
prev := append(u.b, p...)
ublen := len(u.S)
if ublen > len(prev) {
ublen = len(prev)
}
if i := bytes.Index(prev, u.S); i >= 0 {
u.S = nil
// If this is the first read and a match was found, prev isn't
// offset by ublen, yet add it back to i.
if u.b == nil {
i += ublen
}
return i, io.EOF
}
u.b = prev[len(prev)-ublen:]
return n, err
}
Similarly to the above UntilReader, wrap the reader and buffer it.
package main
import (
"bytes"
"io"
"os"
"strings"
)
func main() {
var r io.Reader
src2 := strings.NewReader("abcdefghijklmnop")
r = &ReplaceReader{R: src2, S: []byte("de"), D: []byte("Z")}
io.Copy(os.Stdout, r)
io.WriteString(os.Stdout, "\n")
}
type ReplaceReader struct {
R io.Reader
S []byte
D []byte
buf bytes.Buffer
}
func (u *ReplaceReader) Read(p []byte) (n int, err error) {
io.CopyN(&u.buf, u.R, int64(len(p)+len(u.S)))
buf := bytes.NewBuffer(bytes.Replace(u.buf.Bytes(), u.S, u.D, -1))
u.buf = *buf
return u.buf.Read(p)
}
The io/ioutil package has a
TempFile function.
Create a [N]byte array of a certain size and pass to Read() to read only the
fix record length you wish. Use the functions in the encoding/binary
package to turn the bytes into types you intend to read.
TODO
bytes.Buffer is probably what you want. It is a writer. You can use
io.WriteString and call the buffers String() method to get the built string.
strings.Reader is an io.Reader to an underlying string.
r := strings.NewReader("this is my string")
See this stackoverflow question How to fmt.Printf an interger with thousand commas
Write it yourself or use a library like message[https://godoc.org/golang.org/x/text/message] or humanize[https://github.com/dustin/go-humanize]
First you should consider if this is the right thing to do. Normally the shell is the only thing which expands ~ to the $HOME environment variable. Other programs typically do not do this. It is probably best to accept input filenames as args so that the shell has already expanded them.
If you must continue, use strings.ReplaceAll to replace "~" with os.Getenv("HOME").
For other globbing operations, use filepath.Glob[https://golang.org/pkg/path/filepath/#Glob].
Use os.Rename
Beware: not all platforms support locking.
If you choose to continue you can call syscall.Flock passing the fd returned from Fd() on an os.File
All I want to do is append a small amount of text to the end of a file. Do I still have to use locking?
It depends. The most important part is to understand the open and write semantics of the platform on which you are running. Windows and POSIX behave quite differently in this regard. Ultimately, if two processes (or threads) are going to be opening, writing, and closing a single file, it should probably be locked.
Use os.Stat or if the file is already open, call the Stat method on the *File. The ModTime[https://golang.org/pkg/os/#FileInfo] on the returning FileInfo interface is the modification time of the file.
MultiWriter returns an io.Writer which writes to all of the writers passed to it when written to. Open the files and pass them to MultiWriter.
If possible, you should avoid this. Unless you know with certainty that a file is small enough that it will not use significant amounts of memory, a buffered reader of some kind is always more efficient. See above for reading line by line.
If you must do this, then ioutil.ReadFile reads an entire file into a []byte which you can cast to a string.
Use a custom split function with Scanner and do it just like above reading by lines.
func main() {
paras:=`Hello world
this is a test.
What is a paragraph? What does it mean to be a para?
What do you think?
What can be done?`
scanner := bufio.NewScanner(bytes.NewBufferString(paras))
scanner.Split(func(data []byte, atEOF bool) (int, []byte, error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if i := bytes.Index(data, []byte("\n\n")); i >= 0 {
// We have a full newline-terminated line.
return i + 2, data[0:i], nil
}
if atEOF {
return len(data), data, nil
}
return 0, nil, nil
})
i:=1
for scanner.Scan() {
fmt.Println(i, scanner.Text()) // Println will add back the final '\n'
i++
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
}
This works, but has the drawback that non-empty lines with spaces are not detected as paragraph separators. Detecting such lines is left as an exercise to the reader.
Use a byte slice of length 1 and pass it to the Read() function.
b := make([]byte, 1) some_file.Read(b) fmt.Println(b)
This will work on the keyboard too, but not most terminal programs. You’ll
need to turn off terminal echo processing. The easiest way to do this is
probably to use os/exec to call the stty program.
b := make([]byte, 1)
exec
os.Stdin.Read(b)
fmt.Println(b)
Given Go’s goroutines, you shouldn’t need to do this. Read in a goroutine and communicate over a channel to another goroutine which does what you want when there is data available and when there is not.
Alternately, use a buffered Reader from the bufio package. The Buffered() method on bufio.Reader returns the number of bytes that can be read from the buffer.
Ooops, you just put a tab and a form feed into that string for a filename.
Escape your string.
An easy way of escaping strings like this in Go is to use backticks instead of
double quotes. C:\temp\foo works because backticks don’t expand escape
sequences.
Because there isn’t a dot extension on most files outside of Windows. Glob uses unix globbing conventions.
Know how your operating system and its filesystems work. A read-only file is not locked from deletion. In other words: a read-only file is prevented from being changed, but deleting it does not change that file. It does change the directory in which it exists. Check the permissions of the directory containing the file.
An easy way for a small enough file is to read all of the lines into a slice of strings and select a random element from the slice. See the previous question on selecting a random item from a slice.
The path/filepath package has a Walk function which accepts a WalkFunc which is called for each file or directory in the tree.
You could walk the directory and copy each item found, see the previous question about traversing a directory tree.
Alternately, there is a third party package, github.com/otiai10/copy
encoding/json allows one to easily marshal and unmarshal JSON. Use a json encoder.
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
err = json.NewEncoder(dst).Encode(something)
if err != nil {
log.Fatal(err)
}
Use a json decoder.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
err = json.NewDecoder(src).Decode(something)
if err != nil {
log.Fatal(err)
}
Use the Writer available in encoding/csv.
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
w := csv.NewWriter(dst)
// Use Write, or use WriteAll and pass a [][]string
err = w.Write([]string{"one", "two", "three"})
if err != nil {
log.Fatal(err)
}
Use the Reader available in encoding/csv.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
r := csv.NewReader(src)
// Use Read to get one record or ReadAll to get all of them as [][]string.
r1, err := r.Read()
if err != nil {
log.Fatal(err)
}
r2, err := r.Read()
if err != nil {
log.Fatal(err)
}
theRest, err := r.ReadAll()
r2, err := r.Read()
if err != nil {
log.Fatal(err)
}
Use the Encoder in encoding/xml.
dst, err := os.OpenFile(dstName, os.O_RDWR|os.O_CREATE, 0755))
if err != nil {
log.Fatal(err)
}
defer dst.Close()
enc := xml.NewEncoder(dst)
err = enc.Encode([]string{"one", "two", "three"})
if err != nil {
log.Fatal(err)
}
Use the Decoder in encoding/xml.
src, err := os.Open(srcName)
if err != nil {
log.Fatal(err)
}
defer src.Close()
dec := xml.NewDecoder(src)
err = dec.Decode(&something)
Use the Client in net/http.
resp, err := http.Get("http://example.com/")
io.Copy(os.Stdout, resp.Body)
where you’re setting a value on a field in a library, not an instance var, `http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}`
Use the Server and Muxers in net/http.
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello World, %q", html.EscapeString(r.URL.Path))
})
log.Fatal(http.ListenAndServe(":8080", nil))
None. Use net/http. Seriously. Nearly all of the 3rd party things are built from net/http and only add complexity. Find out what features you need in a 3rd party library before you use one. net/http is complete for the most common tasks.