proposal: os/user: add iterators over users and groups

[mjonaitis1]

Currently os/user package supports users and groups lookup via name or id. Some applications might need functionality to iterate over all available users/groups in the system. On unix there is getent command which can retrieve users or groups by calling getent passwd or getent group respectively. However, this is not very convenient. A go-native functionality could remove the need of calling external commands.

Update: use slice returning function based api instead of callback based iterator api

After receiving some internal feedback, I was recommended to not use callback based iterator API. Since callback based api introduces a lot of unnecessary complexity, slice returning function API might be more reasonable.

Updated public facing API:

package user

// ListUsers returns a slice of all available user entries.
func ListUsers() ([]*User, error) {
	return allUsers()
}

// ListGroups returns a slice of all available group entries.
func ListGroups() ([]*Group, error) {
	return allGroups()
}

Example usage:

// Users
users, err := user.ListUsers()
for _, u := range users {
	fmt.Printf("Username: %s\n", u.Username)
}
if err != nil {
	fmt.Printf("error encountered while iterating users database: %v", err)
}

// Groups
groups, err := user.ListGroups()
for _, g := range groups {
	fmt.Printf("Groupname: %s\n", g.Name)
}
if err != nil {
	fmt.Printf("error encountered while iterating groups database: %v", err)
}

However, more feedback from go maintainers is needed in order to proceed.

Initial callback based proposal

Here is a proposed user facing public api in os/user:

package user

// NextUserFunc is used in users iteration process. It receives *User for each user record.
// If non-nil error is returned from NextUserFunc - iteration process is terminated.
type NextUserFunc func(*User) error

// NextGroupFunc is used in groups iteration process. It receives *Group for each group record.
// If non-nil error is returned from NextGroupFunc - iteration process is terminated.
type NextGroupFunc func(*Group) error

// IterateUsers iterates over user entries. For each retrieved *User entry provided NextUserFunc is called.
//
// On UNIX, if CGO is enabled, getpwent is used in the underlying implementation. Since getpwent is not thread-safe,
// locking is strongly advised.
func IterateUsers(n NextUserFunc) error {
	return iterateUsers(n)
}

// IterateGroups iterates over group entries. For each retrieved *Group entry provided NextGroupFunc is called.
//
// On UNIX, if CGO is enabled, getgrent is used in the underlying implementation. Since getgrent is not thread-safe,
// locking is strongly advised.
func IterateGroups(n NextGroupFunc) error {
	return iterateGroups(n)
}

Example usage:

// Iterating all users
err := user.IterateUsers(func(u *user.User) error {
	fmt.Println(u.Name)
	return nil
})
if err != nil {
	log.Fatalf("error occurred while iterating users: %v \n", err)
}

// Iterating all groups
err := user.IterateGroups(func(g *user.Group) error {
	fmt.Println(g.Name)
	return nil
})
if err != nil {
	log.Fatalf("error occurred while iterating groups: %v \n", err)
}	

[seankhliao]

cc @bradfitz @kevinburke

[gopherbot]

Change https://golang.org/cl/345592 mentions this issue: os/user: add users and groups iterators

[neild]

How does this deal with groups in NIS/AD/other non-local sources?

Looking at the CL, iteration on Unix seems to only consult /etc/passwd and /etc/group.

[mjonaitis1]

@neild when cgo is enabled iterate_cgo.go will be used for unix. This file contains logic that utilises getpwent and getgrent libc routines which deal with fetching users or groups from other than /etc/* sources. These routines can also fetch data through name service switch.

[bjorndm]

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

[motiejus]

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

getpwent/getgrent are part of POSIX.1-2001, and it is the recommended way to get users/groups out of the system. If one wants a reasonable way to retrieve system's users/groups, then they should use the system interfaces. In fact, exactly like golang stdlib does today with Lookup* calls.

The most common and desired case (in fact, the reason why we made this pull request) — the glibc's getpwent cannot be ported to pure go, since it requires dynamically loading C shared libraries; this is just how getpwent of glibc works.

[nightlyone]

There is no POSIX api for such iteration and the implementation might trigger a huge number of element lookups when applied to a corporate user/group database with 100000 and more entries.

I would prefer to call a more specialized user directory function in that case and work with libraries better suited for such lookups and filtering and did so when I needed it.

[rsc]

Why does this need to be in the standard library?

[mjonaitis1]

Why does this need to be in the standard library?

For an internal project, we needed an API to iterate over system users and groups. Go standard library package os/user already handles lookup, but not listing or iteration. While a third-party package for iteration can be done, code will need to be copied, since most of the machinery is private to os/user package. We think it may be useful to have iteration/listing to the broader community.

Furthermore, we have some plans to open source an nss library in Golang. Since our library supports iteration (i.e. backs glibc’s getgrnam(3), getgrgid(3), getpwnam(3), getpwuid(3)), it would be helpful to have an API to piggy-back to; a standard way to iterate over users and groups would allow the library to support a single api and be a drop-in extension.

[mjonaitis1]

@mjonaitis1 I think it would be better to port getpwent and getgrent from C to Go, otherwise pure-Go programs will work differently with regards to this functionality.

Even if getpwent could be ported to pure Go, it might not be a good idea. The setpwent/getpwnent/endpwent appear to have been designed without much regard to multi-threading. For example, if two threads want to iterate and not be exposed to non-deterministic gaps, they must agree to coordinate their iteration sessions through some mutex. This is cumbersome and slow. So instead of porting these functions to go, we can simply reuse them since they are posix compliant.

[rsc]

I don't think Damien's comment has really been answered.
In sites where these entries are provided via network protocol, it may not even be possible to enumerate users and groups.
It's very difficult to say much about how well any of that will work.
And then there is Windows.

I also don't think #47907 (comment) has really been answered. It seems like this can easily live outside the standard library for the few programs that need it.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[mjonaitis1]

I don't think Damien's comment has really been answered. In sites where these entries are provided via network protocol, it may not even be possible to enumerate users and groups. It's very difficult to say much about how well any of that will work. And then there is Windows.

My knowledge about nss is limited, but I assume that libc routines like getpwent take care of the external sources, so iteration of network provided entries should be possible with cgo.

I also don't think #47907 (comment) has really been answered. It seems like this can easily live outside the standard library for the few programs that need it.

The main argument why it should be in the standard library is that there is already similar functionality in os/user package.

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group