Proposal for DNS client and custom DNS queries

[MichaCo]

In System.Net.NameResolution the DNS class gives us a simple API to get the host name and addresses by using the internals of Windows or Linux.

What is missing is an interface to query actual DNS servers (custom endpoints/ports) and get more information then just the IP or local host name.

Use Case

One use case could be accessing service discovery (e.g. Consul) to ask for available services by name via DNS:
A local Consul DNS endpoint runs on port 8600 and you'd either query with qtype SRV or A.
To query for a service by name in consul, you use <servicename>.service.consul.

Example query and answers with dig asking for the consul service itself:

$ dig @127.0.0.1 -p 8600 consul.service.consul SRV

;; QUESTION SECTION:
;consul.service.consul.         IN      SRV

;; ANSWER SECTION:
consul.service.consul.  0       IN      SRV     1 1 8300 hostname.node.dc1.consul.

;; ADDITIONAL SECTION:
hostname.node.dc1.consul. 0     IN      A       127.0.0.1

The SRV answer gives me the port the service instance is running on, the additional answer gives me the IP address. The result can contain multiple instances.

Why another API?

There are a few other implementations for .NET today, but none of which are really maintained as far as I know nor do they have support for netstandard1.x.

Proposed API

Usage

Creating a lookup client with different overloads:

// create a client using the dns server configured by your network interfaces
var lookup = new LookupClient();

// create a client using a DNS server by IP on default port 53
var lookup = new LookupClient(IPAddress.Parse("127.0.0.1"));

// create a client using a DNS server on port 8600
var endpoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 8600);
var lookup = new LookupClient(endpoint);

Using the client to query for standard qtypes:

// query for google.com with qtype ANY
DnsQueryResponse result = await lookup.QueryAsync("google.com", QueryType.ANY);

// qtype A and explicitly setting the qclass of the query
DnsQueryResponse result = await lookup.QueryAsync("google.com", QueryType.A, QueryClass.IN);

// reverse query the host name of an IP Address using arpa
DnsQueryResponse result = await lookup.QueryReverseAsync(IPAddress.Parse("192.168.1.1"));

The result in all cases would contain the response header information, the list of resource records and the question.

API

LookupClient:

public class LookupClient
{
        // dis/enables caching of responses (TTL would be driven by the server's response)
	bool UseCache { get; set; }

	public LookupClient();
	public LookupClient(params IPEndPoint[] nameServers);
	public LookupClient(params IPAddress[] nameServers);
	public LookupClient(DnsMessageHandler messageHandler, ICollection<IPEndPoint> nameServers)

	public Task<DnsQueryResponse> QueryAsync(string query, QueryType queryType);
	public Task<DnsQueryResponse> QueryAsync(string query, QueryType queryType, QueryClass queryClass);
	public Task<DnsQueryResponse> QueryAsync(string query, QueryType queryType, CancellationToken cancellationToken);
	public Task<DnsQueryResponse> QueryAsync(string query, QueryType queryType, QueryClass queryClass, CancellationToken cancellationToken);
	public Task<DnsQueryResponse> QueryReverseAsync(IPAddress ipAddress);
	public Task<DnsQueryResponse> QueryReverseAsync(IPAddress ipAddress, CancellationToken cancellationToken);
}

// message handler could be implemented with UDP or TCP, or cool channels or what ever ;)
public class DnsMessageHandler 
{
       public Task<DnsResponseMessage> QueryAsync(IPEndPoint server, DnsRequestMessage request, CancellationToken cancellationToken);

       // processes the query and generates the actual data to send to the server
       public virtual byte[] GetRequestData(DnsRequestMessage request)

       // processes the raw response data
       public virtual DnsResponseMessage GetResponseMessage(byte[] responseData)
}

// the request message
public class DnsRequestMessage
{
    public DnsRequestHeader Header { get; }
    public DnsQuestion[] Questions { get; }
}

// the request header
public class DnsRequestHeader
{
	public DnsHeaderFlag HeaderFlags {get;}
	public int Id { get; }
	public DnsOpCode OpCode {get;}
	public int QuestionCount { get; }
	public bool UseRecursion {get;}
}

// Question contains the actual query, used by response and request
public class DnsQuestion
{
	public DnsName QueryName { get; }
	public QueryClass QuestionClass { get; }
	public QueryType QuestionType { get; 
}

// the response object returned by the lookup client
public class DnsQueryResponse
{
	public IReadOnlyCollection<DnsResourceRecord> Additionals { get; }
	public IReadOnlyCollection<DnsResourceRecord> AllRecords	
	public IReadOnlyCollection<DnsResourceRecord> Answers { get; }
	public IReadOnlyCollection<DnsResourceRecord> Authorities { get; }
	public string ErrorMessage { get; }
	public bool HasError { get; }
	public DnsResponseHeader Header { get; }
	public IReadOnlyCollection<DnsQuestion> Questions { get; }
}

// see https://tools.ietf.org/html/rfc1035#section-3.2.2 and 3.2.3
public enum QueryType : short
{
        A = 1,
        NS = 2,
        ...
}

// see https://tools.ietf.org/html/rfc1035#section-3.2.4
public enum QueryClass : short
{
	IN = 1,
	CS = 2,
	CH = 3,
	HS = 4
}

There are some more types like return codes / errors etc.

RFC References

Base RFC https://tools.ietf.org/html/rfc1035

Request and response header share the same RFC https://tools.ietf.org/html/rfc6895#section-2. But only some fields / flags are used in the request, others in the response.
The length is always 12 bytes.

                                            1  1  1  1  1  1
              0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |                      ID                       |
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |QR|   OpCode  |AA|TC|RD|RA| Z|AD|CD|   RCODE   | 
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |                QDCOUNT/ZOCOUNT                |
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |                ANCOUNT/PRCOUNT                |
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |                NSCOUNT/UPCOUNT                |
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
             |                    ARCOUNT                    |
             +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

The request or response data follows the header data...

I will not list all the details, only the in my opinion most important qtypes/RRs

A: https://tools.ietf.org/html/rfc1035#section-3.4.1

NS: https://tools.ietf.org/html/rfc1035

AAAA: https://tools.ietf.org/html/rfc3596#section-2.2

PTR (for reverse queries): https://tools.ietf.org/html/rfc1035#section-3.3.12

SRV: https://tools.ietf.org/html/rfc2782

MX: https://tools.ietf.org/html/rfc1035#section-3.3.9

TXT: https://tools.ietf.org/html/rfc1035#section-3.3.14

SOA: https://tools.ietf.org/html/rfc1035#section-3.3.13

Implementation

I couldn't help and did a reference/example implementation of the whole thing here: https://github.com/MichaCo/DnsClient.NET.

If we come up with a better API which goes into corefx I'm happy to discontinue that project. But I needed the functionality now ;)

@CIPop I'm creating this proposal as mentioned in #19351

Thanks,
M

[CIPop]

@davidsh @terrajobst @stephentoub PTAL
/cc @karelz

[CIPop]

// query for google.com with qtype ANY
var result = await lookup.QueryAsync("google.com", QueryType.ANY);

In general we don't use var when we cannot infer the type from the same line of code. It's best if you could update with the actual type returned.

[MichaCo]

@CIPop yup np, done

[CIPop]

There are some more types like return codes / errors etc.

For now, I think this is enough for an initial review but in the final spec we will require the following fully defined DnsResponseMessage, QueryType, DnsHeaderFlag, DnsOpCode, DnsName, DnsResourceRecord, DnsResponseHeader.

[CIPop]

Thanks @MichaCo! LGTM

Maybe not a good place to discuss this but: have you investigated ways to implement the features using OS-level APIs or are you proposing a managed-only (Sockets-based) DNS implementation. (If the latter, would it include DNSSec?)

[karelz]

@CIPop @davidsh please submit the API for review after it is complete from your point of view (feel free to update the original API proposal post or work with @MichaCo) -- simple LGTM agreement from each of view is sufficient.
Given that it is larger API and domain specific, please review it first yourself, before submitting it for formal API review.

cc: @Priya91 @geoffkizer @stephentoub

[MichaCo]

@CIPop I was actually looking at DnsQuery, but I had no time playing with it. I also have no idea about the Linux equivalent so that's why my implementation is managed code only.

Regarding DNSSec, the signature would be transferred as another RR with the response, which can easily be supported with the API. Question is if the API also has to validate the signature against a public key. For my use case right now, I don't need it.

Another option for security would be DNS Cookies https://tools.ietf.org/html/rfc7873 I guess.

[martincostello]

Would the Question property on the DnsRequestMessage class be better as an ICollection<DnsQuestion> or IList<DnsQuestion>? Then calling code could create an instance and then add things to the property, rather than do that separately and then create an array?

As-is

var questions = new List<DnsQuestion>();

for (int i = 0; i < 10; i++)
{
    // Do something real and interesting that conditionally creates the questions
    questions.Add(new DnsQuestion());
}

var message = new DnsRequestMessage();
message.Questions = questions.ToArray();

Proposed

var message = new DnsRequestMessage();

for (int i = 0; i < 10; i++)
{
    // Do something real and interesting that conditionally creates the questions
    message.Questions.Add(new DnsQuestion());
}

[MichaCo]

@CIPop Regarding a native implementation. I did some tests with both DnsQuery and DnsQueryEx - docs.

The implementation is 99% from pinvoke.net (not cleaned up or anything)

DnsQuery works great but uses the standard network interfaces only. There is no option to set server/port. Performance is much faster than my managed code but only if I query the same name over and over. There could still be some caching although I did set the bypass cache flag.
When running random queries against different domains then there is no performance difference at all.

The DnsQueryEx actually supports setting the IPAddress and Port, unfortunately, setting the port other than 53 always results in an error (connection was closed by the remote host). I don't know why and if there is a way to work around though ~~

Did look so promising... Let me know if someone has some hints ;)

[CIPop]

The DnsQueryEx actually supports setting the IPAddress and Port, unfortunately, setting the port other than 53 always results in an error (connection was closed by the remote host). I don't know why and if there is a way to work around though ~

@MichaCo I've talked to the DNS team owning the API and they were able to use it against a custom port. They recommend ensuring that the DNS server is working with a different client such as dig. E.g: dig -p 5353 @<dns server ip> bing.com

[MichaCo]

@CIPop thanks for the followup! I figured it was actually a bug with the port. NetworkOrder from int gave 0, had to cast to short first... my bad ~~

The performance of the native code is not any better than using UDP directly
Example

dotnet run perf -s 127.0.0.1 -p 8600 consul.service.consul -c 10 -r 500
...
;; Managed run took 495ms for 10 clients 500 queries: 10.101 queries per second.
;; DnsQueryEx run took 672ms for 10 clients 500 queries: 7.440 queries per second.

Regarding security, I don't see anything that the DnsQuery/Ex does that for you, so that part would up to use anyways right?

[CIPop]

@CIPop thanks for the followup! I figured it was actually a bug with the port

Glad you found the bug!

Regarding security, I don't see anything that the DnsQuery/Ex does that for you

I'd personally prefer to leave the system-level drivers and services to handle what they already do and implement new code only where absolutely necessary. There are several advantages such as security updates, future protocol support and reducing the code/tests that we have to maintain in two different places (.Net and the OSs).

Does Linux/iOS have a similar API that could be reused?

[CIPop]

While I'm not very familiar with the API, it looks like since Win7, the OS supports DNSSec:

https://technet.microsoft.com/en-us/library/dn593685(v=ws.11).aspx

[CIPop]

@MichaCo I will need to continue my ramp-up to the DNS RFCs and protocols before I review the API proposal. I have been switched to other high-priority items and won't be able to make progress until at least mid-January next year.
The API review shouldn't block the prototyping/implementation efforts.

[MichaCo]

@CIPop no worries, same here ;) I was looking for Linux equivalents to the windows' DnsQuery/Ex. Didn't really made much progress on that either. Have to check res_query.

[jcdickinson]

Has this stalled? I might pick this up.

This isn't very idiotmatic right now - the nomenclature and patterns deviate quite strongly from what you'd expect in .Net. Some suggestions:

public class DnsClient
{
	public bool UseCache { get; set; }

    public bool UseMulticast { get; set; }

	public DnsClient();
	public DnsClient(params IPEndPoint[] nameServers);
	public DnsClient(params IPAddress[] nameServers);

	public ValueTask<DnsResponse> QueryAsync(string query, QueryType queryType);
	public ValueTask<DnsResponse> QueryAsync(string query, QueryType queryType, QueryClass queryClass);
	public ValueTask<DnsResponse> QueryAsync(string query, QueryType queryType, CancellationToken cancellationToken);
	public ValueTask<DnsResponse> QueryAsync(string query, QueryType queryType, QueryClass queryClass, CancellationToken cancellationToken);

	public ValueTask<DnsResponse> ReverseQueryAsync(IPAddress ipAddress);
	public ValueTask<DnsResponse> ReverseQueryAsync(IPAddress ipAddress, CancellationToken cancellationToken);
}

// Removed. We don't see other classes in corelib doing this.
// public class DnsMessageHandler

// Removed. Internal implementation details.
// public class DnsRequestMessage
// public class DnsRequestHeader
// public class DnsQuestion

// the response object returned by the lookup client
// Change name to DnsResponse - DnsQueryResponse is anagolous to DnsRequestResponse, which is strange.
// Should this be changed to a struct?
public readonly struct DnsResponse
{
	public DnsResourceList Additionals { get; }
	public DnsResourceList AllRecords { get; }
	public DnsResourceList Answers { get; }
	public DnsResourceList Authorities { get; }

    // Removed. Makes no sense to echo this back to the query.
	// public IReadOnlyList<DnsQuestion> Questions { get; }

    // Removed. Inline the fields from this directly into the struct/class.
    // public DnsResponseHeader Header { get; }
    // ...

    // Removed. However much I agree that Exceptions are awful, they are 
    // idiomatic in .Net.
    // public string ErrorMessage { get; }
	// public bool HasError { get; }
}

public readonly struct DnsResourceList : IReadOnlyList<DnsResource>
{
    private readonly ReadOnlyMemory<byte> _memory;
    private readonly int[] _offsets;
}

public class DnsException : ExternalException
{
    public DnsError DnsErrorCode { get; }
}

public enum DnsError
{
    Default = 0,

    FormatError = 1,
    ServerFailure = 2,
    NameError = 3,
    // NotImplemented = 4, // NotImplementedException?
    Refused = 5
}

// see https://tools.ietf.org/html/rfc1035#section-3.2.2 and 3.2.3
public enum QueryType // : short
{
    /// <summary>The <c>A</c> record.</summary>
    Address = 1,
    /// <summary>The <c>NS</c> record.</summary>
    NameServer = 2,
    /// <summary>...</summary>
    CanonicalName = 5,
    StartOfZoneAuthority = 6,
    WellKnownService = 11,
    Pointer = 12,
    HostInformation = 13,
    MailboxInformation = 14,
    MailListInformation = 14,
    MailExchange = 15,
    Text = 16,
    AddressV6 = 28
    //...
}

// see https://tools.ietf.org/html/rfc1035#section-3.2.4
public enum QueryClass // : short
{
    /// <summary>The <c>IN</c> class.</summary>
	Internet = 1,
    /// <summary>...</summary>
	CSNet = 2,
	Chaos = 3,
	Hesiod = 4
}

[jnm2]

Looks like you mixed tabs and spaces.

[FranklinYu]

Any progress? I read through this thread and think the current blocker is

Does Linux/iOS have a similar API that could be reused?

Is that right?

Also @jcdickinson why ValueTask instead of Task?

[scalablecory]

I think we should revisit this for .NET 6.

SVCB/HTTPSSVC are useful for HttpClient -- they will significantly reduce the time it takes to establish an HTTP/3 connection, as well as supply data needed for ECH, Alt-Svc, and HSTS-like features.

getaddrinfo does not provide us with this data; we need a proper DNS API.

[geoffkizer]

I think we should revisit this too.

Beyond the SVCB issue, there's also the issue of TTLs -- we can't get these through the current API. HttpClient would very much like to know DNS TTLs so that it can discard connections appropriately when DNS changes. We've heard this ask from customers multiple times, and the best answer we have today is "use PooledConnectionLifetime" which is far from ideal.

[karelz]

Given latest increased need from higher-level libraries, this is most likely going to happen 9.0 in some form.
cc @filipnavara

[filipnavara]

FWIW our use case involves lookups of SRV, TXT and MX records. These are heavily used for various service discovery protocols (Exchange Autodiscover, Kerberos, IMAP/POP3/SMTP/CalDAV server endpoint discovery). Notably, this would also benefit Kerberos.NET which may eventually be used for client authentication scenarios on platforms that don't have native Kerberos support (Android, tvOS).

We currently depend on our own library. On Windows it uses DnsQuery API. On macOS we rely on Bonjour system APIs (DNSService* in /usr/lib/system/libsystem_dnssd.dylib). On mobile platforms we fallback to a managed implementation and get addresses of DNS servers from the system. This provides reasonable balance between respecting system overrides, caches, and portability.