Introspect redirections for HTTP TPC xfers

[PerilousApricot]

To better debug failing HTTP-TPC transfers, it would be nice to expose to users of libdavix the hostname actually responsible for the transfer after redirection, if any.

I originally thought that adding a getCurrentUri analogous to getOriginalUri to expose the protected _current variable from BackendRequest would be sufficient to get things going

davix/src/backend/BackendRequest.hpp

Line 178 in 23a591d

std::shared_ptr<Uri> getOriginalUri() const;

... but this is the wrong place to solve it. In DavixCopyInternal::copy, there is logic to work through the redirections and then find the "correct" host for one side of the transfer:

davix/src/modules/copy/copy.cpp

Lines 229 to 298 in 23a591d

	HttpRequest* request = NULL;
	do {
	nextSrc = _full_url(prevSrc, nextSrc);
	prevSrc = nextSrc;
	if (request) {
	request->discardBody(&internalError);
	if (!internalError)
	request->endRequest(&internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	}
	delete request;
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Hop: {}",
	nextSrc);
	request = context.createRequest(nextSrc, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	request->setRequestMethod("COPY");
	if(parameters->getCopyMode() == CopyMode::Push){
	request->addHeaderField("Destination", destination);
	}else if(parameters->getCopyMode() == CopyMode::Pull){
	request->addHeaderField("Source", destination);
	}
	request->addHeaderField("X-Number-Of-Streams", nstreamsStr);
	// for lcgdm-dav, ask for secure redirection in all cases for COPY
	request->addHeaderField("Secure-Redirection", "1");
	// for lcgdm-dav -> S3, add NoHead flag to suppress final head-to-close request
	if(suppressFinalHead)
	request->addHeaderField("Copy-Flags", "NoHead");
	request->setParameters(requestParams);
	request->beginRequest(&internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	// If we get a X-Delegate-To, before continuing, delegate
	if (request->getAnswerHeader("X-Delegate-To", delegationEndpoint)) {
	std::vector<std::string> delegationEndpoints = parseXDelegateTo(nextSrc, delegationEndpoint, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Got delegation endpoint: {}",
	delegationEndpoint.c_str());
	std::string dlg_id = DavixDelegation::delegate(context, delegationEndpoints,
	parameters, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Got delegation ID {}",
	dlg_id.c_str());
	dlg_id.clear();
	}
	} while (!shouldCancel() && request->getAnswerHeader("Location", nextSrc) && request->getRequestCode() >= 300 && request->getRequestCode() < 400);

There isn't the analogous logic for the 2nd side of the transfer (which is stored in the destination variable regardless of push/pull mode), the Uri is passed as-is as a Source/Destination header, then the resolution of the redirects happens on the machine we're connected to: (the top-level redirector for each site is xrootd-vanderbilt.sites.opensciencegrid.org and xrootd.cmsaf.mit.edu, note that the Source side is the redirector and not the "resolved" version)

DEBUG Davix: > COPY ///store/user/meloam/TESTMV.txt?authz=BearerXXX HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd-se2-vanderbilt.sites.opensciencegrid.org:1094
> Source: https://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt
> X-Number-Of-Streams: 0
> Secure-Redirection: 1
> TransferHeaderAuthorization: xx
> Credential: none
> Authorization: xx
> RequireChecksumVerification: false

Since libdavix doesn't resolve the COPY redirects for the destination, there's no way to report back the actual destination host (in this case). But, we do resolve the PROPFIND verb for the destination:

DEBUG Davix: > PROPFIND //store/temp/user/meloam/hosts.txt HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd.cmsaf.mit.edu:1094
> Depth: 0
> Authorization: xxx
> 

DEBUG Davix: Sending request-line and headers:
DEBUG Davix: Doing DNS lookup on xrootd.cmsaf.mit.edu...
DEBUG Davix: clicert callback 
DEBUG Davix: call client cert callback 
DEBUG Davix: Request sent; retry is 0.
DEBUG Davix: < HTTP/1.1 302 Redirect
DEBUG Davix: < Connection: Keep-Alive
DEBUG Davix: < Content-Length: 0
DEBUG Davix: < Location: https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX
DEBUG Davix: < 
DEBUG Davix: End of headers.
DEBUG Davix: Running post_headers hooks
DEBUG Davix: Add cached redirection <PROPFIND davs://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX>

Is this on purpose? If instead DavixCopyInternal::copy() resolved redirects and stashed them into Davix::DavixCopyInternal, then accessors could be added to DavixCopy for current source/dest Uris. The changes seem relatively straightforward to implement, but it is a behavioral change, so I don't want to implement if there would be no desire to add that functionality.

[mpatrascoiu]

Hello,

This is a great proposal and it targets a problem we hit very often in FTS/Gfal2 transfers.
The proposed solution needs to be adapted, as Davix cannot resolve the passive host. Instead, it will need to be told this value.

Some theory
In an HTTP-TPC, two parties are involved, named active and passive. You are correct, Davix resolves one of them before the copy operation starts, namely the active party. However, Davix has no way of resolving the passive party and in fact, it shouldn't. It's best to let the active party resolve the transfer location. Davix will simply pass the "passive party" location to the active party, which becomes responsible for doing the host resolution and then the file transfer.

Once the HTTP-TPC starts, the active party will continuously send transfer updates back to the initiating client, in this case Davix. These updates are called HTTP-TPC Performance Markers, and one such field is the RemoteConnections field (best documentation I know on this topic is here). The RemoteConnections field contains the IP address of the passive host.

Back to the problem
Knowing the actual hosts involved in the transfer is very valuable. For this reason, I've created the following issue DMC-1358. The change introduces two new getter methods in the Davix::Copy public API:

    std::string getTransferSourceHost() const;
    std::string getTransferDestinationHost() const;

After the transfer is done, they will return either the hostname or IP value, or empty, if no information is available.

The active party is resolved by Davix, just before initiating the COPY request. The passive party is extracted by Davix from the HTTP-TPC Performance Markers, via the RemoteConnections field.

Here is an example of how FTS/Gfal2 will leverage this information:

$ gfal-copy -v https://eospublic.cern.ch:443/eos/opstest/dteam/file.test https://eospps.cern.ch:443/eos/opstest/dteam/file.test

event: [1670855013815] BOTH   http_plugin	TRANSFER:EXIT	https://eospublic.cern.ch:443/eos/opstest/dteam/file.test (p05151113281066.cern.ch) => https://eospps.cern.ch:443/eos/opstest/dteam/file.test ([2001:1458:301:17::100:e])

All Davix changes have been implemented in commit 969b9a5.

Cheers,
Mihai