Is it Byzcoin's job to solve the problem of sharing instanceIDs of deferred contracts ?

[nkcr]

The deferred contract only works if we can successfully share its instanceID to the concerned people. Right now, this process must be handled by the user itself and is hard to do programmatically. How is someone supposed to know that a deferred instance is waiting for him to add its signature on it ? Either this information is given to him via a side channel, or it can be retrieved by browsing the entire chain.

Can/Should we help on that ? On the same way there is a singleton "name" contract that provides a handy way to name instances, should we also have a sort of handy "deferredRegistry" singleton contract, responsible for storing deferred instances waiting for signatures ?

I'm opening this issue to first discuss the "should we do it ?" principle, before entering into technical details.

[ineiti]

It all depends on what you want to do with ByzCoin ;)

1. The ideal thing would be to have a clear separation (as already discussed) with the nodes only doing the consensus, and then other nodes offering full history, and possibly meeting points for deferred contracts.

2. As the development of ByzCoin does not have enough manpower to do such a separation, it's probably OK to "just add another patch". I would at least start with another service, something like ByzCoinProxy or so. Then we can dream of putting stuff like GetUpdates and others into this service, and one day have nodes without this service for the consensus, and nodes with this service as proxies.

3. Then again, that might still be overkill. So the easiest solution is of course to just add a new endpoint to the ByzCoin service. If you can do so with a minimum of protection against DOS attacks that would be nice.

TLDR: go with 2, if you think it's overkill, 3 is also OK with me.

[nkcr]

2 doesn't seem that more complicated compared to 3, so I would go for 2. If I understand well: Implement a ByzcoinProxy service that can register deferred instance IDs (with restrictions) and provide the list of deferred instance IDs (would be publicly available). This service would be called by the Byzcoin service, each time a deferred contract is spawned.

If I summary this in picture:

[ineiti]

Yes, the problem with 2) is that if we're not adding other endpoints to the service, it will be a really lonely one ;) But let's go for 2) - this gives the hope for moving some of the byzcoin-endpoints over there.

I don't remember the deferred transactions: are they stored in a block? Or only once they are fully signed?

Also, there should probably be a 4) unregister, for deferred contracts that are fully signed.

And where is 5) AddTx? The one with the additional signature? Does it go to the byzcoinproxy, or directly to byzcoin? Probably the latter, but it's still worth thinking about it.

[ineiti]

OK, that should probably be ordered differently:
4) addTx (with the additional signature)
5) unregister (if the transaction is complete)

[nkcr]

this gives the hope for moving some of the byzcoin-endpoints over there.

“We must accept finite disappointment, but never lose infinite hope.” – Martin Luther King, Jr. 😃

I don't remember the deferred transactions: are they stored in a block? Or only once they are fully signed?

A "deferred transaction" will spawn an instance of the deferred smart contract.
The deferred instance will contain the transaction to be executed once there is enough signatures on it. So yes, it is stored in a block.

Also, there should probably be a 4) unregister, for deferred contracts that are fully signed

Yes, true. Or I was also thinking about having a more complex registry that would store everything ("pending" and "executed" transactions) and let the user perform "rich" queries like "give me the instance IDs from contract X that have not been executed yet". But in any case Byzcoin must still be able to notify the Proxy once a deferred transaction has been executed.

Does it go to the byzcoinproxy, or directly to byzcoin? Probably the latter, but it's still worth thinking about it.

Yes, that goes through a normal Byzcoin transaction. It corresponds to an Invoke:AddProof instruction on the deferred instance.

So, to summary more formally:

[ineiti]

Some arrows missing to update either the instance through the proxy, or to get the instance from ByzCoin just before addProof.

BTW: I really like https://plantuml.com/ to create diagrams like this, as it allows you to add the diagram directly as a text to git, and then re-create it as needed! Is that a plantuml diagram?

[nkcr]

Some arrows missing to update either the instance through the proxy, or to get the instance from ByzCoin just before addProof.

True, the user needs the instance if it wants to sign it.

BTW: I really like https://plantuml.com/ to create diagrams like this, as it allows you to add the diagram directly as a text to git, and then re-create it as needed! Is that a plantuml diagram?

Yes, I like this way of creating diagrams. My favorite for sequence diagrams: https://sequencediagram.org/

Seems like we can agree on the system. At the end the proxy is pretty simple. Let's start.

[ineiti]

Now a day late I have another gripe: if both services are in the same node, all is OK. But what if the nodes are split?

In fact the AddInstance and RemoveInstance should be replaced by a FollowChain from the ByzcoinProxy. So byzcoin itself does not have to be aware of the proxy, it just works on its own. However, if the admin tells the Proxy to follow a chain, the Proxy will make sure to listen to every block, search for the deferred instances, and write them into his own memory.

Sorry I missed that, looks like an important part of the architecture...

[ineiti]

Regarding sequencediagram.org - I did not find (search) for a way to create the documents offline, like you can do with plantuml. Is it possible?

[nkcr]

the Proxy will make sure to listen to every block

Is there a way to be notified by Byzcoin/Skipchain when a new block is added ? Or does it work by polling periodically the chain ?

Regarding sequencediagram.org - I did not find (search) for a way to create the documents offline, like you can do with plantuml. Is it possible?

At the time I didn't find any nice tool to do that (with a quick search, I would be surprised if there isn't any), but you can still generate a graph locally with a command line: https://plantuml.com/command-line

[ineiti]

I'm using this in the omniledger frontend, and it's working very well - you have to take care about reopening the connection, though:

https://github.com/dedis/cothority/blob/master/external/js/cothority/src/byzcoin/byzcoin-rpc.ts#L661

I think the following API call is the same in go (the naming is a bit unfortunate):

https://github.com/dedis/cothority/blob/master/byzcoin/api.go#L544

For the proxy of c4dt/byzcoin I did a poll, because that was simpler (just calling updateChain ;)

[nkcr]

Update on this issue. It comes that I was re-implementing the features offered by a relational database. At the end, we would like the proxy to answer questions like "Give the me the instance IDs of the deferred instances that have not been executed". Or "Give me all the blocks that contain an instruction about smart contract X".

So, I'm currently implementing a proxy that stores the blockchain on an SQL database. The schema should be pretty straightforward:

The service would then allow one to send an SQL query to the proxy service. To answer our questions from above. ie. getting all not executed deferred instanceIDs, we can run the following query:

Select SmartContract.InstanceID from SmartContract
  WHERE SmartContract.Name == 'DeferredContract' AND
    SmartContract.InstanceID NOT IN (
      SELECT SmartContract.InstanceID from SmartContract
      INNER JOIN Instruction on SmartContract.smartcontract_id = Instruction.smartcontract_id
      INNER JOIN Transaction on Transaction.transaction_id = Instruction.transaction_id
      Where Instruction.type = 'Invoke' AND 
        Instruction.ContractID == 'DeferredContract' AND 
        Instruction.Action == 'ExecProposedTx' AND 
        Transaction.Accepted == True
  )

[ineiti]

That brings up bad memories of my database classes.

From what I understand, the SQL query looks good. I would be very interested to know how long this takes if there are 100k blocks with 10k DeferredContracts and half of them executed.

This would probably also work as a generic proxy interface to byzcoin. I think you can reflect a lot of queries with that. One point where you could go crazy is to actually represent the state of the Instances at a given block-number and then let the user query that one. But for this you'd have to go down a rabbit hole of keeping the global state updated for every block...