Add support for recording SHA3 preimages #3543

Arachnid · 2017-01-10T12:47:37Z

No description provided.

mention-bot · 2017-01-10T12:47:38Z

@Arachnid, thanks for your PR! By analyzing the history of the files in this pull request, we identified @obscuren, @jimenezrick and @karalabe to be potential reviewers.

fjl · 2017-01-10T17:56:12Z

core/database_util.go

@@ -595,6 +600,31 @@ func GetMipmapBloom(db ethdb.Database, number, level uint64) types.Bloom {
 	return types.BytesToBloom(bloomDat)
 }

+func GetPreimageTable(db ethdb.Database) ethdb.Database {
+	return ethdb.NewDatabaseTable(db, preimagePrefix)
+}


naming suggestion: "PreimageTable" (without Get prefix)

fjl · 2017-01-10T17:56:38Z

internal/web3ext/web3ext.go

+		}),
+		new web3._extend.Method({
+			name: 'getPreimage',
+			call: 'debug_getPreimage',


The new operation should be called debug_preimage

fjl · 2017-01-10T17:57:30Z

core/blockchain.go

@@ -1003,6 +1005,10 @@ func (self *BlockChain) InsertChain(chain types.Blocks) (int, error) {
 			if err := WriteMipmapBloom(self.chainDb, block.NumberU64(), receipts); err != nil {
 				return i, err
 			}
+			// Write hash preimages
+			if err := WritePreimages(self.chainDb, block.NumberU64(), self.stateCache.Preimages()); err != nil {


Please write preimages as part of the state commit.

Doing this introduces an import cycle - state imports core (for database_util), which imports state. This could be solved by removing all the preimage stuff from database_util, but reducing duplication in the number of places where we specify table prefixes etc seems like a Good Thing. Thoughts?

fjl · 2017-01-10T17:58:08Z

core/state/statedb.go

+
+func (self *StateDB) Preimages() map[common.Hash][]byte {
+	return self.preimages
+}


You don't need this method if you write them during Commit here.

fjl · 2017-01-10T17:58:45Z

core/state/statedb.go

@@ -199,6 +204,19 @@ func (self *StateDB) Logs() []*types.Log {
 	return logs
 }

+func (self *StateDB) AddPreimage(hash common.Hash, preimage []byte) {


Please document this. It should say that the method is used for recording preimages seen by the VM.

fjl · 2017-01-11T10:08:52Z

eth/api.go

@@ -560,3 +560,8 @@ func (api *PrivateDebugAPI) TraceTransaction(ctx context.Context, txHash common.
 	}
 	return nil, errors.New("database inconsistency")
 }
+
+func (api *PrivateDebugAPI) GetPreimage(ctx context.Context, hash common.Hash) (hexutil.Bytes, error) {


This needs to be renamed to Preimage

fjl · 2017-01-11T10:09:10Z

internal/web3ext/web3ext.go

@@ -385,6 +385,12 @@ web3._extend({
 			call: 'debug_traceTransaction',
 			params: 2,
 			inputFormatter: [null, null]
+		}),
+		new web3._extend.Method({
+			name: 'getPreimage',


Please call it preimage in JS to reduce confusion.

GitCop · 2017-01-11T11:50:57Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 82fb946a1835c35da3fae4d627e19825bfab1107
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

Arachnid · 2017-01-11T11:53:02Z

After running some stats on a full-synced node, there are about 2M EVM sha3 preimages, with ~370MB of values (plus ~40 bytes per key, plus overhead). Given the low-ish volume, I'm thinking it probably makes sense to dump the flag and just record these unconditionally like we do for trie preimages.

For context, there are 21M trie preimages, totalling ~446M of values.

Thoughts?

obscuren · 2017-01-11T14:40:43Z

eth/backend.go

@@ -97,8 +98,9 @@ type Config struct {
 	GpobaseStepUp           int
 	GpobaseCorrectionFactor int

-	EnableJit bool
-	ForceJit  bool
+	EnableJit               bool


This one and ForceJit may be deleted.

GitCop · 2017-01-11T15:31:37Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 82fb946a1835c35da3fae4d627e19825bfab1107
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

obscuren · 2017-01-11T21:35:11Z

core/database_util.go

@@ -595,6 +600,31 @@ func GetMipmapBloom(db ethdb.Database, number, level uint64) types.Bloom {
 	return types.BytesToBloom(bloomDat)
 }

+func PreimageTable(db ethdb.Database) ethdb.Database {


Please document this function

obscuren · 2017-01-11T21:35:20Z

core/database_util.go

+	return ethdb.NewTable(db, preimagePrefix)
+}
+
+func WritePreimages(db ethdb.Database, number uint64, preimages map[common.Hash][]byte) error {


obscuren · 2017-01-11T21:36:14Z

eth/api.go

@@ -560,3 +560,8 @@ func (api *PrivateDebugAPI) TraceTransaction(ctx context.Context, txHash common.
 	}
 	return nil, errors.New("database inconsistency")
 }
+
+func (api *PrivateDebugAPI) Preimage(ctx context.Context, hash common.Hash) (hexutil.Bytes, error) {


GitCop · 2017-01-12T10:26:38Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 82fb946a1835c35da3fae4d627e19825bfab1107
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

fjl · 2017-01-12T18:31:52Z

Please rebase on master to remove the ethdb commit.

…ages

GitCop · 2017-01-12T20:28:34Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 303da33
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

fjl · 2017-01-13T10:04:55Z

@Arachnid tests are broken

GitCop · 2017-01-13T10:34:02Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 303da33
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

obscuren added the in progress label Jan 10, 2017

Arachnid requested review from fjl and obscuren January 10, 2017 12:48

fjl requested changes Jan 10, 2017

View reviewed changes

Arachnid force-pushed the preimage branch 3 times, most recently from 6d122cf to 08c1213 Compare January 11, 2017 09:46

fjl requested changes Jan 11, 2017

View reviewed changes

Arachnid added review and removed in progress labels Jan 11, 2017

obscuren reviewed Jan 11, 2017

View reviewed changes

obscuren suggested changes Jan 11, 2017

View reviewed changes

obscuren approved these changes Jan 12, 2017

View reviewed changes

Arachnid added 6 commits January 12, 2017 20:28

cmd, core, eth, internal, light: Add support for recording SHA3 preim…

7641629

…ages

core, eth, ethdb, internal: Changes in response to review

5793542

internal/web3ext: Remove extraneous 'get'

34c0fd1

Rename GetPreimage to Preimage

303da33

eth: Remove accidentally readded EnableJit/ForceJit variables

f19241b

core, eth: Doc changes

faafd42

Arachnid force-pushed the preimage branch from 1a4b1e7 to faafd42 Compare January 12, 2017 20:28

accounts, core, eth, les, light, tests: Fix tests

7953d43

karalabe modified the milestones: 1.5.8, 1.5.7 Jan 16, 2017

fjl approved these changes Jan 17, 2017

View reviewed changes

fjl merged commit 17d9223 into ethereum:master Jan 17, 2017

fjl removed the review label Jan 17, 2017

farazdagi pushed a commit to status-im/go-ethereum that referenced this pull request Feb 23, 2017

cmd/geth, core: add support for recording SHA3 preimages (ethereum#3543)

7e93cfc

cdetrio mentioned this pull request May 4, 2017

initial support for mapping types ethereum/remix#498

Merged

cdetrio mentioned this pull request Jan 1, 2018

eth/api_tracer.go: enable preimage recording when tracing #15787

Merged

cdetrio mentioned this pull request Apr 21, 2018

testeth via RPC requirements ethereum/retesteth#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for recording SHA3 preimages #3543

Add support for recording SHA3 preimages #3543

Arachnid commented Jan 10, 2017

mention-bot commented Jan 10, 2017

fjl Jan 10, 2017

fjl Jan 10, 2017

fjl Jan 10, 2017

Arachnid Jan 11, 2017 •

edited

Loading

fjl Jan 10, 2017

fjl Jan 10, 2017

fjl Jan 11, 2017

fjl Jan 11, 2017

GitCop commented Jan 11, 2017

Arachnid commented Jan 11, 2017 •

edited

Loading

obscuren Jan 11, 2017

GitCop commented Jan 11, 2017

obscuren Jan 11, 2017

obscuren Jan 11, 2017

obscuren Jan 11, 2017

GitCop commented Jan 12, 2017

fjl commented Jan 12, 2017

GitCop commented Jan 12, 2017

fjl commented Jan 13, 2017

GitCop commented Jan 13, 2017

Add support for recording SHA3 preimages #3543

Add support for recording SHA3 preimages #3543

Conversation

Arachnid commented Jan 10, 2017

mention-bot commented Jan 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arachnid Jan 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitCop commented Jan 11, 2017

Arachnid commented Jan 11, 2017 • edited Loading

Choose a reason for hiding this comment

GitCop commented Jan 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitCop commented Jan 12, 2017

fjl commented Jan 12, 2017

GitCop commented Jan 12, 2017

fjl commented Jan 13, 2017

GitCop commented Jan 13, 2017

Arachnid Jan 11, 2017 •

edited

Loading

Arachnid commented Jan 11, 2017 •

edited

Loading