Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scraper: Change open mode from append to truncate for auth file #2528

Merged

Conversation

jamescowens
Copy link
Member

There is no need to retain prior history in this file, as old eTags are worthless in this context. Using append mode results in a slowing growing file over time as the scraper operates, which wastes storage and slows processing.

There is no need to retain prior history in this file, as old
eTags are worthless in this context.
@jamescowens jamescowens self-assigned this Jul 1, 2022
@jamescowens jamescowens added the bug label Jul 1, 2022
@jamescowens jamescowens added this to the Kermit's Mom milestone Jul 1, 2022
@jamescowens
Copy link
Member Author

This has been tested in the same four-node testnet fork that has been testing the Poll v3 and Project v2.

@jamescowens jamescowens merged commit a0f2ffa into gridcoin-community:development Jul 1, 2022
jamescowens added a commit to jamescowens/Gridcoin-Research that referenced this pull request Jul 31, 2022
Added
 - test: Add TrimString(...) tests gridcoin-community#2447 (@barton2526)
 - test: Add dead code detection gridcoin-community#2449 (@barton2526)
 - test: Add explicit references to related CVE's in comments gridcoin-community#2467 (@barton2526)
 - test: Add testing of ParseInt/ParseUInt edge cases with leading +/-/0:s gridcoin-community#2470 (@barton2526)
 - consensus, contract, mining, researcher, rpc, staking, gui: Implementation of MRC - baseline functionality gridcoin-community#2425 (@jamescowens)
 - consensus: MRC mandatory implementation code gridcoin-community#2471 (@jamescowens)
 - test: Add upstream sync_tests.cpp gridcoin-community#2481 (@barton2526)
 - net: Countermeasures against eclipse attacks gridcoin-community#2454 (@Pythonix)
 - lint: add script to check for https violations gridcoin-community#2491 (@div72)
 - util: Add flatpath BOINC data directory path resolution for Linux gridcoin-community#2499 (@jamescowens)
 - gui: Add beaconExpired() to researchermodel gridcoin-community#2498 (@jamescowens)
 - consensus: Add missing block nVersion check for v12 blocks in AcceptBlock gridcoin-community#2502 (@jamescowens)
 - gui, util: Add AccrualChangedFromStakeOrMRC core signal gridcoin-community#2503 (@jamescowens)
 - util: Change default -dbcache to 100 MB and also implement -txindexdbcache gridcoin-community#2507 (@jamescowens)
 - rpc, util, consensus: Implement exception handling framework for MRC and fix ValidateMRC to deal with testnet consensus issue gridcoin-community#2508 (@jamescowens)
 - gui: Initial implementation of GUI MRC submission form gridcoin-community#2513 (@jamescowens)
 - build: Port over Bitcoin's translation docs gridcoin-community#2439 (@jamescowens)
 - (2/3) build: integrate libsecp256k1 gridcoin-community#2492 (@div72)
 - gui: New MRC request icon gridcoin-community#2526 (@jamescowens)
 - mandatory, voting: Implement poll type validation in protocol gridcoin-community#2522 (@jamescowens)
 - gui, voting: Implement poll additional fields gui components gridcoin-community#2525 (@jamescowens)
 - gui, researcher: Add GDPR protection display gridcoin-community#2527 (@jamescowens)
 - consensus, rpc: Kermit's mom hardfork (2671700) gridcoin-community#2551 (@jamescowens)

Changed
 - net: Hard Coded Seed Node Cleanup gridcoin-community#2427 (@barton2526)
 - script: Add More Generated Files to Gitignore gridcoin-community#2435 (@RoboticMind)
 - gui: Update copyright year to 2022 for Gridcoin About dialog box gridcoin-community#2443 (@jamescowens)
 - rpc: Change type field in ListTransactions to lower case gridcoin-community#2441 (@jamescowens)
 - refactor: Replace memset calls with array initialization gridcoin-community#2452 (@barton2526)
 - refactor: Changed some parameters from pass by value to pass by reference gridcoin-community#2455 (@Pythonix)
 - ci, cd: improve caching gridcoin-community#2461 (@div72)
 - contrib: port recent macdeployqtplus changes gridcoin-community#2465 (@div72)
 - test: Test for expected return values when calling functions returning a success code gridcoin-community#2464 (@barton2526)
 - build: Improve error message when pkg-config is not installed gridcoin-community#2460 (@barton2526)
 - test: Bump shellcheck, mypy versions gridcoin-community#2463 (@barton2526)
 - build: Update depends packages (expat, fontconfig, freetype, libXau, libxcb, xcb_proto, xproto) gridcoin-community#2466 (@barton2526)
 - lint: run mypy over contrib/devtools gridcoin-community#2475 (@barton2526)
 - build, lint: Remove x-prefix's from comparisons, Fix some shell script issues the linter complains about, Re-enable boost include checks gridcoin-community#2478 (@barton2526)
 - test: Avoid copies of CTransaction gridcoin-community#2479 (@barton2526)
 - ci: change windows CI to Focal, modify wrap_wine to use wine64 for 64bit binaries gridcoin-community#2484 (@barton2526)
 - build: Qt 5.15.2 gridcoin-community#2486 (@barton2526)
 - net: No longer send local address in addrMe gridcoin-community#2459 (@Pythonix)
 - voting, gui, rpc: Enhance PollResult and AVW calculation to improve pool handling gridcoin-community#2489 (@jamescowens)
 - (1/3) refactor: port some misc changes from upstream gridcoin-community#2485 (@div72)
 - build: Try posix-specific CXX first for mingw32 host, Fix Windows cross-compiling with Qt 5.15 gridcoin-community#2494 (@barton2526)
 - Improve upon scanforunspent rpc gridcoin-community#2468 (@iFoggz)
 - rpc: Change tail_fee and head_fee to display in GRC rather than Halfords in createmrcrequest gridcoin-community#2501 (@jamescowens)
 - scripted-diff: change http to https in copyright text gridcoin-community#2504 (@div72)
 - qt, refactor: Use enum type as switch argument in *TableModel gridcoin-community#2496 (@barton2526)
 - build, qt: bump Qt5 version to 5.15.3 gridcoin-community#2510 (@barton2526)
 - utils: run commands using utf-8 string on Windows gridcoin-community#2514 (@barton2526)
 - prevector: enforce is_trivially_copyable_v gridcoin-community#2516 (@div72)
 - crypto: Unroll the ChaCha20 inner loop for performance gridcoin-community#2515 (@div72)
 - gui: Modify VerifyTCPPort to use the status of CheckOutboundConnectionCount gridcoin-community#2506 (@jamescowens)
 - gui: Fix transaction history table column size behavior gridcoin-community#2520 (@jamescowens)
 - log: Use consistent wording in random.cpp log gridcoin-community#2538 (@div72)
 - lint: Use newer versions of our lint packages, remove yq gridcoin-community#2541 (@barton2526)
 - (1/2) validation: move CBlock validation methods to validation.cpp gridcoin-community#2539 (@div72)
 - gui: Implement proportional column resizing for Addressbook with memory gridcoin-community#2543 (@jamescowens)
 - build: remove redundant warning flags gridcoin-community#2546 (@barton2526)
 - qt: Prefix makefile variables with QT_ gridcoin-community#2547 (@barton2526)
 - build: remove build stubs for external leveldb gridcoin-community#2550 (@barton2526)
 - build, refactor: Improve package version usage gridcoin-community#2549 (@barton2526)
 - build: minor boost tidyups gridcoin-community#2548 (@barton2526)

Removed
 - rpc, util: Remove caching from BlockFinder gridcoin-community#2490 (@jamescowens)
 - test: remove obsolete check sig test gridcoin-community#2552 (@div72)

Fixed
 - build: fix unoptimized libraries in depends gridcoin-community#2428 (@barton2526)
 - build: don't use deprecated brew package names gridcoin-community#2429 (@barton2526)
 - qt: fix shutdown on MacOS gridcoin-community#2440 (@div72)
 - net: Do not add random inbound peers to addrman gridcoin-community#2451 (@barton2526)
 - util: skip trying to set the locale on NetBSD gridcoin-community#2448 (@barton2526)
 - build: change bundle id gridcoin-community#2462 (@div72)
 - net: Do not propagate obviously poor addresses onto the network gridcoin-community#2453 (@Pythonix)
 - ci: Fix CI build title to reflect that we are building for bionic, not xenial gridcoin-community#2469 (@barton2526)
 - lint: Fix misc typos gridcoin-community#2472 (@barton2526)
 - util: Fix crash when parsing command line with -noincludeconf=0, Properly handle -noincludeconf on command line gridcoin-community#2473 (@barton2526)
 - build: Fix several minor linter errors gridcoin-community#2476 (@jamescowens)
 - tests: Don't access out of bounds array index: array[sizeof(array)] gridcoin-community#2480 (@barton2526)
 - script: Fix and Minify Icon SVG gridcoin-community#2488 (@RoboticMind)
 - gui: Add missing null pointer check for m_beacon gridcoin-community#2500 (@jamescowens)
 - util: Fix BN_zero macro in key.cpp for OpenSSL 3.0 gridcoin-community#2497 (@jamescowens)
 - rpc, contract: Adjust ValidateMRC, CreateMRC, and createmrcrequest to correct provided fee handling gridcoin-community#2505 (@jamescowens)
 - consensus: Move DoS into contract validators to allow variability of DoS based on context and further fixes to ValidateMRC gridcoin-community#2512 (@jamescowens)
 - lockedpool: When possible, use madvise to avoid including sensitive information in core dumps gridcoin-community#2509 (@barton2526)
 - refactor: Fix some minor linter complaints gridcoin-community#2517 (@jamescowens)
 - miner: Miner Logger bug fix gridcoin-community#2518 (@iFoggz)
 - key: properly parse short DER private keys gridcoin-community#2519 (@div72)
 - researcher: Fix ReadClientStateXml() crash on wrong BOINC directory permissions gridcoin-community#2524 (@jamescowens)
 - init: fix daemon forking gridcoin-community#2521 (@div72)
 - scraper: Change open mode from append to truncate for auth file gridcoin-community#2528 (@jamescowens)
 - gui: New mrc contract icon try #2 gridcoin-community#2529 (@jamescowens)
 - gui: Remove white outlines on MRC icon gridcoin-community#2530 (@a123b)
 - voting: Change m_additional_fields serialization gridcoin-community#2531 (@jamescowens)
 - build, qt: Fix `QMAKE_CXXFLAGS` expression for `mingw32` host gridcoin-community#2537 (@div72)
 - logging: fix logging empty thread name gridcoin-community#2535 (@div72)
 - trivial: fix comment in account header guard gridcoin-community#2542 (@div72)
 - build: Restrict check for CRC32C intrinsic to aarch64 gridcoin-community#2544 (@barton2526)
 - build: force CRCCheck in Windows installer gridcoin-community#2545 (@barton2526)
@jamescowens jamescowens deleted the fix_scraper_auth branch November 4, 2022 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant