v0.7.7

What's Changed

• Reduce FDB cache for disk pages by @masih in #2076
• Reduce FDB log process cache to 12GiB by @masih in #2077
• Use the same FDB config as prod on dev by @masih in #2081
• feat: add tornado tenant by @kylehuntsman in #2085
• feat: suspend autoretrieve namespace CD by @kylehuntsman in #2086
• Create KMS key for Tornado team as a tenant by @masih in #2087
• Remove autoretrieve as a tenant by @masih in #2088
• Expand subnets for dev-ue2a-r6a-xl node group by @masih in #2091
• Update to new dhstore by @gammazero in #2083
• Replace dhfind with new dhstore by @gammazero in #2084
• Replace dhfind stateless with new dhstore stateless by @gammazero in #2092
• Update dhstore to new version with dhfind metrics by @gammazero in #2093
• Update to new dhstore with dhfind metrics by @gammazero in #2094
• Deploy HeyFil API on dev and prod with persistent storage by @masih in #2095
• Fix missing subdomain in heyfil prod ingress by @masih in #2096
• Remove ber and cali indexers by @gammazero in #1888
• Remove dhfind instances from dev by @gammazero in #2102
• Remove dhfind instances from production by @gammazero in #2103
• Reduce dhstore stateless instances to 2 by @masih in #2113
• Separate FDB data distributors on prod by @masih in #2114
• Use new FDB indexers as providers backends. by @gammazero in #2116
• Stop dido, kepa, and oden indexers by @gammazero in #2115
• Recover FDB prod cluster from too many old generations by @masih in #2118
• Remove old unused snapshots by @masih in #2117
• Upgrade to the latest FDB controller by @masih in #2119
• Remove index counter functionality by @gammazero in #2101
• Add FDB 7.2 and 7.3 binaries to controller in prep for upgrade by @masih in #2122
• Upgrade production FDB to 7.3.7 by @masih in #2123
• Increase max generations for FDB in prod during upgrade by @masih in #2124
• Revert "Increase max generations for FDB in prod during upgrade" by @masih in #2125
• Disable replace in prod fdb and undeploy dhstore stateless by @masih in #2126
• Replace E-notation in prod FDB config with zeros by @masih in #2127
• Refine prod FDB config to stay below max memory by @masih in #2128
• Resume dhstore stateless on prod and upgrade to FDB client library 7.3.7 by @masih in #2129
• Move prod FDB to Redwood storage engine by @masih in #2130
• Reduce compaction thresholds back on qui by @masih in #2140
• Expand prod dhstore pvc for maximum headroom by @masih in #2139
• Update go-libipni to v0.3.0 by @gammazero in #2150
• Fix panic in bootstrap by @gammazero in #2155
• Create a new dhstore node in dev since dhstore-ago2 is out of space by @gammazero in #2158
• Remove reframe by @gammazero in #2157
• Do not add extra newline after json records by @gammazero in #2161
• Create ECR for telemetry service by @gammazero in #2162
• Exclude FDB indexers from serving as providers backends by @gammazero in #2163
• Shut down FDB on dev by @masih in #2166
• Include telemetry service in dev deploy manifests by @masih in #2167
• Add noop indexer to assigner to avoid crash loopback by @masih in #2169
• Add missing deployment config for telemetry by @gammazero in #2170
• Limit max ad chain depth to prevent large memory use by @gammazero in #2174
• Shut down FDB and indexer backed by it on prod by @masih in #2171
• Remove all non-encrypted indexers and their records by @masih in #2175
• Fix telemetry memory use by removing concurrent distance tracking by @gammazero in #2178
• Do not persist bootstrap peers by @gammazero in #2187
• fix memory leak by @gammazero in #2189
• Allow assigner to start with zero indexers by @gammazero in #2188
• Fix panic due to unassigned bootstrap config functions by @gammazero in #2191
• Fix error count in telemetry by @gammazero in #2194
• Rotate github tokens used by Flux CD by @masih in #2195
• Update to latest version of go-libipni by @gammazero in #2197

New Contributors

• @kylehuntsman made their first contribution in #2085

Full Changelog: v0.7.6...v0.7.7

v0.7.6

What's Changed

• increase pvc size for dhstore-ago2 by @gammazero in #2055
• Deploy latest to inga by @gammazero in #2056
• Revert "Deploy latest to inga" by @gammazero in #2057
• Revert "Revert "Deploy latest to inga"" by @gammazero in #2058
• Investigating high CPU usage: Reduce worker count on inga to 5 by @masih in #2059
• Add old indexers as provider backend while investigating inga issues by @masih in #2060
• Revert inga to 0.7.3 to investigate excessive CPU usage by @masih in #2061
• Increase inga CPU by @gammazero in #2063
• Revert "Add old indexers as provider backend while investigating inga issues" by @masih in #2064
• Increase workers on inga to 50 now that CPU usage has settled by @masih in #2065
• Revert "Increase inga CPU" by @masih in #2066
• Revert "Revert "Add old indexers as provider backend while investigating inga issues"" by @masih in #2068
• Disable index counter on inga to restore service in prod by @masih in #2069
• Make index counts configurable and disable by default by @masih in #2070

Full Changelog: v0.7.5...v0.7.6

v0.7.5

What's Changed

• Deploy new dhstore-ago2 and replace dhfind-ago2 by @gammazero in #2042
• Revert "Deploy new dhstore-ago2 and replace dhfind-ago2" by @gammazero in #2046
• Deploy new dhstore-ago2 and replace dhfind-ago2 by @gammazero in #2048
• Update to latest go-libipni by @gammazero in #2047
• Deploy latest to dev environment by @github-actions in #2044
• Deploy latest to ago by @gammazero in #2049
• Persist LastError across restarts by @gammazero in #2050
• Replace last dev dhfind with dhstore by @gammazero in #2051
• Deploy latest to dev environment by @github-actions in #2053

Full Changelog: v0.7.4...v0.7.5

v0.7.4

What's Changed

• Move dhstore porvy to compute optimised instance types by @masih in #2019
• Reduce resources on provy slightly so that it fits by @masih in #2020
• Reduce under-utilised storage given to FDB log process class by @masih in #2021
• Upgrade to latest FDB operator by @masih in #2022
• Add dedicated coordinators to FDB prod cluster by @masih in #2023
• Exclude FDB backed indexers from provider backend on prod by @masih in #2026
• Increase max outstanding IO operations to 256 by @masih in #2027
• Move cera to compute optimised instance type by @masih in #2028
• Increase fetch key and relocation parallelism on prod FDB by @masih in #2029
• Make rewriting the mirror when resyncing configurable by @gammazero in #2030
• Set last error on sync failure by @gammazero in #2024
• Update version of go-libipni by @gammazero in #2032
• Update the github certificate thumbprints by @gammazero in #2033
• Remove oden as provider backend since inga is caught up by @masih in #2035
• Deploy LastError test code to ago by @gammazero in #2038
• Roll out fix to dhfind provider aggregation by @masih in #2039
• Clear LastError on good sync by @gammazero in #2037

Full Changelog: v0.7.3...v0.7.4

v0.7.3

What's Changed

• Deploy latest to prod environment by @github-actions in #1997
• Deploy fdbmeter to collect FoundationDB prometheus metrics by @masih in #1998
• Upgrade to the latest fdbmeter by @masih in #1999
• Enable write to mirror on inga by @masih in #2000
• Deploy fix to indexstar result aggregation by @masih in #2002
• Remove no longer supported translate reframe flag by @masih in #2003
• Remove redundant reframe routing from cloudfront by @masih in #2005
• Use new go-libipni that has updated latest sync tracking by @gammazero in #2004
• Investigate high FDB storage lag by reducing dhstore batch to 16 by @masih in #2007
• Move inga to instance types with more CPU by @masih in #2008
• Expand private subnets in prod by @masih in #2009
• Deploy latest to dev environment by @github-actions in #1995
• Reduce worker count on inga to 10 by @masih in #2010
• Exclude inga and add FDB-backed indexers as provider backends by @masih in #2011
• New version for latest sync tracking fix by @gammazero in #2012

Full Changelog: v0.7.2...v0.7.3

v0.7.2

What's Changed

• Add oden as indexstar backend in prod due to high 404 rate by @masih in #1979
• Connect FDB backed instances to prod read path by @masih in #1980
• Scale up dhstore stateless in prod to 5 replicas by @masih in #1981
• Exclude FDB backed instances from read path by @masih in #1982
• Deploy fix for bug in dhfind incorrect IPNI non-streaming reponse by @masih in #1984
• Update core and have dhstore as a valuestore by @gammazero in #1989
• Update to go-libipni with shorter data-transfer timeout by @gammazero in #1994

Full Changelog: v0.7.1...v0.7.2

v0.7.1

What's Changed

• Deploy latest assigner by @gammazero in #1897
• Deploy latest assigner to prod by @gammazero in #1898
• Peer dev assigner to prod assigner by @gammazero in #1899
• Deploy latest to dev by @gammazero in #1900
• Deploy latest to prod environment by @github-actions in #1896
• Increase inga indexer volume to 160G by @masih in #1901
• Deploy the latest version of cassette in prod by @masih in #1902
• Update to latest go-libipni by @gammazero in #1903
• Deploy FDB controller manager in prod by @masih in #1904
• Add additional gp3 storage classes to prod by @masih in #1905
• No valuestore for inga by @gammazero in #1907
• Increase PVC of inga to 1Ti by @masih in #1908
• Deploy r6a.2xl worker node group on prod cluster by @masih in #1909
• Fix typo in prod node workgroup IAM role name by @masih in #1910
• Disconnect dido and kepa from reads in prod by @ischasny in #1912
• New indexers for indexing with FDB by @gammazero in #1906
• Deploy latest to cassette in dev and prod by @masih in #1914
• Revert "Disconnect dido and kepa from reads in prod" by @ischasny in #1915
• Bump up number of inga's threads by @ischasny in #1916
• Investigate effect of cancelling WANTs in cassette more aggressively by @masih in #1917
• Add separate metric for non cancel CID count in cassette by @masih in #1919
• Deploy latest to cassette by @masih in #1920
• Change dev assigner service hostname by @gammazero in #1921
• Deploy cassette send timeout by @masih in #1922
• Revert "Revert "Temporarily turn inga off to trigger compaction"" by @ischasny in #1924
• Deploy cassette receiver buffer config by @masih in #1925
• Revert "Deploy cassette receiver buffer config" by @masih in #1926
• Revert "Revert "Revert "Temporarily turn inga off to trigger compaction""" by @ischasny in #1927
• Revert "Revert "Revert "Revert "Temporarily turn inga off to trigger compaction"""" by @ischasny in #1928
• Temporarily disconnect the current dhstore by @ischasny in #1929
• Deploy dhstore with experimental configuration to ago2 by @ischasny in #1930
• Pause ago as it has been imbalanced for the last 2 weeks by @ischasny in #1931
• Try experimental dhstore configuration in prod by @ischasny in #1932
• Move inga to the same instance type as other indexer nodes in prod by @ischasny in #1933
• Add required toleration to inga by @ischasny in #1934
• Revert "Pause ago as it has been imbalanced for the last 2 weeks" by @ischasny in #1935
• Deploy latest to cassette by @masih in #1936
• Deploy latest to cassette by @masih in #1937
• Deploy latest dhstore experimental configuration by @ischasny in #1938
• Deploy latest experimental dhstore to ago2 by @ischasny in #1939
• Increase inga ingest worker count to 15 by @ischasny in #1941
• Upgrade dev to K8S 1.23 by @masih in #1942
• Increase max r6a.xlarge instances to 5 in prod by @ischasny in #1940
• Increase number of CPUs on inga by @ischasny in #1943
• Bump up inga CPU to 30 by @ischasny in #1944
• Reconnect kepa and dido to reads by @ischasny in #1946
• Upgrade prod to K8S 1.23 by @masih in #1947
• Create worker node group on prod for FDB by @masih in #1948
• Remove redundant users from K8S by @masih in #1949
• Bump up inga's ingest worker count on par with others by @ischasny in #1945
• Deploy FDB and stateless dhstore on prod by @masih in #1950
• Reduce inga poll interval by @ischasny in #1951
• Revert "Reconnect kepa and dido to reads" by @ischasny in #1952
• Reduce FDB memory demand to fit allocatable memory in c6a.8xl by @masih in #1953
• Reduce FDB memory demand further by @masih in #1954
• Fix type in prod stateless dhstore deployment by @masih in #1955
• Remove FDB ratekeeper memory limits by @masih in #1956
• Assigner configuration for new prod indexers by @gammazero in #1911
• Disconnect oden by @ischasny in #1958
• Increase CPUs for prod dhstore by @ischasny in #1959
• Give prod dhstore more RAM by @ischasny in #1960
• Override app label in dhstore stateless to restrict service selectors by @masih in #1961
• Deploy new indexstar to dev by @ischasny in #1962
• Deploy latest indexstar to dev by @ischasny in #1964
• Shutdown ber and cali indexers by @gammazero in #1965
• Deploy caching in cassette by @masih in #1963
• Deploy the latest indexstar to prod by @ischasny in #1966
• Deploy latest to cassette with option to not cache 404s by @masih in #1967

Full Changelog: v0.7.0...v0.7.1

v0.7.0

What's Changed

• Update dev double hashing configuration by @ischasny in #1772
• Gracefully skip not found content from HTTP publishers by @masih in #1774
• Remove support for pogreb as valuestore by @gammazero in #1803
• Gen ident utilty by @gammazero in #1806
• Use find command from ipni-cli instead of from provider by @gammazero in #1808
• add error check to e2e test by @gammazero in #1817
• Include version in indexer and assigner health message by @gammazero in #1807
• Freezer monitors mulitple directories different devices by @gammazero in #1797
• Renew GitHub token used by CI/CD by @masih in #1822
• Does position in the indexstar list affect number of received requests by @ischasny in #1836
• Deploy FoundationDB CRDs in dev and define manager manifests by @masih in #1845
• Define new gp3 storgage classes in dev by @masih in #1847
• Return Not Found message with 404 by @gammazero in #1826
• Remove rate limiting by @gammazero in #1835
• When using assigner automatically assign known publishers by @gammazero in #1841
• Expand dev work group subnets for r6a-xl instance types by @masih in #1849
• Move admin and ingest servers, remove libp2p find server by @gammazero in #1838
• Remove commands that are implemented in ipni-cli by @gammazero in #1821
• Remove DH key-sharding config by @gammazero in #1850
• Update dependencies by @gammazero in #1851
• Build container with go1.20.x by @gammazero in #1852
• Investigate web ui issue by intexstar by @masih in #1857
• Define workgroup to run foundationdb nodes by @masih in #1859
• Create new indexer instances by @gammazero in #1864
• Store temporary ad data in separate datastore by @gammazero in #1695
• Connect dhstores to indexstar in dev by @ischasny in #1876
• Config assigner to use new indexers for FDB valuestore by @gammazero in #1879
• Explicitly set dhstore stateless resources in dev by @masih in #1884
• Increase FDB storage processes per pod to 5 by @masih in #1886
• Create FDB CRDs in prod by @ischasny in #1889
• Expose dev assigner p2p port for peering by @gammazero in #1891
• Expose port 3003 on assigner pod by @gammazero in #1892
• Remove support for storethehash valuestore by @gammazero in #1894

Full Changelog: v0.6.9...v0.7.0

v0.6.9

What's Changed

• Deploy latest to dev environment by @github-actions in #1717
• Deploy latest to prod environment by @github-actions in #1731
• Increase worker count on inga to 20 by @gammazero in #1732
• Increase workers on ago to 15 by @gammazero in #1733
• Fewer workers on inga to stop overwhelming dhstore by @gammazero in #1734
• Reduce workers to 5 on inga by @gammazero in #1735
• Reduce inga worker count to 1 by @ischasny in #1736
• Revert "Reduce inga worker count to 1" by @ischasny in #1737
• Reduce inga worker count to 3 by @ischasny in #1738
• Temporarily turn inga off to trigger compaction by @ischasny in #1739
• Set DHBatchSize to 1 on ago as experiment by @gammazero in #1740
• increase ago dh batch size to 8k by @gammazero in #1741
• Revert "Temporarily turn inga off to trigger compaction" by @ischasny in #1742
• Automate image update for lassie-event-recorder by @masih in #1743
• Shutdown ago for now to let dhstore catch up by @gammazero in #1744
• Switch writes to the original dhstore by @ischasny in #1747
• Revert "Shutdown ago for now to let dhstore catch up" by @gammazero in #1746
• Stop ago-indexer and increase dhstore pvc size to max by @gammazero in #1749
• Define initial dhstore cluster in dev by @masih in #1748
• Increase memory and CPU limits for dhfind by @gammazero in #1750
• Bump dhstore CPU to 7 in prod by @ischasny in #1753
• Experiment with larger CPU on dhstore by @masih in #1755
• Start ago again now that dhstore has more space by @gammazero in #1756
• Fix assigner service announce endpoint by @gammazero in #1754
• Switch double hashed writes to Helga by @ischasny in #1758
• Build and push images from dh-key-sharding for testing by @masih in #1761
• deploy assigner that serves correct announce path by @gammazero in #1763
• Deploy a new indexer in dev to test sharding writes by @masih in #1762
• deploy assigner that serves correct announce path by @gammazero in #1764
• Increase max ASG for r6a nodes in dev by @masih in #1765
• Change ExtendedProviders behaviour to 'replace' instead of 'add' by @ischasny in #1751
• Increase CPU limit for dhstore cluster envoy in dev by @masih in #1767

Full Changelog: v0.6.8...v0.6.9

v0.6.8

What's Changed

• Deploy latest to prod environment by @github-actions in #1718
• Roll out lookup coverage check for Internet Archive top CIDs by @masih in #1720
• Update oden and kepa to the latest version by @ischasny in #1719
• Make sti nodes first indexstar backends by @gammazero in #1722
• rename document by @gammazero in #1724
• Bump github.com/docker/distribution from 2.8.0+incompatible to 2.8.2+incompatible by @dependabot in #1723
• Scale up dhfind by @ischasny in #1726
• Bump up dhfind CPU limits in prod by @ischasny in #1727
• Update to libipni with more gs requests by @gammazero in #1728
• Deploy increased gs request limit to ago by @gammazero in #1729

Full Changelog: v0.6.7...v0.6.8