Release Integrate QF with super fluid streamed donations (#1555)

* add activeQfRoundId to sortingBy InstantBoosting * add orderBy totalDonations and totalReactions * feat: add getRecurringDonationStats resolver * fix filtering by QF * remove qfRounds joins for non qf round filters * add some temp logs * remove temp logs * fix: changes test cases of recuring donations stats - create recored with createdAt field in past so test result won't be related to other endpoints test cases * Fix projectActualserviceView * fix stream balance depleted issue (#1496) Co-authored-by: mohammadranjbarz <mranjbar.z2993@gmail.com> * rebuild * refresh and fetch user address separately (#1499) * Added pg_trgm extension migration (#1502) * fix: change recurring donations stats query to single query * fix recurring donation count * WIP: projectIds textArea * fix actual matching cap (#1507) * fix query error * fix user donations count * fix recurring donation count tests * fix user recurring donation query * fix user recurring donation test * add donations relation to qfround * add findArchivedQfRounds endpoint * add findArchivedQfRounds endpoint * feat: add sponsors & banner images upload * add sortBy to findArchivedQfRounds * 1.23.3 * add new test graphql query * add tests for new QfArchivedRounds * fixes on qfArchivedRounds query * add new tests for qfArchivedRounds query * fix findArchivedQfRounds tests * fix: keep already uploaded sponsors images * fix skip and limit for findArchivedQfRounds * Add logs and refactor the bootstrap code to help investigate latency problem * Add poolSize to orm config * Fix eslint errors * remove changing squareRootSumOfProjects when cap is overflown * Trigger ortto activity when user saves their profile info for the first time (#1520) * add newUser to updateUser query * add createOrttoProfile * add createOrttoProfile to NotificationAdapterInterface * add createOrttoProfile to MockNotificationAdapter * add CREATE_ORTTO_PROFILE event * Allow to set the matching pool token & amount to be something other than usd (#1517) * Allow to set the matching pool token & amount to be something other than USD in adminjs * Allow to set the matching pool token & amount to be something other than USD in QFRound table * add null to allocatedTokenSymbol and allocatedTokenChainId * add nullable true to allocatedTokenSymbol and allocatedTokenChainId * add allocatedFundUSDPreferred and allocatedFundUSD to qfRound * Comment migrations * Hotfix db improvements (#1523) * add extra configurations for postgresql connections * add master and slave replication strategy for typeorm * add qfRound to qfRoundStats * fix qfRoundStatsQuery * Hotfix staging fix latency (#1528) * add project donation summary view entity * convert projectQueries to querybuilder * add cache to projectDonationSummary queries * add configurable cache to slow queries * remove massive recurring donation log * add await for project queries * Add logs to projectVerificationForm * Add logs to projectVerificationForm * fix: add project Ids list textarea for qf round edit * Master to staging (#1543) * Hotfix db improvements (#1523) (#1524) * add extra configurations for postgresql connections * add master and slave replication strategy for typeorm Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> * Fix/db replica production (#1525) * Hotfix db improvements (#1523) * add extra configurations for postgresql connections * add master and slave replication strategy for typeorm * Define db read only configs --------- Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> * update comment * Hotfix latency issues for prod (#1529) * Hotfix staging fix latency (#1528) * add project donation summary view entity * convert projectQueries to querybuilder * add cache to projectDonationSummary queries * add configurable cache to slow queries * remove massive recurring donation log * add await for project queries * Add informative logs for draft donation service job (#1537) * Fix eslint errors * Fix/master test (#1541) * Fixed master test issue * Returned test to master pipeline * Comment executing donation summary view --------- Co-authored-by: Mohammad Ranjbar Z <mranjbar.z2993@gmail.com> --------- Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> Co-authored-by: Carlos <carlos.quintero096@gmail.com> Co-authored-by: mohammadranjbarz <mranjbar.z2993@gmail.com> * Fix/word similarity - staging (#1546) * Hotfix db improvements (#1523) (#1524) * add extra configurations for postgresql connections * add master and slave replication strategy for typeorm Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> * Fix/db replica production (#1525) * Hotfix db improvements (#1523) * add extra configurations for postgresql connections * add master and slave replication strategy for typeorm * Define db read only configs --------- Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> * update comment * Hotfix latency issues for prod (#1529) * Hotfix staging fix latency (#1528) * add project donation summary view entity * convert projectQueries to querybuilder * add cache to projectDonationSummary queries * add configurable cache to slow queries * remove massive recurring donation log * add await for project queries * Add informative logs for draft donation service job (#1537) * Fix eslint errors * Fix/master test (#1541) * Fixed master test issue * Returned test to master pipeline * Comment executing donation summary view --------- Co-authored-by: Mohammad Ranjbar Z <mranjbar.z2993@gmail.com> * Fixed word similarity issue * Removed unused import --------- Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> Co-authored-by: Carlos <carlos.quintero096@gmail.com> Co-authored-by: mohammadranjbarz <mranjbar.z2993@gmail.com> * remove refresh_project_summary_from totals and add user cache (#1539) * Remove users field from project * Remove users field from filterProjectsQuery * Remove projects field from user * remove added logs * fix eslint errors * remove users from Project.create * remove users from Project.create in testUtils.ts * remove projectOwnerId * replace admin with adminUserId * Add recurring donation join to donations() endpoint (#1554) related to #1483 * replace admin with adminUserId in SEED data * replace admin with adminUserId * replace admin with adminUserId in projectResolver.ts * replace admin with adminUserId in projectsTab.ts * replace admin with adminUserId in projectResolver.test.ts * replace admin with adminUserId in projectResolver.ts * add allocatedFundUSD and allocatedTokenSymbol to qfArchivedRounds * fix nullable * remove admin from project * replace admin with adminUserId * replace admin with adminUserId * add adminUserId field * drop admin column * fix: add telegram to ProjectSocialMediaType enum to allow adding telegram url * Add some logs * Fix eslint errors * Add maxQueuedJobs for draft donation worker * Fix eslint errors * fix unstable test case * Disable concurrency for draft donation worker * add indexes to project_summary_view (#1568) * improve projectBySlug query * fix: change output types to float * Ignore small differences of amounts when matching draft donation for erc20 * add graphql-fields * add getVerificationFormStatusByProjectId * refactor projectBySlug * add findProjectIdBySlug * fix projectBySlug tests with new changes * fix projectBySlug tests with new changes * fix projectBySlug tests with new changes * remove projectVerificationForm assert * add logs * fix title in should return projects with indicated slug test * Add streamed mini donations to qf round (#1557) * Add streamed mini donations to qf round related to Giveth/giveth-dapps-v2#3284 * Fix eslint error * Fix eslint errors * make verificationFormStatus field public * Reduce test runnning time (#1574) * fix: reduce test runnning time * fix: add permissions test case * fix: add permissions test case * fix: refactor (excluding permissions test cases from global beforeEach) --------- Co-authored-by: Ramin <raminramazanpour@gmail.com> Co-authored-by: Meriem-BM <barhoumi.meriem1@gmail.com> Co-authored-by: CarlosQ96 <92376054+CarlosQ96@users.noreply.github.com> Co-authored-by: Amin Latifi <a.latifi.al@gmail.com> Co-authored-by: Carlos <carlos.quintero096@gmail.com>
Giveth · May 21, 2024 · 39c3e59 · 39c3e59
1 parent c42bd24
commit 39c3e59
Show file tree

Hide file tree

Showing 56 changed files with 973 additions and 434 deletions.
diff --git a/.DS_Store b/.DS_Store
diff --git a/migration-old-backup/1714018700116-add_archived_QFRound_fields.ts b/migration-old-backup/1714018700116-add_archived_QFRound_fields.ts
@@ -6,23 +6,15 @@ export class AddArchivedQFRoundFields1714018700116
   public async up(queryRunner: QueryRunner): Promise<void> {
     await queryRunner.query(`
           ALTER TABLE "qf_round" 
-          ADD COLUMN IF NOT EXISTS "bannerBgImage" character varying
-        `);
-
-    await queryRunner.query(`
-          ALTER TABLE "qf_round" 
+          ADD COLUMN IF NOT EXISTS "bannerBgImage" character varying,
           ADD COLUMN IF NOT EXISTS "sponsorsImgs" character varying[] DEFAULT '{}' NOT NULL
         `);
   }
 
   public async down(queryRunner: QueryRunner): Promise<void> {
     await queryRunner.query(`
           ALTER TABLE "qf_round" 
-          DROP COLUMN IF EXISTS "bannerBgImage"
-        `);
-
-    await queryRunner.query(`
-          ALTER TABLE "qf_round" 
+          DROP COLUMN IF EXISTS "bannerBgImage",
           DROP COLUMN IF EXISTS "sponsorsImgs"
         `);
   }

diff --git a/migration/1654415838996-fillRelatedAddressesFromProjectsTable.ts b/migration/1654415838996-fillRelatedAddressesFromProjectsTable.ts
@@ -15,7 +15,7 @@ const insertRelatedAddress = async (params: {
                   "networkId", address, "projectId", "userId", "isRecipient")
                   VALUES (${networkId}, '${project.walletAddress?.toLowerCase()}', ${
                     project.id
-                  }, ${Number(project.admin)}, true);
+                  }, ${project.adminUserId}, true);
               `,
   );
 };

diff --git a/migration/1712853017092-UserNewRoleQfManager.ts b/migration/1712853017092-UserNewRoleQfManager.ts
@@ -4,22 +4,24 @@ export class UserNewRoleQfManager1712853017092 implements MigrationInterface {
   public async up(queryRunner: QueryRunner): Promise<void> {
     // add enum qfManager to user table column role
     await queryRunner.query(
-      `
-      ALTER TYPE user_role_enum ADD VALUE 'qfManager';
-      ALTER TABLE "user" ALTER COLUMN "role" TYPE user_role_enum USING "role"::text::user_role_enum;
-      ALTER TABLE "user" ALTER COLUMN "role" SET DEFAULT 'restricted';
+      `DO $$
+        BEGIN
+            IF NOT EXISTS (
+                SELECT 1
+                FROM pg_enum
+              WHERE pg_enum.enumtypid = 'user_role_enum'::regtype 
+                  AND pg_enum.enumlabel = 'qfManager'
+            ) THEN
+                BEGIN
+                    EXECUTE 'ALTER TYPE user_role_enum ADD VALUE ''qfManager''';
+                END;
+            END IF;
+        END $$;
       `,
     );
   }
 
   public async down(queryRunner: QueryRunner): Promise<void> {
-    // remove enum qfManager from user table column role
-    await queryRunner.query(
-      `
-      ALTER TYPE user_role_enum DROP VALUE 'qfManager';
-      ALTER TABLE "user" ALTER COLUMN "role" TYPE user_role_enum USING "role"::text::user_role_enum;
-      ALTER TABLE "user" ALTER COLUMN "role" SET DEFAULT 'restricted';
-      `,
-    );
+    await queryRunner.query(''); // no need to remove enum value
   }
 }
diff --git a/migration/1715521134568-relate_current_mini_stream_donations_to_qf_Rounds.ts b/migration/1715521134568-relate_current_mini_stream_donations_to_qf_Rounds.ts
@@ -0,0 +1,25 @@
+import { MigrationInterface, QueryRunner } from 'typeorm';
+
+export class RelateCurrentMiniStreamDonationsToQfRounds1715521134568
+  implements MigrationInterface
+{
+  public async up(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(`
+            UPDATE donation
+            SET "qfRoundId" = qf_round.id
+            FROM project_qf_rounds_qf_round, qf_round
+            WHERE donation."recurringDonationId" IS NOT NULL
+                AND donation."projectId" = project_qf_rounds_qf_round."projectId"
+                AND project_qf_rounds_qf_round."qfRoundId" = qf_round."id"
+                AND donation."createdAt" BETWEEN qf_round."beginDate" AND qf_round."endDate"
+        `);
+  }
+
+  public async down(_queryRunner: QueryRunner): Promise<void> {
+    // Since the `up` migration changes data based on existing conditions rather than schema,
+    // rolling back would ideally require prior knowledge of the previous state, which might
+    // not be practical or possible to restore.
+    // Therefore, typically for data migrations, the down method might be left empty or
+    // could reset changes based on specific requirements.
+  }
+}
diff --git a/migration/1715556030126-dropAdminColumn.ts b/migration/1715556030126-dropAdminColumn.ts
@@ -0,0 +1,11 @@
+import { MigrationInterface, QueryRunner } from 'typeorm';
+
+export class DropAdminColumn1715556030126 implements MigrationInterface {
+  public async up(queryRunner: QueryRunner): Promise<void> {
+    await queryRunner.query(`
+            ALTER TABLE "project"
+            DROP COLUMN "admin";
+        `);
+  }
+  public async down(_queryRunner: QueryRunner): Promise<void> {}
+}
diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -41,6 +41,7 @@
     "form-data": "^3.0.1",
     "google-spreadsheet": "^3.2.0",
     "graphql": "16.8.1",
+    "graphql-fields": "^2.0.3",
     "graphql-tag": "^2.12.6",
     "graphql-upload": "15.0.2",
     "handlebars": "4.7.7",
@@ -175,6 +176,7 @@
     "test:fillSnapshotBalance": "NODE_ENV=test mocha ./test/pre-test-scripts.ts  ./src/services/cronJobs/fillSnapshotBalances.test.ts",
     "test:donationService": "NODE_ENV=test mocha ./test/pre-test-scripts.ts ./src/repositories/qfRoundHistoryRepository.test.ts  ./src/services/donationService.test.ts",
     "test:draftDonationService": "NODE_ENV=test mocha -t 99999 ./test/pre-test-scripts.ts  src/services/chains/evm/draftDonationService.test.ts src/repositories/draftDonationRepository.test.ts src/workers/draftDonationMatchWorker.test.ts src/resolvers/draftDonationResolver.test.ts",
+    "test:draftDonationWorker": "NODE_ENV=test mocha -t 99999 ./test/pre-test-scripts.ts src/workers/draftDonationMatchWorker.test.ts",
     "test:draftRecurringDonationService": "NODE_ENV=test mocha -t 99999 ./test/pre-test-scripts.ts  src/services/chains/evm/draftRecurringDonationService.test.ts",
     "test:userService": "NODE_ENV=test mocha ./test/pre-test-scripts.ts  ./src/services/userService.test.ts",
     "test:lostDonations": "NODE_ENV=test mocha ./test/pre-test-scripts.ts  ./src/services/cronJobs/importLostDonationsJob.test.ts",
@@ -201,6 +203,7 @@
     "test:bootstrap": "NODE_ENV=test mocha ./test/pre-test-scripts.ts  ./src/server/bootstrap.test.ts",
     "test:utils": "NODE_ENV=test mocha ./src/utils/utils.test.ts",
     "start": "NODE_ENV=development ts-node-dev --project ./tsconfig.json --respawn ./src/index.ts",
+    "start:test": "NODE_ENV=development ts-node-dev --project ./tsconfig.json --respawn ./test.ts",
     "serve": "pm2 startOrRestart ecosystem.config.js --node-args='--max-old-space-size=8192'",
     "db:migrate:run:test": "NODE_ENV=test npx typeorm-ts-node-commonjs migration:run -d ./src/ormconfig.ts",
     "db:migrate:revert:test": "NODE_ENV=test npx typeorm-ts-node-commonjs migration:revert -d ./src/ormconfig.ts",

diff --git a/src/adapters/notifications/NotificationCenterAdapter.ts b/src/adapters/notifications/NotificationCenterAdapter.ts
@@ -980,7 +980,6 @@ const getEmailDataDonationAttributes = async (params: {
     title: project.title,
     firstName: user.firstName,
     userId: user.id,
-    projectOwnerId: project.admin,
     slug: project.slug,
     projectLink: `${process.env.WEBSITE_URL}/project/${project.slug}`,
     amount,

diff --git a/src/entities/project.ts b/src/entities/project.ts
@@ -151,10 +151,6 @@ export class Project extends BaseEntity {
   @Column('text', { array: true, default: '{}' })
   slugHistory?: string[];
 
-  @Field({ nullable: true })
-  @Column({ nullable: true })
-  admin?: string;
-
   @Field({ nullable: true })
   @Column({ nullable: true })
   description?: string;
@@ -269,11 +265,6 @@ export class Project extends BaseEntity {
   @Column('jsonb', { nullable: true })
   contacts: ProjectContacts[];
 
-  @ManyToMany(_type => User, user => user.projects)
-  @Field(_type => [User], { nullable: true })
-  @JoinTable()
-  users: User[];
-
   @Field(() => [Reaction], { nullable: true })
   @OneToMany(_type => Reaction, reaction => reaction.project)
   reactions?: Reaction[];
@@ -318,6 +309,7 @@ export class Project extends BaseEntity {
   adminUser: User;
 
   @Column({ nullable: true })
+  @Field(_type => Int)
   @RelationId((project: Project) => project.adminUser)
   adminUserId: number;
 
@@ -582,10 +574,6 @@ export class Project extends BaseEntity {
     }
   }
 
-  owner() {
-    return this.users[0];
-  }
-
   @BeforeUpdate()
   async updateProjectDescriptionSummary() {
     await Project.update(

diff --git a/src/entities/projectDonationSummaryView.ts b/src/entities/projectDonationSummaryView.ts
@@ -0,0 +1,25 @@
+import { Field, ObjectType } from 'type-graphql';
+import {
+  Column,
+  PrimaryColumn,
+  BaseEntity,
+  ViewEntity,
+  ViewColumn,
+} from 'typeorm';
+
+@ViewEntity('project_donation_summary_view', { synchronize: false })
+@ObjectType()
+export class ProjectDonationSummaryView extends BaseEntity {
+  @Field()
+  @ViewColumn()
+  @PrimaryColumn()
+  projectId: number;
+
+  @ViewColumn()
+  @Column('double precision')
+  sumVerifiedDonations: number;
+
+  @ViewColumn()
+  @Column('int')
+  uniqueDonorsCount: number;
+}
diff --git a/src/entities/user.ts b/src/entities/user.ts
@@ -5,14 +5,12 @@ import {
   CreateDateColumn,
   Entity,
   Index,
-  JoinTable,
-  ManyToMany,
   OneToMany,
   OneToOne,
   PrimaryGeneratedColumn,
   UpdateDateColumn,
 } from 'typeorm';
-import { Project, ProjStatus, ReviewStatus } from './project';
+import { ProjStatus, ReviewStatus } from './project';
 import { Donation, DONATION_STATUS } from './donation';
 import { Reaction } from './reaction';
 import { AccountVerification } from './accountVerification';
@@ -161,11 +159,6 @@ export class User extends BaseEntity {
   })
   referredEvent?: ReferredEvent;
 
-  @Field(_type => [Project])
-  @ManyToMany(_type => Project, project => project.users)
-  @JoinTable()
-  projects?: Project[];
-
   @Column('bool', { default: false })
   segmentIdentified: boolean;
 
@@ -216,6 +209,10 @@ export class User extends BaseEntity {
         status: DONATION_STATUS.VERIFIED,
       })
       .andWhere(`donation."recurringDonationId" IS NULL`)
+      .cache(
+        `user-donationsCount-normal-${this.id}`,
+        Number(process.env.USER_STATS_CACHE_TIME || 60000),
+      )
       .getCount();
 
     // Count for recurring donations
@@ -224,6 +221,10 @@ export class User extends BaseEntity {
     )
       .where(`recurring_donation."donorId" = :donorId`, { donorId: this.id })
       .andWhere('recurring_donation.totalUsdStreamed > 0')
+      .cache(
+        `user-donationsCount-recurring-${this.id}`,
+        Number(process.env.USER_STATS_CACHE_TIME || 60000),
+      )
       .getCount();
 
     // Sum of both counts
@@ -239,6 +240,10 @@ export class User extends BaseEntity {
         `project.statusId = ${ProjStatus.active} AND project.reviewStatus = :reviewStatus`,
         { reviewStatus: ReviewStatus.Listed },
       )
+      .cache(
+        `user-likedProjectsCount-recurring-${this.id}`,
+        Number(process.env.USER_STATS_CACHE_TIME || 60000),
+      )
       .getCount();
 
     return likedProjectsCount;

diff --git a/src/repositories/powerBoostingRepository.ts b/src/repositories/powerBoostingRepository.ts
@@ -165,7 +165,11 @@ export const findPowerBoostingsCountByUserId = async (
     .leftJoin('powerBoosting.user', 'user')
     .addSelect(publicSelectionFields)
     .where(`percentage > 0`)
-    .andWhere(`"userId" =${userId}`);
+    .andWhere(`"userId" =${userId}`)
+    .cache(
+      `findPowerBoostingsCountByUserId-recurring-${userId}`,
+      Number(process.env.USER_STATS_CACHE_TIME || 60000),
+    );
   return query.getCount();
 };
 

diff --git a/src/repositories/previousRoundRankRepository.test.ts b/src/repositories/previousRoundRankRepository.test.ts
@@ -13,6 +13,7 @@ import {
   saveProjectDirectlyToDb,
   saveUserDirectlyToDb,
   SEED_DATA,
+  dbIndependentTests,
 } from '../../test/testUtils';
 import {
   insertSinglePowerBoosting,
@@ -44,12 +45,19 @@ describe(
   projectsThatTheirRanksHaveChangedTestCases,
 );
 
-beforeEach(async () => {
-  await AppDataSource.getDataSource().query('truncate power_snapshot cascade');
+beforeEach(async function () {
+  const { title } = this.currentTest?.parent || {};
+
+  if (title && dbIndependentTests.includes(title)) {
+    return;
+  }
+
+  await AppDataSource.getDataSource().query('TRUNCATE power_snapshot CASCADE');
   await PowerBalanceSnapshot.clear();
   await PowerBoostingSnapshot.clear();
   await PreviousRoundRank.clear();
   await PowerRound.clear();
+
   await createSomeSampleProjectsAndPowerViews();
 });
 

diff --git a/src/repositories/projectAddressRepository.test.ts b/src/repositories/projectAddressRepository.test.ts
@@ -142,7 +142,7 @@ function addNewProjectAddressTestCases() {
       ...createProjectData(),
       walletAddress,
       verified: true,
-      admin: String(user.id),
+      adminUserId: user.id,
     });
     const newAddress = generateRandomEtheriumAddress();
     const newRelatedAddress = await addNewProjectAddress({
@@ -199,7 +199,7 @@ function addBulkNewProjectAddressTestCases() {
       ...createProjectData(),
       walletAddress,
       verified: true,
-      admin: String(user.id),
+      adminUserId: user.id,
     });
     const newAddress = generateRandomEtheriumAddress();
     await addBulkNewProjectAddress([
@@ -224,7 +224,7 @@ function addBulkNewProjectAddressTestCases() {
       ...createProjectData(),
       walletAddress,
       verified: true,
-      admin: String(user.id),
+      adminUserId: user.id,
     });
     const newAddress1 = generateRandomEtheriumAddress();
     const newAddress2 = generateRandomEtheriumAddress();