-
-
Notifications
You must be signed in to change notification settings - Fork 503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Perf: refactor DistributionsByCountry#report using raw SQL query #4841
Open
coalest
wants to merge
7
commits into
rubyforgood:main
Choose a base branch
from
coalest:improve-distributions-by-country-report-performance
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+170
−104
Open
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
f99e5d1
Perf: refactor DistributionsByCountry#report using raw SQL query
coalest 245a805
Fix query spec
coalest c86f218
Refactor after review
coalest 257cf8f
Fix flaky query spec
coalest 2c1124d
Merge branch 'main' into improve-distributions-by-country-report-perf…
coalest 6b8d7c1
Fix join clause in SQL query
coalest f14d05d
Change query spec to be closer to original spec
coalest File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
class DistributionSummaryByCountyQuery | ||
DISTRIBUTION_BY_COUNTY_SQL = <<~SQL.squish.freeze | ||
/* Calculate total item quantity and value per distribution */ | ||
WITH distribution_totals AS | ||
( | ||
SELECT DISTINCT d.id, | ||
d.partner_id, | ||
COALESCE(SUM(li.quantity) OVER (PARTITION BY d.id), 0) AS quantity, | ||
COALESCE(SUM(COALESCE(i.value_in_cents, 0) * li.quantity) OVER (PARTITION BY d.id), 0) AS amount | ||
FROM distributions d | ||
JOIN line_items li ON li.itemizable_id = d.id AND li.itemizable_type = 'Distribution' | ||
JOIN items i ON i.id = li.item_id | ||
WHERE d.issued_at BETWEEN :start_date AND :end_date | ||
AND d.organization_id = :organization_id | ||
GROUP BY d.id, li.id, i.id | ||
), | ||
/* Match distribution totals with client share and counties. | ||
If distribution has no associated county, set county name to "Unspecified" | ||
and set region to ZZZ so it will be last when sorted */ | ||
totals_by_county AS | ||
( | ||
SELECT dt.id, | ||
dt.quantity, | ||
dt.amount, | ||
COALESCE(psa.client_share::float / 100, 1) AS percentage, | ||
COALESCE(c.name, 'Unspecified') county_name, | ||
COALESCE(c.region, 'ZZZ') county_region | ||
FROM distribution_totals dt | ||
LEFT JOIN partner_profiles pp ON pp.id = dt.partner_id | ||
LEFT JOIN partner_served_areas psa ON psa.partner_profile_id = pp.partner_id | ||
LEFT JOIN counties c ON c.id = psa.county_id | ||
UNION | ||
/* Previous behavior was to add a row for unspecified counties | ||
even if all distributions have an associated county */ | ||
SELECT 0 AS id, | ||
0 AS quantity, | ||
0 AS amount, | ||
1 AS percentage, | ||
'Unspecified' AS county_name, | ||
'ZZZ' AS county_region | ||
) | ||
/* Distribution value and quantities per county share may not be whole numbers, | ||
so we cast to an integer for rounding purposes */ | ||
SELECT tbc.county_name AS name, | ||
SUM((tbc.quantity * percentage)::int) AS quantity, | ||
SUM((tbc.amount * percentage)::int) AS amount | ||
FROM totals_by_county tbc | ||
GROUP BY county_name, county_region | ||
ORDER BY county_region ASC; | ||
SQL | ||
|
||
def initialize(organization_id:, start_date: nil, end_date: nil) | ||
@organization_id = organization_id | ||
@start_date = start_date || "1000-01-01" | ||
@end_date = end_date || "3000-01-01" | ||
end | ||
|
||
def call | ||
execute(to_sql(DISTRIBUTION_BY_COUNTY_SQL)).to_a | ||
end | ||
|
||
private | ||
|
||
def execute(sql) | ||
ActiveRecord::Base.connection.execute(sql) | ||
end | ||
|
||
def to_sql(query) | ||
ActiveRecord::Base.sanitize_sql_array( | ||
[ | ||
query, | ||
organization_id: @organization_id, | ||
start_date: @start_date, | ||
end_date: @end_date | ||
] | ||
) | ||
end | ||
end |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
RSpec.describe DistributionSummaryByCountyQuery do | ||
let(:organization) { create(:organization, name: "Some Unique Name") } | ||
let(:item_1) { create(:item, value_in_cents: 1050, organization: organization) } | ||
let(:partner_1) do | ||
create(:partner, organization:, without_profile: true) do |p| | ||
p.profile = create(:partner_profile, partner: p, organization:) do |pp| | ||
pp.served_areas = create_list(:partners_served_area, 4, partner_profile: pp, client_share: 25) do |sa| | ||
sa.county = create(:county) | ||
end | ||
end | ||
end | ||
end | ||
let(:partner_2) do | ||
create(:partner, organization:, without_profile: true) do |p| | ||
p.profile = create(:partner_profile, partner: p, organization:) do |pp| | ||
pp.served_areas = create_list(:partners_served_area, 5, partner_profile: pp, client_share: 20) do |sa, i| | ||
# create one overlapping service area | ||
sa.county = i.zero? ? partner_1.profile.served_areas[0].county : create(:county) | ||
end | ||
end | ||
end | ||
end | ||
|
||
let(:now) { Time.current.to_datetime } | ||
|
||
let(:params) { {organization_id: organization.id, start_date: nil, end_date: nil} } | ||
|
||
describe "call" do | ||
it "will have 100% unspecified shows if no served_areas" do | ||
create(:distribution, :with_items, item: item_1, organization: organization) | ||
breakdown = DistributionSummaryByCountyQuery.new(**params).call | ||
expect(breakdown.size).to eq(1) | ||
expect(breakdown[0]["quantity"]).to eq(100) | ||
expect(breakdown[0]["amount"]).to be_within(0.01).of(105000.0) | ||
end | ||
|
||
it "divides the item numbers and values according to the partner profile" do | ||
create(:distribution, :with_items, item: item_1, organization: organization, partner: partner_1) | ||
breakdown = DistributionSummaryByCountyQuery.new(**params).call | ||
expect(breakdown.size).to eq(5) | ||
expect(breakdown[4]["quantity"]).to eq(0) | ||
expect(breakdown[4]["amount"]).to be_within(0.01).of(0) | ||
3.times do |i| | ||
expect(breakdown[i]["quantity"]).to eq(25) | ||
expect(breakdown[i]["amount"]).to be_within(0.01).of(26250.0) | ||
end | ||
end | ||
|
||
it "handles multiple partners with overlapping service areas properly" do | ||
create(:distribution, :with_items, item: item_1, organization: organization, partner: partner_1, issued_at: now) | ||
create(:distribution, :with_items, item: item_1, organization: organization, partner: partner_2, issued_at: now) | ||
breakdown = DistributionSummaryByCountyQuery.new(**params).call | ||
num_with_45 = 0 | ||
num_with_20 = 0 | ||
num_with_0 = 0 | ||
# The result will have at least 1 45 and at least 1 20, and 1 0. Anything else will be either 45 or 25 or 20 | ||
breakdown.each do |sa| | ||
if sa["quantity"] == 45 | ||
expect(sa["amount"]).to be_within(0.01).of(47250.0) | ||
num_with_45 += 1 | ||
end | ||
|
||
if sa["quantity"] == 25 | ||
expect(sa["amount"]).to be_within(0.01).of(26250.0) | ||
end | ||
if sa["quantity"] == 20 | ||
expect(sa["amount"]).to be_within(0.01).of(21000.0) | ||
num_with_20 += 1 | ||
end | ||
if sa["quantity"] == 0 | ||
expect(sa["amount"]).to be_within(0.01).of(0) | ||
end | ||
end | ||
expect(num_with_45).to be > 0 | ||
expect(num_with_20).to be > 0 | ||
expect(num_with_0).to eq 0 | ||
end | ||
end | ||
end |
63 changes: 0 additions & 63 deletions
63
spec/services/distributions_by_county_report_service_spec.rb
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason this needs to be an instance instead of just having a class function? It's easier to test, stub and work with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also can we return a data class instead of a hash?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No real reason, current queries in this folder vary (sometimes
call
is an instance method sometimes a class method).Changed it to be a class method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.