Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: upgrading django-storages to 1.10.1 #32571

Merged
merged 11 commits into from
Aug 28, 2023
Merged

feat!: upgrading django-storages to 1.10.1 #32571

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3845490
feat!: upgrading django-storages to 1.10.1
awais786 Jun 26, 2023
63eaa2e
chore: Adding test to verify s3boto3 storages with invalid params.
awais786 Aug 18, 2023
42e1766
chore: Adding test to verify s3boto3 storages with invalid params.
awais786 Aug 18, 2023
b0c1368
chore: Adding test to verify s3boto3 storages with invalid params.
awais786 Aug 18, 2023
0ef4e8f
Merge branch 'master' into django-storages-upgrade
awais786 Aug 21, 2023
81a89b4
chore: Update test_video_utils.py
awais786 Aug 21, 2023
74afbf0
chore: fixing test.
awais786 Aug 21, 2023
fe4c54e
Merge branch 'master' into django-storages-upgrade
awais786 Aug 22, 2023
1b6cc9b
chore: adding test to verify default-acl behaviour.
awais786 Aug 22, 2023
7db65b5
Merge branch 'master' into django-storages-upgrade
awais786 Aug 24, 2023
b089c69
Merge branch 'master' into django-storages-upgrade
awais786 Aug 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions cms/djangoapps/contentstore/tests/test_video_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ def test_video_backend(self):
@override_settings(VIDEO_IMAGE_SETTINGS={
'STORAGE_CLASS': 'storages.backends.s3boto3.S3Boto3Storage',
'STORAGE_KWARGS':
{'bucket_name': 'test', 'default_acl': None, 'base_url': '/', 'location': 'abc/def'}}
{'bucket_name': 'test', 'default_acl': None, 'location': 'abc/def'}}
Copy link
Contributor Author

@awais786 awais786 Aug 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructor now verifies its passing params and base_url is unknown for this package.

)
def test_boto3_backend_with_params(self):
storage = get_storage_class(
Expand Down Expand Up @@ -417,7 +417,6 @@ def test_storage_without_global_default_acl_setting(self):
obj.upload_fileobj.assert_called_with(
content,
ExtraArgs={
'ACL': 'public-read', # it will come from 1.9.1
'ContentType': 'text/plain',
}
)
Expand Down
4 changes: 2 additions & 2 deletions cms/envs/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2497,9 +2497,9 @@
# STORAGE_KWARGS=dict(bucket='video-image-bucket'),
STORAGE_KWARGS=dict(
location=MEDIA_ROOT,
base_url=MEDIA_URL,
),
DIRECTORY_PREFIX='video-images/',
BASE_URL=MEDIA_URL,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These values override for sandbox and stage/prod and there base_url does not exists. Its for devstack.

Copy link
Contributor Author

@awais786 awais786 Aug 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are default settings allowed params as per base.py

{'access_key': '**', 'secret_key': '**', 'file_overwrite': True, 'object_parameters': {},
 'bucket_name': 'abc', 'querystring_auth': True, 'querystring_expire': 3600, 'signature_version': None, 
'location': '', 'custom_domain': None, 'cloudfront_signer': None, 'addressing_style': None, 'secure_urls': True, 'file_name_charset': 'utf-8', 'gzip': False, 
'gzip_content_types': ('text/css', 'text/javascript', 'application/javascript', 'application/x-javascript', 'image/svg+xml'), 'url_protocol': 'http:', 'endpoint_url': None, 'proxies': None, 
'region_name': None, 'use_ssl': True, 'verify': None, 'max_memory_size': 0, 'default_acl': None}

If any value pass to constructor it verifies all those params against above params and throws error if its not exists in defaults.

https://github.com/jschneier/django-storages/blob/master/CHANGELOG.rst#general-5

)

VIDEO_IMAGE_MAX_AGE = 31536000
Expand All @@ -2512,9 +2512,9 @@
# STORAGE_KWARGS=dict(bucket='video-transcripts-bucket'),
STORAGE_KWARGS=dict(
location=MEDIA_ROOT,
base_url=MEDIA_URL,
),
DIRECTORY_PREFIX='video-transcripts/',
BASE_URL=MEDIA_URL,
)

VIDEO_TRANSCRIPTS_MAX_AGE = 31536000
Expand Down
6 changes: 3 additions & 3 deletions cms/envs/devstack-experimental.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,17 +512,17 @@ VIDEO_IMAGE_MAX_AGE: 31536000
VIDEO_IMAGE_SETTINGS:
DIRECTORY_PREFIX: video-images/
STORAGE_KWARGS:
base_url: /media/
location: /edx/var/edxapp/media//
VIDEO_IMAGE_MAX_BYTES: 2097152
VIDEO_IMAGE_MIN_BYTES: 2048
BASE_URL: /media/
VIDEO_TRANSCRIPTS_MAX_AGE: 31536000
VIDEO_TRANSCRIPTS_SETTINGS:
DIRECTORY_PREFIX: video-transcripts/
STORAGE_KWARGS:
base_url: /media/
location: /edx/var/edxapp/media//
location: edx/var/edxapp/media//
VIDEO_TRANSCRIPTS_MAX_BYTES: 3145728
BASE_URL: /media/
VIDEO_UPLOAD_PIPELINE:
BUCKET: ''
ROOT_PATH: ''
Expand Down
2 changes: 1 addition & 1 deletion cms/envs/test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,9 +303,9 @@
VIDEO_IMAGE_MIN_BYTES=2 * 1024, # 2 KB
STORAGE_KWARGS=dict(
location=MEDIA_ROOT,
base_url=MEDIA_URL,
),
DIRECTORY_PREFIX='video-images/',
BASE_URL=MEDIA_URL,
)
VIDEO_IMAGE_DEFAULT_FILENAME = 'default_video_image.png'

Expand Down
4 changes: 2 additions & 2 deletions cms/urls.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,12 +277,12 @@
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

urlpatterns += static(
settings.VIDEO_IMAGE_SETTINGS['STORAGE_KWARGS']['base_url'],
settings.VIDEO_IMAGE_SETTINGS['BASE_URL'],
document_root=settings.VIDEO_IMAGE_SETTINGS['STORAGE_KWARGS']['location']
)

urlpatterns += static(
settings.VIDEO_TRANSCRIPTS_SETTINGS['STORAGE_KWARGS']['base_url'],
settings.VIDEO_TRANSCRIPTS_SETTINGS['BASE_URL'],
document_root=settings.VIDEO_TRANSCRIPTS_SETTINGS['STORAGE_KWARGS']['location']
)

Expand Down
5 changes: 3 additions & 2 deletions lms/envs/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3801,9 +3801,10 @@ def _make_locale_paths(settings): # pylint: disable=missing-function-docstring
# STORAGE_KWARGS=dict(bucket='video-image-bucket'),
STORAGE_KWARGS=dict(
location=MEDIA_ROOT,
base_url=MEDIA_URL,
),
DIRECTORY_PREFIX='video-images/',
BASE_URL=MEDIA_URL,

)

VIDEO_IMAGE_MAX_AGE = 31536000
Expand All @@ -3817,9 +3818,9 @@ def _make_locale_paths(settings): # pylint: disable=missing-function-docstring
# STORAGE_KWARGS=dict(bucket='video-transcripts-bucket'),
STORAGE_KWARGS=dict(
location=MEDIA_ROOT,
base_url=MEDIA_URL,
),
DIRECTORY_PREFIX='video-transcripts/',
BASE_URL=MEDIA_URL,
)

VIDEO_TRANSCRIPTS_MAX_AGE = 31536000
Expand Down
8 changes: 4 additions & 4 deletions lms/envs/devstack-experimental.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -614,17 +614,17 @@ VIDEO_IMAGE_MAX_AGE: 31536000
VIDEO_IMAGE_SETTINGS:
DIRECTORY_PREFIX: video-images/
STORAGE_KWARGS:
base_url: /media/
location: /edx/var/edxapp/media//
location: edx/var/edxapp/media//
VIDEO_IMAGE_MAX_BYTES: 2097152
VIDEO_IMAGE_MIN_BYTES: 2048
BASE_URL: /media/
VIDEO_TRANSCRIPTS_MAX_AGE: 31536000
VIDEO_TRANSCRIPTS_SETTINGS:
DIRECTORY_PREFIX: video-transcripts/
STORAGE_KWARGS:
base_url: /media/
location: /edx/var/edxapp/media//
location: edx/var/edxapp/media//
VIDEO_TRANSCRIPTS_MAX_BYTES: 3145728
BASE_URL: /media/
VIDEO_UPLOAD_PIPELINE:
BUCKET: ''
ROOT_PATH: ''
Expand Down
5 changes: 2 additions & 3 deletions requirements/constraints.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@ celery>=5.2.2,<6.0.0
# required for celery>=5.2.0;<5.3.0
click>=8.0,<9.0

# django-storages version 1.9 drops support for boto storage backend.
# 1.9 gives an error for details https://github.com/jschneier/django-storages/issues/831
django-storages==1.9.1
# django-storages version 1.10.1 is major upgrade and require multiple settings changes.
django-storages==1.10.1


# The team that owns this package will manually bump this package rather than having it pulled in automatically.
Expand Down
2 changes: 1 addition & 1 deletion requirements/edx/base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ django-statici18n==2.4.0
# -r requirements/edx/kernel.in
# lti-consumer-xblock
# xblock-drag-and-drop-v2
django-storages==1.9.1
django-storages==1.10.1
# via
# -c requirements/edx/../constraints.txt
# -r requirements/edx/kernel.in
Expand Down
2 changes: 1 addition & 1 deletion requirements/edx/development.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -567,7 +567,7 @@ django-statici18n==2.4.0
# -r requirements/edx/testing.txt
# lti-consumer-xblock
# xblock-drag-and-drop-v2
django-storages==1.9.1
django-storages==1.10.1
# via
# -c requirements/edx/../constraints.txt
# -r requirements/edx/doc.txt
Expand Down
2 changes: 1 addition & 1 deletion requirements/edx/doc.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,7 +417,7 @@ django-statici18n==2.4.0
# -r requirements/edx/base.txt
# lti-consumer-xblock
# xblock-drag-and-drop-v2
django-storages==1.9.1
django-storages==1.10.1
# via
# -c requirements/edx/../constraints.txt
# -r requirements/edx/base.txt
Expand Down
2 changes: 1 addition & 1 deletion requirements/edx/testing.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ django-statici18n==2.4.0
# -r requirements/edx/base.txt
# lti-consumer-xblock
# xblock-drag-and-drop-v2
django-storages==1.9.1
django-storages==1.10.1
# via
# -c requirements/edx/../constraints.txt
# -r requirements/edx/base.txt
Expand Down