feat: truncate parsed uploads to prevent database and frontend blocking caused by excessively large files #3914
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…MB to 100MB to allow larger file uploads
…t message to display the correct file size limit of 100 bytes instead of 10 bytes
…n VertexBuildResponse class 📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries
… instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long
…transformations for performance optimization
🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object
…s for better clarity and consistency 📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation 📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models
… class to use a new helper function truncate_long_strings for better code readability and maintainability
dosubot
bot
added
size:L
github-actions
bot
added
enhancement
…te module to improve code organization and reusability 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error
dosubot
bot
added
size:M
…te long strings in dictionaries and lists to prevent exceeding the maximum text length.
dosubot
bot
added
size:L
…efining maximum text length allowed in the application
…o match new location in util module
ogabrielluiz
requested changes
Sep 25, 2024
...end/src/CustomNodes/GenericNode/components/outputModal/components/switchOutputView/index.tsx
Outdated
Show resolved
Hide resolved
...end/src/CustomNodes/GenericNode/components/outputModal/components/switchOutputView/index.tsx
Outdated
Show resolved
Hide resolved
…function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures 🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation.
…truncate_long_strings function 📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function
♻️ (AssemblyAIFormatTranscript.py): Change typing annotations to lowercase for consistency ♻️ (AssemblyAIListTranscripts.py): Change typing annotations to lowercase for consistency ♻️ (LangChainHubPrompt.py): Remove duplicate import statement to improve code readability ♻️ (model.py): Remove unnecessary blank lines to enhance code readability ♻️ (model.py): Remove unnecessary blank lines to enhance code readability ♻️ (constants.py): Update MAX_TEXT_LENGTH constant to have consistent spacing ♻️ (util.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string quotes to be consistent throughout the file ♻️ (test_truncate_long_strings_on_objects.py): Update string
…ettings function to allow configuring maximum file size for uploads
… size upload from utility store to improve code modularity and reusability 🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails
…ger used and update MAX_TEXT_LENGTH constant to a higher value
…value from config response
…ileSizeUpload function to handle maximum file size upload in bytes
…Upload method to handle maximum file size upload functionality in the UtilityStoreType
…ove code readability and consistency by removing unnecessary whitespace and aligning assignment operators.
…code readability and maintain consistency
…flow into cz/limitCsvView
github-actions
bot
added
enhancement
…e is larger than the maximum allowed size in MB
…pload state 🐛 (chatView/index.tsx): add validation to check if file size exceeds maxFileSizeUpload limit before uploading 🐛 (chatView/index.tsx): handle error response when uploading file and display error message
…yStore to check file size before uploading to prevent exceeding the maximum allowed file size 📝 (chatInput/index.tsx): add error handling for file size exceeding the maximum allowed size and display an error alert with the appropriate message
…ng file size validation errors 🔧 (FileInput/index.tsx): Update file input component to use global alert store and utility store for error handling and max file size configuration
…load a file larger than the specified limit of 0.001MB. This test includes mocking API response, checking for required environment variables, interacting with the page elements, and validating error message display.
…found issue in test case
…flow into cz/limitCsvView
diogocabral
pushed a commit
to headlinevc/langflow
that referenced
this pull request
Nov 26, 2024
…ng caused by excessively large files (langflow-ai#3914) * 📝 (constants.ts): increase maxSizeFilesInBytes constant value from 10MB to 100MB to allow larger file uploads * 🐛 (inputFileComponent): fix bug in setting the maximum file size alert message to display the correct file size limit of 100 bytes instead of 10 bytes * 📝 (schemas.py): Add a new field_serializer method to serialize data in VertexBuildResponse class 📝 (schemas.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize outputs in TransactionBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries 📝 (model.py): Add a new field_serializer method to serialize data and artifacts in VertexBuildBase class 📝 (model.py): Add a new truncate_text helper function to safely truncate text in nested dictionaries * 🐛 (schemas.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (model.py): fix truncation length of text fields to 10 characters instead of 99999 🐛 (index.tsx): truncate resultMessage to 99999 characters and add message if text is too long * 🔧 (switchOutputView/index.tsx): Use useMemo to memoize resultMessage transformations for performance optimization * 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (model.py): Fix typo in the path for 'base_retriever' data field 🐛 (index.tsx): Fix logic to correctly handle resultMessageMemoized when it is an object * 📝 (model.py): refactor truncate_text function to truncate_long_strings for better clarity and consistency 📝 (model.py): update serialize_outputs and serialize_artifacts functions to use truncate_long_strings for string truncation 📝 (model.py): introduce MAX_TEXT_LENGTH constant for defining the maximum length of text to truncate in the models * 📝 (schemas.py): refactor serialize_data method in VertexBuildResponse class to use a new helper function truncate_long_strings for better code readability and maintainability * 🔧 (schemas.py): Move the `truncate_long_strings` function to a separate module to improve code organization and reusability 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error 🔧 (model.py): Import the `truncate_long_strings` function from the correct module to fix the reference error * 📝 (util.py): add function truncate_long_strings to recursively truncate long strings in dictionaries and lists to prevent exceeding the maximum text length. * 📝 (constants.py): add constant MAX_TEXT_LENGTH with value 99999 for defining maximum text length allowed in the application * 📝 (model.py): update import path for truncate_long_strings function to match new location in util module * ✨ (test_truncate_long_strings_on_objects.py): Add unit tests for the function truncate_long_strings to ensure correct behavior when truncating long strings in various data structures 🐛 (switchOutputView/index.tsx): Fix truncation logic to correctly truncate long strings by adding ellipsis at the end instead of displaying additional text about truncation. * [autofix.ci] apply automated fixes * ✨ (test_truncate_long_strings_on_objects.py): Update import path for truncate_long_strings function 📝 (test_truncate_long_strings_on_objects.py): Add additional tests for handling negative, zero, and small max_length values in truncate_long_strings function * ♻️ (schemas.py): refactor import statement to use the updated module name util_strings instead of util for better clarity and consistency. * 📝 (model.py): Update import path for util_strings module to fix module import error 📝 (util.py): Remove redundant code for truncating long strings and move it to a separate util_strings module for better organization and separation of concerns. * 📝 (schemas.py): refactor serialize_data method to handle both BaseModel and non-BaseModel data inputs in VertexBuildResponse class * 📝 (util_strings.py): Update util_strings.py to improve string truncation function for dictionaries and lists 🔧 (test_truncate_long_strings_on_objects.py): Update test cases for string truncation function to cover additional scenarios and edge cases * Update src/backend/base/langflow/utils/util_strings.py Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org> * 📝 (vite.config.mts): update environment variable MAX_FILE_SIZE to be defined in vite config for frontend to use in the application. * 📝 (constants.ts): update maxSizeFilesInBytes constant to use process.env.MAX_FILE_SIZE environment variable for configurable file size limit 📝 (constants.ts): add MAX_TEXT_LENGTH constant with a value of 99999 for maximum text length limit * 📝 (switchOutputView/index.tsx): import MAX_TEXT_LENGTH constant from shared constants file to improve code organization and reusability * ✨ (langflow/__main__.py): add support for defining maximum file size for upload in MB to improve user experience and prevent large file uploads * 🐛 (files.py): add validation to check if uploaded file size exceeds the maximum allowed size before processing it * ✨ (schemas.py): add max_file_size_upload field to ConfigResponse schema to handle maximum file size allowed for upload * 🔧 (vite.config.mts): remove MAX_FILE_SIZE environment variable configuration as it is no longer needed * ✨ (base.py): introduce max_file_size_upload setting to limit the file size for uploads in MB * 🐛 (util.py): add support for setting max_file_size_upload in update_settings function to allow configuring maximum file size for uploads * 📝 (inputFileComponent/index.tsx): add support for retrieving max file size upload from utility store to improve code modularity and reusability 🐛 (inputFileComponent/index.tsx): fix error handling logic to display error message when uploading a file fails * 📝 (constants.ts): remove maxSizeFilesInBytes constant as it is no longer used and update MAX_TEXT_LENGTH constant to a higher value * ✨ (use-get-config.ts): add functionality to set max file size upload value from config response * ✨ (utilityStore.ts): introduce maxFileSizeUpload property and setMaxFileSizeUpload function to handle maximum file size upload in bytes * ✨ (frontend): introduce maxFileSizeUpload property and setMaxFileSizeUpload method to handle maximum file size upload functionality in the UtilityStoreType * ♻️ (util_strings.py): refactor truncate_long_strings function to improve code readability and consistency by removing unnecessary whitespace and aligning assignment operators. * 🐛 (files.py): fix formatting issue in the raise statement to improve code readability and maintain consistency --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@langflow.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a feature to limit the size of parsed files during uploads. The primary goal is to prevent the database and the front end from being blocked by excessively large files, which can cause performance issues and degrade user experience.
Implemented a recursive function to analyze schemas for fields containing excessive character lengths and truncate them accordingly.
Introduced a frontend and backend safeguard to truncate excessively long values, preventing potential crashes in the user interface.
Added a new .env variable LANGFLOW_MAX_FILE_SIZE_UPLOAD with a default value of 100 MB. This allows users to customize the maximum file upload size according to their needs.