This commit focuses on optimizing the utility modules in the codebase… #775
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## [1.27.0](ScrapeGraphAI/Scrapegraph-ai@v1.26.7...v1.27.0) (2024-10-26) ### Features * add conditional node structure to the smart_scraper_graph and implemented a structured way to check condition ([cacd9cd](ScrapeGraphAI@cacd9cd)) * add integration with scrape.do ([ae275ec](ScrapeGraphAI@ae275ec)) * add model integration gpt4 ([51c55eb](ScrapeGraphAI@51c55eb)) * implement ScrapeGraph class for only web scraping automation ([612c644](ScrapeGraphAI@612c644)) * Implement SmartScraperMultiParseMergeFirstGraph class that scrapes a list of URLs and merge the content first and finally generates answers to a given prompt. ([3e3e1b2](ScrapeGraphAI@3e3e1b2)) * refactoring of export functions ([0ea00c0](ScrapeGraphAI@0ea00c0)) * refactoring of get_probable_tags node ([f658092](ScrapeGraphAI@f658092)) * refactoring of ScrapeGraph to SmartScraperLiteGraph ([52b6bf5](ScrapeGraphAI@52b6bf5)) ### Bug Fixes * fix export function ([c8a000f](ScrapeGraphAI@c8a000f)) * fix the example variable name ([69ff649](ScrapeGraphAI@69ff649)) * remove variable "max_result" not being used in the code ([e76a68a](ScrapeGraphAI@e76a68a)) ### chore * fix example ([9cd9a87](ScrapeGraphAI@9cd9a87)) ### Test * Add scrape_graph test ([cdb3c11](ScrapeGraphAI@cdb3c11)) * Add smart_scraper_multi_parse_merge_first_graph test ([464b8b0](ScrapeGraphAI@464b8b0)) ### CI * **release:** 1.26.6-beta.1 [skip ci] ([e0fc457](ScrapeGraphAI@e0fc457)) * **release:** 1.27.0-beta.1 [skip ci] ([9266a36](ScrapeGraphAI@9266a36)) * **release:** 1.27.0-beta.10 [skip ci] ([eee131e](ScrapeGraphAI@eee131e)) * **release:** 1.27.0-beta.2 [skip ci] ([d84d295](ScrapeGraphAI@d84d295)) * **release:** 1.27.0-beta.3 [skip ci] ([f576afa](ScrapeGraphAI@f576afa)) * **release:** 1.27.0-beta.4 [skip ci] ([3d6bbcd](ScrapeGraphAI@3d6bbcd)) * **release:** 1.27.0-beta.5 [skip ci] ([5002c71](ScrapeGraphAI@5002c71)) * **release:** 1.27.0-beta.6 [skip ci] ([94b9836](ScrapeGraphAI@94b9836)) * **release:** 1.27.0-beta.7 [skip ci] ([407f1ce](ScrapeGraphAI@407f1ce)) * **release:** 1.27.0-beta.8 [skip ci] ([4f1ed93](ScrapeGraphAI@4f1ed93)) * **release:** 1.27.0-beta.9 [skip ci] ([fd57cc7](ScrapeGraphAI@fd57cc7))
… for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout. Optimize utils modules for better performance and maintainability - Improve HTML cleanup and minification: - Combine regex operations for better performance - Add better error handling for HTML processing - Optimize tag removal and attribute filtering - Enhance deep copy functionality: - Add special case handling for primitive types - Improve type checking and error handling - Optimize recursive copying for collections - Refactor web search functionality: - Add input validation and error handling - Split search logic into separate helper functions - Improve proxy handling and configuration - Add better timeout and error management - Optimize URL filtering and processing Technical improvements: - Better type hints and documentation - More efficient data structures - Improved error handling and validation - Reduced code duplication - Better separation of concerns No breaking changes - all existing functionality maintained
|
🎉 This PR is included in version 1.28.0-beta.1 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.28.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout.
Optimize utils modules for better performance and maintainability
Improve HTML cleanup and minification:
Enhance deep copy functionality:
Refactor web search functionality:
Technical improvements:
No breaking changes - all existing functionality maintained