New Filter plugin from_csv #2037

Ajpantuso · 2021-03-17T19:49:04Z

SUMMARY

This filter adapts the csv parsing features from the read_csv module to use directly with ansible text/variables

Fixes: #1881

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

plugins/filter/from_csv.py
plugins/modules/read_csv.py
plugins/module_utils/csv.py

ADDITIONAL INFORMATION

Feature set and core implementation is pulled directly from read_csv and adapted to the filter structure.
Common functions have been refactored into module_utils/csv.py which both read_csv and from_csv utilize

Notable Changes:

Return of read_csv is a tuple of list and dict with populated result switching to dict if key is provided. For the purposes of a filter I have chosen to make the result a list of anonymous dicts which can be keyed using jinja e.g. some_var | community.general.from_csv | groupby('key') | flatten is equivalent to keyed output of read_csv
Since keys have been removed there is no need for the unique parameter
Updated instances of ansible.module_utils._text.to_text to ansible.module_utils._text.to_native

Integration tests were added which cover basic functions, but could be updated for better coverage.

Unit tests also added for csv.py.

Wasn't sure how to properly credit the author of read_csv so I updated copyright as if it was an update to the original module

changelogs/fragments/XXXX-add-from-csv-filter.yml

plugins/filter/from_csv.py

ansibullbot · 2021-03-18T02:29:37Z

cc @dagwieers
click here for bot help

changelogs/fragments/2037-add-from-csv-filter.yml

felixfontein · 2021-03-19T18:16:37Z

plugins/module_utils/csv.py

+
+def read_csv(data, dialect, fieldnames=None):
+
+    data = to_native(data, errors='surrogate_or_strict')


In the original code, data = to_text(data, errors='surrogate_or_strict') was used in case of PY3 and nothing in case of PY2. What is the reason behind doing it differently here?

The original module was leveraging the fact that it knew the file was being read in binary format before processing.
However to make this compatible with input passed to a filter (And other general use cases) the read_csv function needs to also convert text input in python 2 using to_native which directs to to_bytes in that case.
Moreover to_bytes just returns binary data untouched. See here

Co-authored-by: Felix Fontein <felix@fontein.de>

changelogs/fragments/2037-add-from-csv-filter.yml

felixfontein · 2021-03-21T12:22:08Z

@Ajpantuso thanks for working on this!

* Added from_csv filter and integration tests * Cleaning up whitespace * Adding changelog fragment * Updated changelog fragment name * Removed temp fragment * Refactoring csv functions Part 1 * Syncing refactored csv modules/filters * Adding unit tests for csv Module_Util * Updating changelog fragment * Correcting whitespace in unit test * Improving changelog fragment Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/2037-add-from-csv-filter.yml Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 6529390)

* Added from_csv filter and integration tests * Cleaning up whitespace * Adding changelog fragment * Updated changelog fragment name * Removed temp fragment * Refactoring csv functions Part 1 * Syncing refactored csv modules/filters * Adding unit tests for csv Module_Util * Updating changelog fragment * Correcting whitespace in unit test * Improving changelog fragment Co-authored-by: Felix Fontein <felix@fontein.de> * Update changelogs/fragments/2037-add-from-csv-filter.yml Co-authored-by: Felix Fontein <felix@fontein.de> (cherry picked from commit 6529390) Co-authored-by: Ajpantuso <ajpantuso@gmail.com>

v3.2.0 community.crypto - acme module_utils - the ``acme`` module_utils has been split up into several Python modules (ansible-collections/community.crypto#184). - acme_* modules - codebase refactor which should not be visible to end-users (ansible-collections/community.crypto#184). - acme_* modules - support account key passphrases for ``cryptography`` backend (ansible-collections/community.crypto#197, ansible-collections/community.crypto#207). - acme_certificate_revoke - support revoking by private keys that are passphrase protected for ``cryptography`` backend (ansible-collections/community.crypto#207). - acme_challenge_cert_helper - add ``private_key_passphrase`` parameter (ansible-collections/community.crypto#207). community.docker - docker_swarm_service - change ``publish.published_port`` option from mandatory to optional. Docker will assign random high port if not specified (ansible-collections/community.docker#99). community.general - archive - refactored some reused code out into a couple of functions (ansible-collections/community.general#2061). - csv module utils - new module_utils for shared functions between ``from_csv`` filter and ``read_csv`` module (ansible-collections/community.general#2037). - ipa_sudorule - add support for setting sudo runasuser (ansible-collections/community.general#2031). - jenkins_job - add a ``validate_certs`` parameter that allows disabling TLS/SSL certificate validation (ansible-collections/community.general#255). - kibana_plugin - add parameter for passing ``--allow-root`` flag to kibana and kibana-plugin commands (ansible-collections/community.general#2014). - proxmox - added ``purge`` module parameter for use when deleting lxc's with HA options (ansible-collections/community.general#2013). - proxmox inventory plugin - added ``tags_parsed`` fact containing tags parsed as a list (ansible-collections/community.general#1949). - proxmox_kvm - added new module parameter ``tags`` for use with PVE 6+ (ansible-collections/community.general#2000). - rax - elements of list parameters are now validated (ansible-collections/community.general#2006). - rax_cdb_user - elements of list parameters are now validated (ansible-collections/community.general#2006). - rax_scaling_group - elements of list parameters are now validated (ansible-collections/community.general#2006). - read_csv - refactored read_csv module to use shared csv functions from csv module_utils (ansible-collections/community.general#2037). - redfish_* modules, redfish_utils module utils - add support for Redfish session create, delete, and authenticate (ansible-collections/community.general#1975). - snmp_facts - added parameters ``timeout`` and ``retries`` to module (ansible-collections/community.general#980). - vdo - add ``force`` option (ansible-collections/community.general#2101). community.network - edgeos_config - match the space after ``set`` and ``delete`` commands (ansible-collections/community.network#199). - nclu - execute ``net commit description <description>`` only if changed ``net pending``'s diff field (ansible-collections/community.network#219). community.postgresql - postgresql_info - add the ``patch``, ``full``, and ``raw`` values of the ``version`` return value (ansible-collections/community.postgresql#68). - postgresql_ping - add the ``patch``, ``full``, and ``raw`` values of the ``server_version`` return value (ansible-collections/community.postgresql#70). community.zabbix - zabbix_agent - added support for installations on arm64 systems (ansible-collections/community.zabbix#320). - zabbix_proxy - now supports configuring StatsAllowedIP (ansible-collections/community.zabbix#337). - zabbix_server - added support for installtions on arm64 systems (ansible-collections/community.zabbix#320). - zabbix_web - added support for installtions on arm64 systems (ansible-collections/community.zabbix#320). dellemc.openmanage - ome_template - Allows to deploy a template on device groups. hetzner.hcloud - Add firewalls to hcloud_server module ovirt.ovirt - cluster_upgrade - Add correlation-id header (oVirt/ovirt-ansible-collection#222). - engine_setup - Add skip renew pki confirm (oVirt/ovirt-ansible-collection#228). - examples - Add recipe for removing DM device (oVirt/ovirt-ansible-collection#233). - hosted_engine_setup - Filter devices with unsupported bond mode (oVirt/ovirt-ansible-collection#226). - infra - Add reboot host parameters (oVirt/ovirt-ansible-collection#231). - ovirt_disk - Add SATA support (oVirt/ovirt-ansible-collection#225). - ovirt_user - Add ssh_public_key (oVirt/ovirt-ansible-collection#232) purestorage.flasharray - purefa_maintenance - New module to set maintenance windows - purefa_pg - Add support to rename protection groups - purefa_syslog - Add support for naming SYSLOG servers for Purity//FA 6.1 or higher purestorage.flashblade - purefb_certs - Add update functionality for array cert - purefb_fs - Add multiprotocol ACL support - purefb_info - Add information regarding filesystem multiprotocol (where available) - purefb_info - Add new parameter to provide details on admin users - purefb_info - Add replication performace statistics - purefb_s3user - Add ability to remove an S3 users existing access key

Ajpantuso added 3 commits March 17, 2021 15:05

Added from_csv filter and integration tests

9109088

Cleaning up whitespace

def912a

Adding changelog fragment

97d5e5b

ansibullbot added affects_2.10 community_review feature This issue/PR relates to a feature request integration tests/integration needs_triage new_plugin New plugin tests tests labels Mar 17, 2021

Updated changelog fragment name

087d4fe

felixfontein reviewed Mar 17, 2021

View reviewed changes

changelogs/fragments/XXXX-add-from-csv-filter.yml Outdated Show resolved Hide resolved

plugins/filter/from_csv.py Outdated Show resolved Hide resolved

Ajpantuso marked this pull request as ready for review March 17, 2021 20:18

Ajpantuso added 6 commits March 17, 2021 16:21

Removed temp fragment

578df40

Refactoring csv functions Part 1

0adf98b

Syncing refactored csv modules/filters

744f83e

Adding unit tests for csv Module_Util

32fd05b

Updating changelog fragment

6f54d68

Correcting whitespace in unit test

a0e691d

ansibullbot added files module module module_utils module_utils plugins plugin (any type) unit tests/unit labels Mar 18, 2021

felixfontein reviewed Mar 19, 2021

View reviewed changes

Improving changelog fragment

64dcc66

Co-authored-by: Felix Fontein <felix@fontein.de>

felixfontein approved these changes Mar 21, 2021

View reviewed changes

felixfontein added the backport-2 label Mar 21, 2021

felixfontein reviewed Mar 21, 2021

View reviewed changes

changelogs/fragments/2037-add-from-csv-filter.yml Outdated Show resolved Hide resolved

Update changelogs/fragments/2037-add-from-csv-filter.yml

3cbcec6

ansibullbot removed the needs_triage label Mar 21, 2021

felixfontein merged commit 6529390 into ansible-collections:main Mar 21, 2021

patchback bot mentioned this pull request Mar 21, 2021

[PR #2037/65293909 backport][stable-2] New Filter plugin from_csv #2074

Merged

Ajpantuso deleted the filter_from_csv branch March 21, 2021 14:39

Andersson007 mentioned this pull request Apr 19, 2021

BOTMETA.yml: filter/from_csv - add a maintainer #2272

Merged

patchback bot mentioned this pull request Apr 19, 2021

[PR #2272/20401c63 backport][stable-2] BOTMETA.yml: filter/from_csv - add a maintainer #2278

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Filter plugin from_csv #2037

New Filter plugin from_csv #2037

Ajpantuso commented Mar 17, 2021 •

edited

Loading

ansibullbot commented Mar 18, 2021

felixfontein Mar 19, 2021

Ajpantuso Mar 19, 2021

felixfontein commented Mar 21, 2021


		def read_csv(data, dialect, fieldnames=None):

		data = to_native(data, errors='surrogate_or_strict')

New Filter plugin from_csv #2037

New Filter plugin from_csv #2037

Conversation

Ajpantuso commented Mar 17, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ansibullbot commented Mar 18, 2021

felixfontein Mar 19, 2021

Choose a reason for hiding this comment

Ajpantuso Mar 19, 2021

Choose a reason for hiding this comment

felixfontein commented Mar 21, 2021

Ajpantuso commented Mar 17, 2021 •

edited

Loading