Export all survey tables to CSV

[fungjj92]

Overview

Site admins want the data as CSV to do analyses. This PR adds a view at http://localhost:8000/beekeepers/export for admins to download tables as a zip of CSVs. SQL queries are hardcoded.

Connects #384
Connects #385

Demo

The idea: (i fussed with the filename timestamp a little since this screenshot)

The date formatter doesn't allow colons in the time (only -). It's livable. It's main purpose is to give the files a unique ID.

Non-admins can't access:

Testing Instructions

Visit the URL and sign in as an admin. Check that the zip downloads and the CSVs look good.

[rajadain]

This also needs a management command, so it can run like this:

$ ./scripts/manage.sh beekeepers_export
+ ARGS=beekeepers_export
+ [[ 1 -eq 1 ]]
+ [[ beekeepers_export == runserver ]]
+ vagrant ssh app -c 'cd /opt/app && envdir /etc/icp.d/env ./manage.py beekeepers_export'
Export Complete
Connection to 127.0.0.1 closed.

I added a commit for that (since it was easier than writing it out in a comment).

Currently any new export overwrites the old files. Can we make it so that the exports are zipped into an archive with a timestamp, so that later exports do not overwrite the old ones?

[rajadain]

Can we have the CSV export quote string and date fields? The current export looks like this:

id,inspection_date,colony_name,colony_alive,colony_loss_reason,num_bodies_supers_deep,num_bodies_supers_medium,num_bodies_supers_shallow,activity_since_last,queenright,same_queen,queen_stock,queen_source,varroa_count_performed,varroa_count_technique,varroa_count_result,varroa_treatment,survey_id
1,2018-10-09,A,t, ,0,0,3, ,f,YES, ,NON_LOCAL_COMMERCIAL,f, ,10, ,5
2,2018-11-12,A,t, ,0,0,3, ,f,YES, ,NON_LOCAL_COMMERCIAL,f, ,10, ,8
3,2018-12-27,A,t, ,0,0,3, ,f,YES, ,NON_LOCAL_COMMERCIAL,f, ,10,CHEMICAL_FORMIC_ACID_FORMIC_PRO;CHEMICAL_OXALIC_ACID_DRIBBLE;CHEMICAL_THYMOL_MENTHOL_APILIFE;CHEMICAL_SYNTHETIC_CHECKMITE_PLUS;MECHANICAL_QUEEN_MANIPULATION;OTHER-heat vision;CHEMICAL_ORGANIC_OTHER-freeze breath;MECHANICAL_OTHER-punching,7
4,2019-01-14,A,f,INADEQUETE_FOOD_STORES;COLONY_TOO_SMALL_IN_NOVEMBER,0,2,0,REMOVED_BROOD,f,YES,"",NON_LOCAL_COMMERCIAL,f,"",0,CHEMICAL_OXALIC_ACID_DRIBBLE;CHEMICAL_SYNTHETIC_CHECKMITE_PLUS;CHEMICAL_ORGANIC_OTHER-asdsadadas,21
5,2019-01-15,A,t,"",23,0,0,FED_POLLEN_PROTEIN,f,YES,"",NON_LOCAL_COMMERCIAL,f,"",0,CHEMICAL_OXALIC_ACID_VAPORIZATION;CHEMICAL_SYNTHETIC_APIVAR;CHEMICAL_ORGANIC_OTHER-asdsadadas,22
6,2019-01-16,A,t,"",0,0,0,"",f,YES,"",REQUEENED,f,"",0,CHEMICAL_THYMOL_MENTHOL_APILIFE;CHEMICAL_SYNTHETIC_APISTAN;MECHANICAL_QUEEN_MANIPULATION;CHEMICAL_ORGANIC_OTHER-asdsadadas,25

whereas it should look like this:

"id","inspection_date","colony_name","colony_alive","colony_loss_reason","num_bodies_supers_deep","num_bodies_supers_medium","num_bodies_supers_shallow","activity_since_last","queenright","same_queen","queen_stock","queen_source","varroa_count_performed","varroa_count_technique","varroa_count_result","varroa_treatment","survey_id"
1,"2018-10-09","A",TRUE," ",0,0,3," ",FALSE,"YES"," ","NON_LOCAL_COMMERCIAL",FALSE," ",10," ",5
2,"2018-11-12","A",TRUE," ",0,0,3," ",FALSE,"YES"," ","NON_LOCAL_COMMERCIAL",FALSE," ",10," ",8
3,"2018-12-27","A",TRUE," ",0,0,3," ",FALSE,"YES"," ","NON_LOCAL_COMMERCIAL",FALSE," ",10,"CHEMICAL_FORMIC_ACID_FORMIC_PRO;CHEMICAL_OXALIC_ACID_DRIBBLE;CHEMICAL_THYMOL_MENTHOL_APILIFE;CHEMICAL_SYNTHETIC_CHECKMITE_PLUS;MECHANICAL_QUEEN_MANIPULATION;OTHER-heat vision;CHEMICAL_ORGANIC_OTHER-freeze breath;MECHANICAL_OTHER-punching",7
4,"2019-01-14","A",FALSE,"INADEQUETE_FOOD_STORES;COLONY_TOO_SMALL_IN_NOVEMBER",0,2,0,"REMOVED_BROOD",FALSE,"YES","","NON_LOCAL_COMMERCIAL",FALSE,"",0,"CHEMICAL_OXALIC_ACID_DRIBBLE;CHEMICAL_SYNTHETIC_CHECKMITE_PLUS;CHEMICAL_ORGANIC_OTHER-asdsadadas",21
5,"2019-01-15","A",TRUE,"",23,0,0,"FED_POLLEN_PROTEIN",FALSE,"YES","","NON_LOCAL_COMMERCIAL",FALSE,"",0,"CHEMICAL_OXALIC_ACID_VAPORIZATION;CHEMICAL_SYNTHETIC_APIVAR;CHEMICAL_ORGANIC_OTHER-asdsadadas",22
6,"2019-01-16","A",TRUE,"",0,0,0,"",FALSE,"YES","","REQUEENED",FALSE,"",0,"CHEMICAL_THYMOL_MENTHOL_APILIFE;CHEMICAL_SYNTHETIC_APISTAN;MECHANICAL_QUEEN_MANIPULATION;CHEMICAL_ORGANIC_OTHER-asdsadadas",25

That'll protect from any commas in the content (which may be in the free-entry fields).

[fungjj92]

The good news is that postgres or psycopg must add quotes when appropriate. Here is the output of april surveys when a string entry contains commas (last row).

survey_id,colony_loss_reason
55,varroa mites
60,varroa mites
63,INADEQUETE_FOOD_STORES;OTHER-super sad bee killer
64,VARROA_MITES;INADEQUETE_FOOD_STORES;OTHER-super sad bee killer
65,VARROA_MITES;OTHER-super sad bee killer2
66,COLONY_TOO_SMALL;PESTICIDE_EXPOSURE;OTHER-
67,VARROA_MITES;POOR_WEATHER_CONDITIONS;OTHER-
69,"VARROA_MITES;OTHER-super, sad, bee, killer"

[rajadain]

Heads up that there's a merge conflict now.

[fungjj92]

Ready for a look

[rajadain]

Taking a look now.

[rajadain]

+1 tested, this works well. Export had the right columns and consistent timestamp. With a few cleanup commits this should be ready.

[fungjj92]

Thanks! Helpful clean up.

[rajadain]

Still needs a couple of fixes before it's green.

[fungjj92]

Non-staff user:

The button on web and on a screen reader:

[rajadain]

Taking another look.

[rajadain]

+1 tested. Good work!

[fungjj92]

😅 Thanks for the reviews.