Variables sent via 'environment' when using 'django_manage' and 'become_user' are not present in host #5374
Comments
|
Files identified in the description: If these files are incorrect, please update the |
ansibullbot
added
bug
|
I just realised of this bit in the documentation of django_manage:
Later, in the notes section, there is this:
I just realised that I forgot to mention that my virtual environment was created using Python 3 Does |
|
I just checked the code and, if I am not mistaken, I see that it's checking for the existence of Not that it has anything to do with this issue, but I had hoped to export the necessary Django variables inside the |
|
This is a modified playbook I am running now, to try and see what is happening.
---
- hosts: django
tasks:
- ansible.builtin.import_tasks: tasks/django/manage.yml
- name: builtin | template | template host variables file
ansible.builtin.template:
src: templates/django/hostvars.j2
dest: "host_vars/{{ inventory_hostname_short }}.yml"
mode: 0640
output_encoding: "utf-8"
delegate_to: localhost
- name: builtin | include_vars | load host vars into the 'envvars' variable
ansible.builtin.include_vars:
file: "host_vars/{{ inventory_hostname_short }}.yml"
name: envvars
---
# Host variables to be set as environment variables in tasks that need it
POSTGRES_PASSWORD: "<password>"
POSTGRES_USER: "dbuser"
POSTGRES_DB: "dbname"
POSTGRES_HOST: "dbhost"
POSTGRES_PORT: 5432
POSTGRES_SSLMODE: "verify-full"
POSTGRES_SSLCA: "/etc/ssl/certs/ISRG_Root_X1.pem"
POSTGRES_APPNAME: "myproject"
DJANGO_SITE_NAME: "mysite"
DJANGO_SITE_PASSWORD: "mypassword"
DJANGO_SITE_USER: "myuser"
DJANGO_SITE_ID: 2
DJANGO_SECRET_KEY: "<very-long-and-random-secret>"
[..]
- name: builtin | shell | capture DJANGO_ environment variables
debugger: on_failed
ansible.builtin.shell:
cmd: "env | grep DJANGO_"
register: out
environment: "{{ envvars }}"
become: true
become_user: django
become_method: su
- name: builtin | debug | pinrt content of out.stdout
ansible.builtin.debug:
var: out.stdout
- name: builtin | debug | print variable 'envvars'
ansible.builtin.debug:
var: envvars
- name: community.general | django_manage | populate the static subdirectory
community.general.django_manage:
command: collectstatic
clear: yes
project_path: "/opt/django/src"
virtualenv: "/opt/django/venv"
become: true
become_user: django
become_method: su
environment: "{{ envvars }}"Only the last task is failing. Here is the output: TASK [builtin | shell | capture DJANGO_ environment variables] *************************************************************************************************************
changed: [django1.donmain.com]
TASK [builtin | debug | pinrt content of out.stdout] ***********************************************************************************************************************
ok: [django1.django.com] => {
"out.stdout": "DJANGO_SITE_USER=mysite\nDJANGO_MEDIA_BASE=/opt/django/media\nDJANGO_SITE_NAME=mysite\nDJANGO_SITE_ID=2\nDJANGO_SECRET_KEY=<very-secret-key>\nDJANGO_LOGS_DIR=/opt/django/logs\nDJANGO_SETTINGS_MODULE=myproject.settings.production\nDJANGO_DEBUG=0\nDJANGO_STATIC_BASE=/opt/django/static\nDJANGO_SITE_PASSWORD=mypassword\nDJANGO_SITE_VERSION=57a2f3c168d86243f03809e5d02a0f50a8fa892e"
}
TASK [builtin | debug | print variable 'envvars'] **************************************************************************************************************************
ok: [django1.domain.com] => {
"envvars": {
"DJANGO_DEBUG": 0,
"DJANGO_LOGS_DIR": "/opt/django/logs",
"DJANGO_MEDIA_BASE": "/opt/django/media",
"DJANGO_SECRET_KEY": "<very-secret-key>",
"DJANGO_SETTINGS_MODULE": "myproject.settings.production",
"DJANGO_SITE_ID": 2,
"DJANGO_SITE_NAME": "mysite",
"DJANGO_SITE_PASSWORD": "mypassword",
"DJANGO_SITE_USER": "myuser",
[..]
}
}
TASK [community.general | django_manage | populate the static subdirectory] ************************************************************************************************
fatal: [django1.domain.com]: FAILED! => {"changed": false, "cmd": ["./manage.py", "collectstatic", "--noinput", "--clear"], "msg": "\n:stderr: Traceback (most recent call last):\n File \"/opt/django/venv/lib/python3.9/site-packages/django/core/management/__init__.py\", line 204, in fetch_command\n app_name = commands[subcommand]\nKeyError: 'collectstatic'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/django/src/./manage.py\", line 22, in <module>\n execute_from_command_line(sys.argv)\n File \"/opt/django/venv/lib/python3.9/site-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line\n utility.execute()\n File \"/opt/django/venv/lib/python3.9/site-packages/django/core/management/__init__.py\", line 375, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File \"/opt/django/venv/lib/python3.9/site-packages/django/core/management/__init__.py\", line 211, in fetch_command\n settings.INSTALLED_APPS\n File \"/opt/django/venv/lib/python3.9/site-packages/django/conf/__init__.py\", line 57, in __getattr__\n self._setup(name)\n File \"/opt/django/venv/lib/python3.9/site-packages/django/conf/__init__.py\", line 44, in _setup\n self._wrapped = Settings(settings_module)\n File \"/opt/django/venv/lib/python3.9/site-packages/django/conf/__init__.py\", line 107, in __init__\n mod = importlib.import_module(self.SETTINGS_MODULE)\n File \"/usr/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n File \"<frozen importlib._bootstrap_external>\", line 790, in exec_module\n File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n File \"/opt/django/src/black_pearl/settings/production.py\", line 3, in <module>\n from black_pearl.settings.common import *\n File \"/opt/django/src/black_pearl/settings/common.py\", line 301, in <module>\n path_app = import_module(app).__path__\n File \"/usr/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\nModuleNotFoundError: No module named 'None'\n", "path": "/opt/django/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "syspath": ["/tmp/ansible_community.general.django_manage_payload_l53eeo4g/ansible_community.general.django_manage_payload.zip", "/usr/lib/python39.zip", "/usr/lib/python3.9", "/usr/lib/python3.9/lib-dynload", "/usr/local/lib/python3.9/dist-packages", "/usr/lib/python3/dist-packages"]}
PLAY RECAP *****************************************************************************************************************************************************************
django1.domain.com : ok=14 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0This is being executed as:
May it be happening that, when |
|
Hi @jsabater thanks for your detailed report. I will be taking a look at it throughout this week. |
|
Hi @jsabater I am glancing over this right now, couple of answers and thoughts:
|
|
Dear @russoz,
Also, for your information, I forgot to mention that I also tried passing the variables explicitly, without success: - name: community.general | django_manage | populate the static subdirectory
community.general.django_manage:
command: collectstatic
clear: yes
project_path: "/opt/django/src"
virtualenv: "/opt/django/venv"
become: true
become_user: django
become_method: su
environment:
DJANGO_DEBUG: 0,
DJANGO_LOGS_DIR: "/opt/django/logs",
DJANGO_MEDIA_BASE: "/opt/django/media",
DJANGO_SECRET_KEY: "<very-secret-key>",
DJANGO_SETTINGS_MODULE: "myproject.settings.production",
DJANGO_SITE_ID: 2,
DJANGO_SITE_NAME: "mysite",
DJANGO_SITE_PASSWORD: "mypassword",
DJANGO_SITE_USER: "myuser",
[..]I tried both as it appears here (a dictionary, if I am not mistaken), and using scores (a list of very small dictionaries, if I am not mistaken). |
|
I gave the playbook a try using these options: But no success either. I went to the container and, as root, executed the following command: ~# sudo --login -u django envThe output of the command was correct, as it included all the environment variables that are being set at the |
|
Hi @jsabater Thanks for the further examples. I expect to have more time for this on Friday this week, but will try to make some progress before that. |
|
Ah, @jsabater probably not going to affect anything, but just for the sake of completeness when I reproduce the error, what version of Django are you using? |
|
Right now the requirements file says ~# grep -i django requirements.txt
Django==2.1.15
django-cms==3.7.4
djangocms-installer==1.2.3
djangocms-admin-style==1.5
django-treebeard==4.3.1
django-parler==2.1
djangocms-bootstrap4==1.6.0 # bootsrap4 cms plugins
djangocms-text-ckeditor==3.10.0 # rich text areas
djangocms-link==2.6.1 # for link cms plugins, required by bootsrap4
djangocms-style==2.3.0
djangocms-googlemap==1.4.0
djangocms-snippet==2.3.0
djangocms-video==2.3.0
djangocms-column==1.11.0
djangocms-file==2.4.0
djangocms-picture==2.4.0
djangocms-page-meta==1.0.1
djangocms-page-sitemap==0.8.1
djangocms-redirect==0.5.0
djangocms-blog==1.2.3
djangocms-transfer
django-admin-sortable==2.2.3
django-classy-tags==1.0.0
django-colorfield==0.3.2
django-compressor==2.4
django-enumchoicefield==2.0.0
django-filer==1.7.1
django-meta==2.0.0
django-privacy-mgmt==0.0.20
djangorestframework==3.11.2
django-recaptcha3==0.2.0
django-robots==4.0
django-sass-processor==0.8
django-sekizai==1.1.0
django-solo==1.1.3
django-flatpickr==1.0.1
django-payments==0.13.0
django-payments-redsys==0.3.post1
django-pwa==1.0.10 |
|
Also, for your information, for the time being I've resorted to this in order to have a working playbook:
The environment variables set by the Hopefully we'll be able to resolve this issue and be able to use Please let me know if you require anything else. I can provide the full playbook, if needed, and even the source code of the project. |
|
I just realised how old the version is and I've requested them to update to the latest. I'll try and run the same playbook on the latest version of Django as soon as possible, just in case the source of the error could be there, and get back to you. |
|
I very much doubt that will fix the issue, but yeah, upgrading that is definitely a good idea. |
|
As to your idempotency question, I believe it does indeed delegate that aspect to |
|
@jsabater I am looking for examples show Django using env vars and the ones I found[1] seem to point to:
However, I do not see that package in your requirements, nor have you shown exactly how the env vars are being used. On top of that, I could not help noticing that in the error message on the output of the playbook presented above the reported problem is a failed import in Python. Could you please paste an excerpt of your I am thinking of adding your case to test files under [1] https://djangocentral.com/environment-variables-in-django/ |
|
Dear, @russoz! I just went through all the files inside the While I have to admit that loading the environment variables (a.k.a. settings) from a file rather than the shell environment is (and always has been, generally speaking) my preferred solution, it is just one way to do things, and should not alter the issue at hand. Nonetheless, I've taken the chance to convince the devs of the Django application to change the way variables are loaded so that:
So, answering your question, they way env vars are being used right now is:
SECRET_KEY = os.getenv('DJANGO_SECRET_KEY')
SITE_ID = int(os.getenv("DJANGO_SITE_ID", 1))
SITE_NAME = os.getenv('DJANGO_SITE_NAME') or os.getenv('USER')
LOGS_DIR = os.getenv('DJANGO_LOGS_DIR', '/opt/django/logs')And used or altered, then used, as the devs see fit. When running commands such as [1] Incidentally, I've instructed them not to use |
|
Regarding the exception being thrown (failed import), it made me wonder whether
My playbook has this value, as you can see in this thread: Then I opened the #!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "black_pearl.settings.development")
try:
from django.core.management import execute_from_command_line
[..]Then I checked the environment variables: $ env | grep DJANGO_SETTINGS_MODULE
DJANGO_SETTINGS_MODULE=myproject.settings.productionSo when you use the console to run the command Finally, just out of curiosity, I changed the value of the ---
- hosts: django1.domain.com
tasks:
- name: builtin | include_vars | load host vars
ansible.builtin.include_vars:
file: "host_vars/{{ inventory_hostname_short }}.yml"
name: envvars
- name: community.general | django_manage | update database schema
community.general.django_manage:
command: migrate
settings: black_pearl.settings.production
project_path: "/opt/django/src"
virtualenv: "/opt/django/venv"
become: true
become_user: django
become_method: su
become_flags: '--login'
environment: "{{ envvars }}"And got the following error: # ansible-playbook django-test.yml
PLAY [django1.domain.com] ********************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************************************************************************************************
ok: [django1.domain.com]
TASK [builtin | include_vars | load host vars] ****************************************************************************************************************************************************************************************************
ok: [django1.domain.com]
TASK [community.general | django_manage | update database schema] *********************************************************************************************************************************************************************************
fatal: [django1.domain.com]: FAILED! => {"changed": false, "msg": "Failed to find required executable \"virtualenv\" in paths: /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/usr/local/sbin"}
PLAY RECAP ****************************************************************************************************************************************************************************************************************************************
django1.domain.com : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0I hope that it provides a hint or helps somehow :-) |
|
Incidentally, since this thread is becoming fairly dense in terms of information and details, I provided the latest test bed and error (Failed to find required executable "virtualenv" in paths) because, as stated across the thread, the Python virtual environment was not created using ---
- name: builtin | pip | create virtual environment and upgrade pip
ansible.builtin.pip:
name: pip
state: latest
virtualenv: /opt/django/venv
virtualenv_command: /usr/bin/python3 -m venv
extra_args: "--quiet --no-cache-dir --trusted-host pypi.org"
become: true
become_user: django
become_method: su
environment:
https_proxy: "{{ http_proxy_url }}" |
|
Hi @jsabater I wrote a Python3-version of the code to ensure a virtual environment is built, and I started wondering why I was having a bad gut feeling about it. And I noted that:
Given that, I think we should deprecate the behaviour "create venv if one does not exist". The responsibility for creating and populating correctly the venv does not belong to this module. Additionally that would remove the dependency with What do you think? |
|
I think that your reasoning is sound and I could not agree more with it.
…--
Jaume Sabater
Ubi sapientas ibi libertas
On Sat, Oct 22, 2022, 01:38 Alexei Znamensky ***@***.***> wrote:
Hi @jsabater <https://github.com/jsabater> I wrote a Python3-version of
the code to ensure a virtual environment is built, and I started wondering
why I was having a bad gut feeling about it. And I noted that:
1. In the spirit of "small specialized tools" from "Unix philosophy",
we are trying to solve the problem "build a virtualenv" inside the module
that is supposed to "manage django apps", so that sounds like a bad idea
from the start.
2. Suppose we have a pristine server where we run this module. No venv
exists, so this module will ensure one is there. However, at a bare
minimum, a virtualenv used to run ./manage.py *should have Django
installed in it*, and the new virtualenv does not!!! Now, if we decide
to "solve" that by, hey, installing django in that venv from within this
module, not only we make problem 1 above worse, but we also create further
problems: which version should be installed? what other packages must be
installed in that venv?
Given that, I think we should deprecate the behaviour "create venv if one
does not exist". The responsibility for creating and populating correctly
the venv does not belong to this module. Additionally that would remove the
dependency with virtualenv and ditch the debate on python -m venv.
What do you think?
—
Reply to this email directly, view it on GitHub
<#5374 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABGHTSCP2W6JVZO73F26BDWEMSNRANCNFSM6AAAAAARGPDXGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Just a heads up that the dev team ended up importing variables from a How did those tests go? Any news on why the environment variables are not reaching the final I saw that the pull request was merged into the next version of |
|
Files identified in the description: If these files are incorrect, please update the |
|
hi @jsabater thanks for following up. The PR was about tidying up some things we identified in the module, but I haven't had the time to work on those tests yet. I intend to write the automated tests that will verify the environment variables work. My expectation is that they work fine - it is not a feature of this module, but one of Ansible core itself - so any problems with that feature would likely trigger many complaints in many different contexts. I suspect the problem you experienced derived from the particular deployment setup. |
|
@jsabater there you have it - environment variables arriving safely at |
|
I'll show the test to the devs and try to get some feedback. Feel free to close the issue if you consider it solved and thank you very much for your time and feedback. |
|
Thank you for reporting. I will close as per your comment, but feel free to reopen if you believe that we did not covered some angle of that problem. |
Summary
have a file with a list of variables and their values, originally templated but now static as I try to debug this issue. It contains a number of variables that are required by Django, such as
DJANGO_SECRET_KEY,DJANGO_SITE_NAME, and so on.I use
ansible.builtin.include_varsto load these variables into a variable in the playbook.I use
community.general.django_manageto run themigratecommand, using theenvironment:option to pass the previously loaded variables, but Django complains that it cannot find them.I've checked with
ansible-playbook -vvvand, apparently, they are being sent to the host.I am not sure whether it has to do with
environment:or with using such directive when also usingbecome,become_userandbecome_method. I have tried withbecome_method: sudoand the end result is the same.I've found issue 2568 [1] from 2013 where the proposed solution was using the
environment:directive, which is also backed by [the docs](name: envvars), if I've read correctly.[1] ansible/ansible#2568
Thanks in advance.
Issue Type
Bug Report
Component Name
community.general.django_manage
Ansible Version
Community.general Version
Configuration
This is my playbook:
djangois a group of hosts, defined in the/etc/ansible/hostsas:The playbook is being executed like this:
ansible-playbook django-test.yml --limit django1.domain.comExpected Results
I expected the command to execute without issues, finding the environment variables sent to do its job.
Actual Results
This is an excerpt of the output using
-vvv:Code of Conduct
The text was updated successfully, but these errors were encountered: