Skip to content

Using the Vagrant Dev Environment

Jump to bottom
Chris Houseknecht edited this page Aug 10, 2021 · 3 revisions

Installing Pulp and galaxy NG from source

Pulp Installer is a vagrant configuration based on forklift.

Requirements

1. Install Vagrant and its plugins

On a fedora system

sudo dnf install ansible vagrant-libvirt vagrant-sshfs @virtualization

On a debian system

# virtualbox (requires sid in sources)
sudo apt install ansible vagrant vagrant-sshfs virtualbox/sid

# libvirt
sudo apt install ansible vagrant vagrant-sshfs vagrant-libvirt dnsmasq libvirt-clients libvirt-daemon libvirt-dbus qemu-system-x86 qemu-utils
sudo usermod -aG libvirt,libvirt-qemu,libvirtdbus $USER

On a Mac

brew install ansible
brew cask install virtualbox
brew cask install vagrant

On other host systems

Refer to the package manager and search for equivalent packages. For example, pacman -S vagrant

Install vagrant plugins

Required vagrant plugins:

vagrant plugin install vagrant-sshfs

Optional plugins:

vagrant plugin install vagrant-libvirt       # to connect to libvirt 
vagrant plugin install vagrant-hostmanager   # to manage local dns

2. Clone the repositories from source

TIP: replace :pulp/ and :ansible/ with your own github username if you plan to work on your own forks.

# required
git clone git@github.com:pulp/pulp_installer
git clone git@github.com:pulp/pulp_ansible.git
git clone git@github.com:pulp/pulp_container.git
git clone git@github.com:ansible/galaxy_ng.git
git clone git@github.com:pulp/pulpcore.git

# optional
git clone git@github.com:ansible/galaxy-importer.git
git clone git@github.com:ansible/ansible-hub-ui.git

Ensure repositories are located on the same folder level

$ tree -a -L 1
.
├── ansible-hub-ui/
├── galaxy-importer/
├── galaxy_ng/
├── pulp_ansible/
├── pulp_container/
├── pulpcore/
└── pulp_installer/

At this point you can manually checkout each repo to desired branch/tag/commit or skip this part and let the installer do it for you with compatible versions listed on the local.dev-config.yml

cd pulpcore
git checkout 3.11  # note 3.11 is a branch, thus NEWER than 3.11.1 indicated in setup.py
cd ../pulp_ansible
git checkout 0.7.3
cd ../pulp_container
git checkout 2.5.2
cd ..

3. Set your working directory to the pulp_installer directory

cd pulp_installer

4. make sure you're running the latest compatible version of pulp_installer.

git checkout 3.11

5. Initialize submodules

git submodule update --init

6. Create the installer config file

In the root of the pulp_installer directory create a new file named local.dev-config.yml with the following contents:

---
# Pulp Source
pulp_source_dir: "/home/vagrant/devel/pulpcore"
pulp_pip_editable: true
pulp_git_url: https://github.com/pulp/pulpcore.git
pulp_git_revision: "3.14.4"

# Pulp plugins and Python libs
pulp_install_plugins:
  pulp-ansible:
    source_dir: "/home/vagrant/devel/pulp_ansible"
  pulp-container:
    source_dir: "/home/vagrant/devel/pulp_container"
  galaxy-importer:
    source_dir: "/home/vagrant/devel/galaxy-importer"
  galaxy-ng:
    source_dir: "/home/vagrant/devel/galaxy_ng"

# Pulp configuration
pulpcore_version: 3.14.4
pulp_devel_supplement_bashrc: true
pulp_default_admin_password: password
pulp_webserver_disable_https: true
pulp_user: "vagrant"
developer_user: "vagrant"
developer_user_home: "/home/vagrant"
pulp_workers: 4
pulp_api_workers: 4
pulp_settings:
  secret_key: "unsafe_default"
  content_origin: "http://{{ ansible_fqdn }}"
  x_pulp_api_host: 127.0.0.1
  x_pulp_api_port: 24817
  x_pulp_api_user: "admin"
  x_pulp_api_password: "{{ pulp_default_admin_password }}"
  x_pulp_api_prefix: "pulp_ansible/galaxy/automation-hub/api"
  galaxy_require_content_approval: "False"
  pulp_token_auth_disabled: "True"
  galaxy_api_default_distribution_base_path: "published"

# Galaxy Configuration
# Set this __galaxy variables according to your needs.
# __galaxy_profile: 'insights'or 'standalone'
__galaxy_profile: 'standalone'
# __galaxy_dev_source_path: `:` separated relative paths to the repos you cloned.
__galaxy_dev_source_path: 'pulpcore:pulp_ansible:pulp_container:galaxy_ng:galaxy-importer'
# __galaxy_lock_requirements: Set to 0 to avoid pinning of galaxy_ng/setup.py versions
__galaxy_lock_requirements: '0'

WARNING: When provisioning the VM you can see errors such as Version Conflict Error and those errors are all related to set the correct version/branch/tag on each repo.

7. Start the vagrant VM

Use of the the available boxes or run vagrant status to see the list of available boxes.

Example:

NOTE the following commands must run inside pulp_installer directory.

vagrant up --provider=libvirt pulp3-source-fedora33  # recommended
vagrant up --provider=libvirt pulp3-source-centos8   # if you need RHEL specific features

The above command will use --provider to provision a Vm and you use libvirt or virtualbox, ensure the respective service is running and accessible. Then it will use local.dev-config.yml to configure the VM.

You can use the virtualbox application or virt-manager to check the state of the VM or run vagrant status pulp3-source-fedora33

NOTE: the libvirt plugin is not available on all platforms, skip --provider=libvirt if things break.

WARNING This command may take several minutes to run, it may ask your root password and in case of Version Conflict Error refer to https://github.com/ansible/galaxy_ng/wiki/Installing-from-source---development-environment/_edit#2-clone-the-repositories-from-source step.

WARNING Vagrant silently ignores --provider=... if user before up. The right syntax is vagrant up --provider=..., not vagrant --provider=... up.

8. Access Galaxy NG and PULP

Pulp-Installer will expose the services on the DNS <box-name>.localhost.example.com for example, if you installed on a fedora system it will be http://pulp3-source-fedora33.localhost.example.com/ui/

If you installed vagrant-hostmanager you can then run vagrant hostmanager to update your hosts file.

Otherwise you will need to add manually to the /etc/hosts file. run vagrant ssh pulp3-source-fedora33 to connect to the VM and then ifconfig to see its ipaddress and then.

# /etc/hosts
...
192.168.121.51 pulp3-source-fedora33.localhost.example.com

To enter the SSH just run vagrant ssh pulp3-source-fedora33

The http server will either listen on http://pulp3-source-fedora33.localhost.example.com (port 80), or on http://localhost:8080. (This probably depends on the chosen vagrant provider?)

9. Optional - Switch to the source version of galaxy-importer by doing the following:

```
## SSH into the vagrant box: 
$ vagrant ssh pulp3-source-fedora33

## Within the vagrant box, install the local copy of `galaxy-importer` and restart Pulp:
$ source /usr/local/lib/pulp/bin/activate
$ cd /home/vagrant/devel/galaxy-importer
$ pip install --upgrade -e .
$ prestart
```

10. Optional - Enable running ansible-test during Collection import:

```
# SSH into the vagrant guest:
$ vagrant ssh pulp-source-fedora32

# Install podman-docker
$ sudo yum install podman-docker

# Configure galaxy-importer
$ sudo mkdir /etc/galaxy-importer
```

Copy the following to `/etc/galaxy-importer/galaxy-importer.cfg`

```
[galaxy-importer]
LOG_LEVEL_MAIN = INFO
RUN_FLAKE8 = True
RUN_ANSIBLE_TEST = True
INFRA_LOCAL_IMAGE = True
INFRA_LOCAL_DOCKER = False
INFRA_OSD = False
```

Troubleshooting

NOTE When using Centos 8, there's currently a bug in vagrant-sshfs that causes the fuse-sshfs package to not install in the guest. Until that gets fixed, best to use Fedora 31+ to test an Enterprise Linux distro.

To use Centos 8 with Virtualbox (assuming the vagrant-sshfs issue is fixed), check vagrant/boxes.d/30-source.yaml to see if the box being referenced points to a URL. If so, take a look at https://cloud.centos.org/centos/8-stream/x86_64/images/, and update the URL to reference an image that's compatible with Virtualbox. The delivered URL was pointing to a Libvirt compatible box.

If using Centos 7 with a clone of the ansible-hub-ui project, the UI will not build without first upgrading the version of Node. This might be accomplished by adding an inline script to the config section of the Vagrantfile. Otherwise, expect the build to fail :-(

Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory - libvirtd or libvirt-daemon-system needs to be installed and running

Call to virConnectOpen failed: authentication unavailable: no polkit agent available to authenticate action 'org.libvirt.unix.manage' - the current user needs to be a member of the libvirt system group

Working on master branches

I need to work on pulp_ansible or pulp_container master how do I do?

If you need to work on pulp_ansible,pulpcore or pulp_container master branches do the following:

  1. first do the normal provisioning using compatible versions/tags/branched described above
  2. ssh in to the VM vagrant ssh pulp3-source-fedora33
  3. stop pulp serviced pstop
  4. go to the repo and checkout to the desired branch or tag, you can do that inside VM in /home/vagrant/devel or on your local host directory as they are mounted inside VM.
  5. Run workon pulp inside the VM ssh session and then run django-admin migrate and resolve any conflict
  6. restart pulp services pstart

Tips and tricks

The installation comes with some useful dev aliases, once in a vagrant ssh session you can:

Activate pulp virtualenv

workon pulp
  • phelp: List all available aliases.
  • pstart: Start all pulp-related services
  • pstop: Stop all pulp-related services
  • prestart: Restart all pulp-related services
  • pstatus: Report the status of all pulp-related services
  • pdbreset: Reset the Pulp database - THIS DESTROYS YOUR PULP DATA
  • pclean: Restore pulp to a clean-installed state - THIS DESTROYS YOUR PULP DATA
  • pjournal: Interact with the journal for pulp-related unit
  • reset_pulp2: Resets Pulp 2 - drop the DB, remove content and publications from FS, restart services.
  • populate_pulp2_iso: Syncs 4 ISO repos.
  • populate_pulp2_rpm: Sync 1 RPM repo.
  • populate_pulp2_docker: Sync 1 Docker repo.
  • populate_pulp2: Reset Pulp 2 and sync ISO, RPM, Docker repos.
  • pyclean: Cleanup extra python files
  • pfixtures: Run pulp-fixtures container in foreground
  • pbindings: Create and install bindings. Example usage: pbindings pulpcore python
  • pminio: Switch to minio for S3 testing. For stopping it: pminio stop

Sponsored by Red Hat, Inc.

Clone this wiki locally