Source install of CKAN 2.9.4 on OS X 11.6

Source Install CKAN on OS X 11.6

1. [Optional] Build out conda distribution

Other flavours of python 3 probably work though at the time of writing 3.10 suffers from a pg-config error and 3.9 suffers from xml.etree.ElementTree.Element' object has no attribute 'getiterator'

$ conda create -n python-3-6-8
$ conda activate python-3-6-8
$ conda config --set channel_priority false
$ conda install python=3.6.8

2. Install CKAN, make new venv

$ git clone git@github.com:ckan/ckan.git
$ python -m venv ckan
$ cd ckan
$ source bin/activate

3. Install required services (postgres, redis, solr, tomcat)

$ brew update 
$ brew install postgresql redis solr
# Optional:
$ brew install tomcat

4. Install project dependencies

$ pip install --upgrade pip
$ pip install setuptools==44.1.0
$ pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.9.4#egg=ckan[requirements]'
$ pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.9.4#egg=ckan[dev-requirements]'

⚠️ During installation I found it was necessary to also install pymagic (as per this solution)

$ pip uninstall python-magic
$ pip install python-magic-bin==0.4.14

5. Start postgresql and initialise databases

Helpful primer on using postgres helpers on osx
https://www.codementor.io/@engineerapart/getting-started-with-postgresql-on-mac-osx-are8jcopb

$ brew services start postgresql
# can confirm this is working with
$ psql postgres

# list databases
$$ \l
$$ exit

# create ckan user on postgres
createuser -S -D -R -P ckan_default
# write and store password

# create initial db
$ createdb -O ckan_default ckan_default -E utf-8

Create ckan folders and a config

sudo mkdir -p /etc/ckan/default
sudo chown -R `whoami` /etc/ckan/
ckan generate config /etc/ckan/default/ckan.ini
ln -s ckan/config/who.ini /etc/ckan/default/who.ini

6. Create ckan config and update

$ ckan generate config /etc/ckan/default/ckan.ini

Update ckan.ini to include

sqlalchemy.url = postgresql://ckan_default:{PASSWORD}@localhost/ckan_default

7. Start redis and update storage config

Primer: https://redis.io/topics/quickstart

$ brew services start redis
$ redis-cli ping
# => pong

Update storage settings in ckan.ini

⚠️ I also needed to update asset ownership at this stage

$ sudo chown -R `whoami` /var/lib/ckan
$ sudo chmod -R u+rwx /var/lib/ckan
$ mkdir /var/lib/ckan/default

Then update ckan.ini

## Storage Settings
ckan.storage_path = /var/lib/ckan/default

8. Setup solr

a. start services

$ brew services start solr

http://localhost:8983/ => this should now point at solr

c. add CKAN Core to solr and populate solr config

Add core at http://localhost:8983/solr/#/~cores

name: ckan
instanceDir: ckan

⚠️ This will probably shout about missing files, which you can update below and try again

d. update schema for new ckan core:

Copy ckan/ckan/config/solr/schema.xml to /usr/local/var/lib/solr/ckan/schema.xml

The default ckan schema is out of date for solr 8, so update:

# change 
<defaultSearchField>text</defaultSearchField>
<solrQueryParser defaultOperator="AND"/>

# to
<df>text</df>
<solrQueryParser q.op="AND"/>

e. gap fill ckan solr config

Copy protwords, stopwords etc to ckan core in solr:

https://github.com/ckan/ckan/blob/master/contrib/docker/solr/Dockerfile#L11-L19

by the end of this you should have something like:

$ ls /usr/local/var/lib/solr/ckan

conf            data            solrconfig.xml
core.properties protwords.txt   stopwords.txt
currency.xml    schema.xml.bak  synonyms.txt

f. update ckan.ini to reflect new ckan core

With the Ckan data created in solr, /etc/ckan/default/ckan.ini needs updating to point at the new location

## Search Settings
ckan.site_id = default
solr_url = http://127.0.0.1:8983/solr/ckan

9. Initialise and Start ckan

# initialise DB
$ ckan -c /etc/ckan/default/ckan.ini db init
$ CKAN_INI=/etc/ckan/default/ckan.ini ckan paster db init

# start CKAN
$ ckan -c /etc/ckan/default/ckan.ini run

Visit http://localhost:5000/