Package providing simple Python access to data in:
- Google Analytics
- Google AdManager (GAM earlier DoubleClick for Publishers, DFP)
- Google sheets
- Google BigQuery
- MOAT
- Qubole
- Rubicon
- AWS Athena
- AWS s3
- MySQL
- neo4j
Sroka library was checked to work for Python >=3.8, <=3.11.
Install requirements and enable custom githooks:
pip install -r requirements.txt
git config --local core.hooksPath .githooks/
Check style with flake8:
flake8 .
Please target Pull Requests against dev branch.
pip install sroka
pip install git+ssh://git@github.com/Wikia/sroka
in home folder create ~/.sroka_config (hidden folder) file where you will store:
config.inifile based onconfig.sample.iniwith information to access Qubole, MOAT, Athena, S3 and Rubiconclient_secrets.jsonfor GA accessad_manager.jsonfor GAM accesscredentials.jsonfor Google sheets accessbigquery_credentials.jsonfor BigQuery access
Alternatively, you may set localization of your files during analysis:
from sroka.config.config import setup_env_variables
from sroka.config.config import setup_client_secret
from sroka.config.config import setup_admanager_config
from sroka.config.config import setup_bigquery_config
from sroka.config.config import setup_google_sheets_credentials
setup_env_variables('/file_path/config.ini')
setup_client_secret('/file_path/client_secrets.json')
setup_admanager_config('/file_path/ad_manager.json')
setup_bigquery_config('/file_path/bigquery_credentials.json')
setup_google_sheets_credentials('/file_path/credentials.json')- Use this wizard to create or select a project in the Google Developers Console and automatically turn on the API. Click Continue, then Go to credentials.
- On the Add credentials to your project page, click the Cancel button.
- At the top of the page, select the OAuth consent screen tab. Select an Email address, enter a Product name if not already set, and click the Save button.
- Select the Credentials tab, click the Create credentials button and select OAuth client ID.
- Select the application type Other, enter the chosen name, and click the Create button.
- Click OK to dismiss the resulting dialog.
- Click the file_download (Download JSON) button to the right of the client ID.
- Follow these instructions
- while adding a service account note that the role needs to have necessary viewing and reporting permissions.
You should end up with .json (!) file with credentials
- Make sure the Name in "OAuth 2.0 client IDs" matches the service account in "Service account keys": here
- Create GAM account as service account not a new user: https://support.google.com/admanager/answer/6078734?hl=en
- Once you have a service account, it can be used to access data in different networks. Simply add it as a new service account through GAM UI of the second network.
- Additional information can be specified in
config.inifile:
- network code - a default value that can be overwritten in a function call
- application name - custom name of your network, if not specified, a generic value will be passed.
In order to authorize in Google Sheets you need to generate credentials in Google Console:
You should end up with credentials.json file that should be downloaded to ~/.sroka_config folder.
Go to link and follow up instructions
within Setting up authentication section. You should end up with bigquery_credentials.json
json file that should be downloaded to ~/.sroka_config folder.
- Find your Qubole API Token (go to user -> My Profile -> my_account -> API Token -> show)
- Copy your Qubole API Token to
config.inifile
- You should have your aws_access_key_id and aws_secret_access_key from registration process in AWS console.
- s3bucket_name can be found in AWS console in Athena view when you click
Settings, there you haveQuery result location. The name of location withouts3://and/is what you need. - For Athena usage you need to set also region (AWS regional endpoint), e.g.
'us-east-1'
- You should have your id, username and password from Rubicon
- Copy values to
config.inifile in relevant fields
- In order to connect to a remote MySQL server, you need to provide the
hostandportvalues in the configuration. If it is accessible through a unix socket, you need to provide the path to this socket instead in theunix_socketconfiguration field. - If the MySQL server is protected by user credentials, you need to provide the
userandpasswordvalues in the configuration. - You can optionally specify the database to which you want to connect in the
databaseconfiguration field.
If you see an error like ValueError: unknown locale: UTF-8
Please add to ~/.bash_profile lines like this:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
-
If
PyYAMLpackage is not building correctly, it may be caused by the fact that newer versions of pip won’t uninstall the package because it’s handled by disutils. Please installPyYAMLpackage first with--ignore-installedflag. -
If numpy gets messed up during sroka installation it is probably caused by multiple versions installed. Please uninstall all using pip uninstall and then reinstall latest one.
If you encounter RefreshError similar to
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n "error": "invalid_grant",\n "error_description": "Bad Request"\n}')
, try removing all files from ~/.cache directory.
All people that contributed to sroka development before going opensource (including CR and QA):