Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade all code to be Python3 only #5065

Merged
merged 45 commits into from
Jan 21, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
23b20cd
Do not run py27 on tox
humitos Jan 3, 2019
678027c
Do not run py27 on Travis
humitos Jan 3, 2019
f7dea72
pyupgrade 1.11.0 on all code
humitos Jan 3, 2019
f707466
Use unicode string for Sphinx config file
humitos Jan 3, 2019
1b58939
Remove unused imports to avoid lint issues
humitos Jan 3, 2019
271884a
Run tests with python3.6 on Travis
humitos Jan 3, 2019
849abb3
Do not include search tests when running migrations env
humitos Jan 3, 2019
c78187b
isort all the files
humitos Jan 8, 2019
c56b289
Remove six and future from requirements
humitos Jan 8, 2019
979f216
Manually remove usage of `future` module
humitos Jan 8, 2019
f124be5
Remove skipif
humitos Jan 8, 2019
05f5309
Remove usage of `from django.utils import six`
humitos Jan 8, 2019
685e547
Use ipaddress from standard library
humitos Jan 8, 2019
796d09b
Remove six leafovers
humitos Jan 8, 2019
c2a662c
Bug fix on types checking
humitos Jan 8, 2019
bc3afc7
Merge branch 'master' into humitos/deprecate-python2
humitos Jan 8, 2019
686c820
Import missing modules removed by mistake
humitos Jan 8, 2019
25eb8c4
Keep __future__ in conf.py because we build with Py2 also
humitos Jan 8, 2019
507d8a0
Merge branch 'humitos/deprecate-python2' of github.com:rtfd/readthedo…
humitos Jan 8, 2019
06b07cc
Make our generated conf.py compatible with Py2 and Py3
humitos Jan 8, 2019
e066c0d
Import missing modules
humitos Jan 8, 2019
d606c25
Upgrade common submodule
humitos Jan 8, 2019
3219f7a
Run pre-commit on all the files
humitos Jan 8, 2019
5b88c07
Add docstring to config.__init__ file
humitos Jan 8, 2019
0c535fe
More linting issues fixed
humitos Jan 9, 2019
f182059
pre-commit run on some missing files (manually applyed)
humitos Jan 9, 2019
47e5e7e
Update common pre-commit configs
humitos Jan 9, 2019
efa731f
Upgdate common submodule
humitos Jan 9, 2019
632e18c
pre-commit all the files again
humitos Jan 9, 2019
d8dacde
Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos…
humitos Jan 21, 2019
0f7b8d7
pyupgrade over new code with conflicts
humitos Jan 21, 2019
1ffc03e
isort new files
humitos Jan 21, 2019
e363f6f
Remove future leafovers
humitos Jan 21, 2019
5798557
isort over migration files
humitos Jan 21, 2019
55e1a2f
isort on all tests files
humitos Jan 21, 2019
4aded8b
Remove skipif for six.PY2
humitos Jan 21, 2019
447e3b7
Update common submodule with the proper PR config
humitos Jan 21, 2019
b5bc9c7
pre-commit all the files
humitos Jan 21, 2019
7d08eed
Run add-trailing-comma over all the files
humitos Jan 21, 2019
022594f
Do not touch conf.py for testing
humitos Jan 21, 2019
91a205e
Last pre-commit pass
humitos Jan 21, 2019
9f11db5
Lint changes
humitos Jan 21, 2019
ed67c54
noqa on one line
humitos Jan 21, 2019
6092d0e
Remove py36 from travis.matrix.include
humitos Jan 21, 2019
f68d84a
Move ES env vars to py36 job
humitos Jan 21, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 3 additions & 4 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
language: python
python:
- 2.7
- 3.6
env:
- ES_VERSION=1.3.9 ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
matrix:
include:
- python: 3.6
env: TOXENV=py36 ES_VERSION=1.3.9 ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
- python: 3.6
env: TOXENV=docs
- python: 3.6
Expand Down Expand Up @@ -45,6 +44,6 @@ notifications:

branches:
only:
- master
- master
- rel # Community release branch
- relcorp # Corporate release branch
2 changes: 1 addition & 1 deletion common
3 changes: 2 additions & 1 deletion readthedocs/__init__.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# -*- coding: utf-8 -*-

"""Read the Docs."""

import os.path

from future.moves.configparser import RawConfigParser
from configparser import RawConfigParser


def get_version(setupcfg_path):
Expand Down
6 changes: 4 additions & 2 deletions readthedocs/analytics/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
"""App init"""
# -*- coding: utf-8 -*-

default_app_config = 'readthedocs.analytics.apps.AnalyticsAppConfig' # noqa
"""App init."""

default_app_config = 'readthedocs.analytics.apps.AnalyticsAppConfig' # noqa
5 changes: 3 additions & 2 deletions readthedocs/analytics/apps.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# -*- coding: utf-8 -*-

"""Django app config for the analytics app."""

from __future__ import absolute_import
from django.apps import AppConfig


class AnalyticsAppConfig(AppConfig):

"""Analytics app init code"""
"""Analytics app init code."""

name = 'readthedocs.analytics'
verbose_name = 'Analytics'
37 changes: 20 additions & 17 deletions readthedocs/analytics/tasks.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Tasks for Read the Docs' analytics"""
# -*- coding: utf-8 -*-

from __future__ import absolute_import
"""Tasks for Read the Docs' analytics."""

from django.conf import settings

Expand All @@ -11,24 +11,24 @@


DEFAULT_PARAMETERS = {
'v': '1', # analytics version (always 1)
'aip': '1', # anonymize IP
'v': '1', # analytics version (always 1)
'aip': '1', # anonymize IP
'tid': settings.GLOBAL_ANALYTICS_CODE,

# User data
'uip': None, # User IP address
'ua': None, # User agent
'uip': None, # User IP address
'ua': None, # User agent

# Application info
'an': 'Read the Docs',
'av': readthedocs.__version__, # App version
'av': readthedocs.__version__, # App version
}


@app.task(queue='web')
def analytics_pageview(url, title=None, **kwargs):
"""
Send a pageview to Google Analytics
Send a pageview to Google Analytics.

:see: https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters
:param url: the URL of the pageview
Expand All @@ -37,18 +37,21 @@ def analytics_pageview(url, title=None, **kwargs):
"""
data = {
't': 'pageview',
'dl': url, # URL of the pageview (required)
'dt': title, # Title of the page
'dl': url, # URL of the pageview (required)
'dt': title, # Title of the page
}
data.update(DEFAULT_PARAMETERS)
data.update(kwargs)
send_to_analytics(data)


@app.task(queue='web')
def analytics_event(event_category, event_action, event_label=None, event_value=None, **kwargs):
def analytics_event(
event_category, event_action, event_label=None, event_value=None,
**kwargs
):
"""
Send an analytics event to Google Analytics
Send an analytics event to Google Analytics.

:see: https://developers.google.com/analytics/devguides/collection/protocol/v1/devguide#event
:param event_category: the category of the event
Expand All @@ -58,11 +61,11 @@ def analytics_event(event_category, event_action, event_label=None, event_value=
:param kwargs: extra event parameters to send to GA
"""
data = {
't': 'event', # GA event - don't change
'ec': event_category, # Event category (required)
'ea': event_action, # Event action (required)
'el': event_label, # Event label
'ev': event_value, # Event value (numeric)
't': 'event', # GA event - don't change
'ec': event_category, # Event category (required)
'ea': event_action, # Event action (required)
'el': event_label, # Event label
'ev': event_value, # Event value (numeric)
}
data.update(DEFAULT_PARAMETERS)
data.update(kwargs)
Expand Down
4 changes: 1 addition & 3 deletions readthedocs/analytics/tests.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
from __future__ import absolute_import, unicode_literals

# -*- coding: utf-8 -*-
from django.test import TestCase

from .utils import anonymize_ip_address, anonymize_user_agent
Expand Down Expand Up @@ -29,4 +28,3 @@ def test_anonymize_ua(self):
anonymize_user_agent('Some rare user agent'),
'Rare user agent',
)

33 changes: 15 additions & 18 deletions readthedocs/analytics/utils.py
Original file line number Diff line number Diff line change
@@ -1,26 +1,23 @@
"""Utilities related to analytics"""
# -*- coding: utf-8 -*-

"""Utilities related to analytics."""

from __future__ import absolute_import, unicode_literals
import hashlib
import ipaddress
import logging

import requests
from django.conf import settings
from django.utils.encoding import force_text, force_bytes
from django.utils.crypto import get_random_string
import requests
from django.utils.encoding import force_bytes, force_text
from user_agents import parse

try:
# Python 3.3+ only
import ipaddress
except ImportError:
from .vendor import ipaddress

log = logging.getLogger(__name__) # noqa
log = logging.getLogger(__name__) # noqa


def get_client_ip(request):
"""Gets the real IP based on a request object"""
"""Gets the real IP based on a request object."""
ip_address = request.META.get('REMOTE_ADDR')

# Get the original IP address (eg. "X-Forwarded-For: client, proxy1, proxy2")
Expand All @@ -32,7 +29,7 @@ def get_client_ip(request):


def anonymize_ip_address(ip_address):
"""Anonymizes an IP address by zeroing the last 2 bytes"""
"""Anonymizes an IP address by zeroing the last 2 bytes."""
# Used to anonymize an IP by zero-ing out the last 2 bytes
ip_mask = int('0xFFFFFFFFFFFFFFFFFFFFFFFFFFFF0000', 16)

Expand All @@ -46,7 +43,7 @@ def anonymize_ip_address(ip_address):


def anonymize_user_agent(user_agent):
"""Anonymizes rare user agents"""
"""Anonymizes rare user agents."""
# If the browser family is not recognized, this is a rare user agent
parsed_ua = parse(user_agent)
if parsed_ua.browser.family == 'Other' or parsed_ua.os.family == 'Other':
Expand All @@ -56,7 +53,7 @@ def anonymize_user_agent(user_agent):


def send_to_analytics(data):
"""Sends data to Google Analytics"""
"""Sends data to Google Analytics."""
if data.get('uip') and data.get('ua'):
data['cid'] = generate_client_id(data['uip'], data['ua'])

Expand All @@ -74,7 +71,7 @@ def send_to_analytics(data):
resp = requests.post(
'https://www.google-analytics.com/collect',
data=data,
timeout=3, # seconds
timeout=3, # seconds
)
except requests.Timeout:
log.warning('Timeout sending to Google Analytics')
Expand All @@ -85,10 +82,10 @@ def send_to_analytics(data):

def generate_client_id(ip_address, user_agent):
"""
Create an advertising ID
Create an advertising ID.

This simplifies things but essentially if a user has the same IP and same UA,
this will treat them as the same user for analytics purposes
This simplifies things but essentially if a user has the same IP and same
UA, this will treat them as the same user for analytics purposes
"""
salt = b'advertising-client-id'

Expand Down
Empty file.
Loading