Merge develop with 2.2.0 back to master

CHANGELOG.md

@@ -1,7 +1,30 @@
-# Changelog for the ViUR Server (https://github.com/viur-framework/server)
+# Changelog
 
+This file documents any relevant changes done to ViUR server since version 2.
+
+## [2.2.0 Etna] - 2018-04-23
+
+### Added
+- A new Rate-Limit module that can be used to prevent DoS / Brute-Force on certain resources.
+- New keyBone() class introduced for a better distinction between bones containing an entity key and those which don't.
+
+### Changed
+- Return descriptions in selectBones translated from our json render
+- More descriptive error messages returned from password bone
+- The new "params" Parameter introduced in 2.1.0 should now always have a default value (None)
+- *[Breaking]*: Merged *selectOneBone* and *selectMultiBone* into a single *selectBone* which supports the multiple flag,
+  the previous bone names are still supported and cause a deprecation warning in the logs.
+
+### Fixed
+- Resolving the name of skeletons in prototypes/hierarchy when used on multi-lang bones
+- Unserializing of booleanBones failed if the database contained a unicode string
+- Several errors caused by None returned from getSearchDocumentFields / getSearchTags
+- Exception causing ``toDB()`` to fail if a bone should be serialized which isn't in the valuesCache object. Fixes #7.
+- Prevent text/string Bones from restoring old (non-multilang data) if set back to empty values. Fixes #6.
+- Use urlsafe_b64decode for filename decoding (used by some old IE on WinXP). Fixes #38.
+- Prevent skeleton from the server to take precedence over the ones defined in the application.
+- The indexed-flag had been ignored on edit - causing all fields to be indexed. Fixes #34.
 
-## [Unreleased]
 
 ## [2.1.0] - 2017-10-25
 
@@ -77,7 +100,7 @@
 ## [2.0.1] - 2017-01-04
 
 ### Added
- - IDs of entries tranferred with dbtransfer are now marked as in-use inside the datastore
+ - IDs of entries transferred with dbtransfer are now marked as in-use inside the datastore
  - conf["viur.cacheEnvironmentKey"] can now raise RuntimeError to indicate that this request cannot be cached
  - Added the render attribute to BasicApplications
 

README.md

@@ -1,33 +1,34 @@
 # ViUR server
 
-**ViUR** is a free software development framework for the [Google App Engine](https://appengine.google.com).
+This is the main component of **ViUR**, a free software development framework for the [Google App Engine](https://appengine.google.com).
 
 ## About
 
 ViUR provides a clear concept for implementing agile data management software that runs on top of the [Google App Engine](https://appengine.google.com) platform. It is written in the [Python programming language](https://python.org) for a maximum flexibility and easy-to-manage code. Its underlying database is Googles Cloud Datastore, which is a scalable document database.
 
 The ViUR framework targets to the implementation of information systems, which are integrated, web-based applications or services performing data management and deployment operations of any kind. Therefore, ViUR is explicitly not claimed to be a content management system, although content management can be a part of a ViUR information system.
 
-## Quick Start
+## Getting started
 
-To quickly setup a running ViUR web-application from scratch, follow these
-simple steps:
+Visit [viur.is](https://www.viur.is/) to get latest news and documentation on ViUR and how it is used.
 
-1. Install the prerequisites Python and [Google Cloud SDK for Python](https://cloud.google.com/sdk) for your operating system.
+To quickly setup a running ViUR web-application from scratch, follow these simple steps:
+
+1. Install the prerequisites [Python 2.7](https://www.python.org/downloads/) and [Google Cloud SDK for Python](https://cloud.google.com/sdk) for your operating system.
 2. Create an empty folder for your project.
-3. Download setup.py from https://www.viur.is/package/download/setup/latest and save it into the newly created folder.
-4. Run ``setup.py`` from that folder - it will do the rest for you!
-5. Locally run the development server with ``dev_appserver.py -A <your-project-key> <project-dir>``
+3. [Download setup.py](https://www.viur.is/package/download/setup/latest) and save it into the newly created folder.
+4. Run ``python2 setup.py`` from that folder - it will do the rest for you!
+5. Locally start the development server with ``dev_appserver.py -A <your-project-key> <project-dir>``
    or deploy it to the world with ``gcloud app deploy --project <your-project-key> <your-project-dir>``.
 
-All quick start steps as shell command snippet:
+All quick start steps on a bash:
 
-```sh
-mkdir hello-viur                                                      # Setup project folder
-cd hello-viur                                                         # Change into this folder
-wget -qO setup.py https://www.viur.is/package/download/setup/latest   # Download latest setup
-python setup.py                                                       # Run ViUR setup tool
-dev_appserver.py -A hello-viur .                                      # Start Google App Engine
+```bash
+$ mkdir hello-viur                                                      # Setup project folder
+$ cd hello-viur                                                         # Change into this folder
+$ wget -qO setup.py https://www.viur.is/package/download/setup/latest   # Download latest setup
+$ python setup.py                                                       # Run ViUR setup tool
+$ dev_appserver.py -A hello-viur .                                      # Start Google App Engine
 ```
 
 Visit the official [ViUR online documentation](https://docs.viur.is/latest) for more information.
@@ -36,9 +37,9 @@ Visit the official [ViUR online documentation](https://docs.viur.is/latest) for
 
 We take a great interest in your opinion about ViUR. We appreciate your feedback and are looking forward to hear about your ideas. Share your visions or questions with us and participate in ongoing discussions.
 
-- [ViUR on the web](https://www.viur.is)
+- [ViUR website](https://www.viur.is)
 - [#ViUR on freenode IRC](https://webchat.freenode.net/?channels=viur)
-- [ViUR on Google Community](https://plus.google.com/communities/102034046048891029088)
+- [ViUR on GitHub](https://github.com/viur-framework)
 - [ViUR on Twitter](https://twitter.com/weloveViUR)
 
 ## Credits
@@ -49,7 +50,7 @@ Help of any kind to extend and improve or enhance this project in any kind or wa
 
 ## License
 
-ViUR is Copyright (C) 2012-2017 by Mausbrand Informationssysteme GmbH.
+Copyright (C) 2012-2018 by Mausbrand Informationssysteme GmbH.
 
 Mausbrand and ViUR are registered trademarks of Mausbrand Informationssysteme GmbH.
 

__init__.py

@@ -16,7 +16,7 @@
    I N F O R M A T I O N    S Y S T E M
 
  ViUR® SERVER
- Copyright 2012-2017 by mausbrand Informationssysteme GmbH
+ Copyright 2012-2018 by mausbrand Informationssysteme GmbH
 
  ViUR® is a free software development framework for the Google App Engine™.
  More about ViUR can be found at http://www.viur.is/.
@@ -25,7 +25,7 @@
  See file LICENSE for more information.
 """
 
-__version__ = (2, 1, 0)  # Which API do we expose to our application
+__version__ = (2, 2, 0)  # Which API do we expose to our application
 
 import sys, traceback, os, inspect
 

bones/__init__.py

@@ -1,13 +1,13 @@
 # -*- coding: utf-8 -*-
 
 from server.bones.bone import baseBone, boneFactory
+from server.bones.keyBone import keyBone
 from server.bones.stringBone import stringBone
 from server.bones.textBone import textBone
 from server.bones.numericBone import numericBone
 from server.bones.colorBone import colorBone
 from server.bones.credentialBone import credentialBone
-from server.bones.selectOneBone import selectOneBone
-from server.bones.selectMultiBone import selectMultiBone, selectAccessMultiBone
+from server.bones.selectBone import selectBone, selectOneBone, selectMultiBone, selectAccessBone
 from server.bones.booleanBone import booleanBone
 from server.bones.relationalBone import relationalBone
 from server.bones.treeItemBone import treeItemBone

bones/bone.py

@@ -88,7 +88,7 @@ def __init__(	self, descr="", defaultValue=None, required=False, params=None, mu
 			:type visible: bool
 
 			.. NOTE::
-				The kwarg 'multiple' is not supported by all bones (fe. selectOneBone)
+				The kwarg 'multiple' is not supported by all bones
 
 		"""
 		from server.skeleton import _boneCounter
@@ -179,7 +179,7 @@ def serialize( self, valuesCache, name, entity ):
 			:type name: String
 			:returns: dict
 		"""
-		if name != "key":
+		if name in valuesCache:
 			entity.set( name, valuesCache[name], self.indexed )
 		return( entity )
 

bones/booleanBone.py

@@ -4,7 +4,7 @@
 
 class booleanBone( baseBone ):
 	type = "bool"
-	trueStrs = [ str(True), "1", "yes" ]
+	trueStrs = [unicode(True), u"1", u"yes"]
 
 	@staticmethod
 	def generageSearchWidget(target,name="BOOLEAN BONE"):
@@ -72,7 +72,7 @@ def unserialize(self, valuesCache, name, expando):
 		"""
 		if name in expando:
 			val = expando[ name ]
-			if str( val ) in self.trueStrs:
+			if unicode(val) in self.trueStrs:
 				valuesCache[name] = True
 			else:
 				valuesCache[name] = False

bones/keyBone.py

@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+from server.bones.bone import baseBone
+from server.utils import normalizeKey
+
+class keyBone(baseBone):
+	kind = "key"
+
+	def __init__(self, descr="Key", readOnly=True, visible=False, **kwargs):
+		super(keyBone, self).__init__(descr=descr, readOnly=readOnly, visible=visible, **kwargs)
+
+
+	def refresh(self, valuesCache, boneName, skel):
+		"""
+			Refresh all values we might have cached from other entities.
+		"""
+		if boneName in valuesCache and valuesCache[boneName]:
+			valuesCache[boneName] = normalizeKey(valuesCache[boneName])
+

bones/numericBone.py

@@ -72,6 +72,8 @@ def fromClient( self, valuesCache, name, data ):
 
 
 	def serialize( self, valuesCache, name, entity ):
+		if not name in valuesCache:
+			return entity
 		if isinstance( valuesCache[name],  float ) and valuesCache[name]!= valuesCache[name]: # NaN
 			entity.set( name, None, self.indexed )
 		else:
@@ -112,3 +114,4 @@ def buildDBFilter(self, name, skel, dbFilter, rawFilter, prefix=None):
 	def getSearchDocumentFields(self, valuesCache, name, prefix = ""):
 		if isinstance(valuesCache.get(name), int) or isinstance(valuesCache.get(name), float):
 			return [search.NumberField(name=prefix + name, value=valuesCache[name])]
+		return []

bones/passwordBone.py

@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+from server import utils
 from server.bones import stringBone
 from hashlib import sha256
 import hmac
@@ -60,23 +61,40 @@ class passwordBone( stringBone ):
 	def isInvalid(self, value):
 		if not value:
 			return False
-		if len(value)<self.minPasswordLength:
-			return "Password to short"
-		# Run our passwort test suite
+
+		if len(value) < self.minPasswordLength:
+			return _("The entered password is to short - it requires at least {{length}} characters.",
+			            length=self.minPasswordLength)
+
+		# Run our password test suite
 		testResults = []
 		for test in self.passwordTests:
 			testResults.append(test(value))
-		if sum(testResults)<self.passwordTestThreshold:
-			return("Your password isn't strong enough!")
+
+		if sum(testResults) < self.passwordTestThreshold:
+			return _("The entered password is too weak.")
+
 		return False
 
+	def fromClient( self, valuesCache, name, data ):
+		value = data.get(name)
+		if not value:
+			return "No value entered"
+
+		err = self.isInvalid(value)
+		if err:
+			return err
+
+		valuesCache[name] = value
+
 	def serialize( self, valuesCache, name, entity ):
 		if valuesCache.get(name,None) and valuesCache[name] != "":
-			salt = ''.join( [ random.choice(string.ascii_lowercase + string.ascii_uppercase + string.digits) for x in range(self.saltLength) ] )
+			salt = utils.generateRandomString(self.saltLength)
 			passwd = pbkdf2( valuesCache[name][ : conf["viur.maxPasswordLength"] ], salt )
 			entity.set( name, passwd, self.indexed )
 			entity.set( "%s_salt" % name, salt, self.indexed )
-		return( entity )
+
+		return entity
 
 	def unserialize( self, valuesCache, name, values ):
-		return( {name: ""} )
+		return {name: ""}

bones/relationalBone.py

@@ -241,10 +241,10 @@ def serialize(self, valuesCache, name, entity ):
 				if self.indexed:
 					if refData:
 						for k, v in refData.items():
-							entity[ "%s.dest.%s" % (name,k) ] = v
+							entity.set("%s.dest.%s" % (name,k), v, True)
 					if usingData:
 						for k, v in usingData.items():
-							entity[ "%s.rel.%s" % (name,k) ] = v
+							entity.set("%s.rel.%s" % (name,k), v, True)
 					#for k, v in valuesCache[name].items():
 					#	if (k in self.refKeys or any( [ k.startswith("%s." %x) for x in self.refKeys ] ) ):
 					#		entity[ "%s.%s" % (name,k) ] = v