CHANGES.txt

.. default-role:: literal

==========================
 Changelog for processing
==========================

Changes in 0.52
---------------

* On versions 0.50 and 0.51 Mac OSX `Lock.release()` would fail with
  `OSError(errno.ENOSYS, "[Errno 78] Function not implemented")`.
  This appears to be because on Mac OSX `sem_getvalue()` has not been
  implemented.

  Now `sem_getvalue()` is no longer needed.  Unfortunately, however,
  on Mac OSX `BoundedSemaphore()` will not raise `ValueError` if it
  exceeds its initial value.

* Some changes to the code for the reduction/rebuilding of connection
  and socket objects so that things work the same on Windows and Unix.
  This should fix a couple of bugs.

* The code has been changed to consistently use "camelCase" for
  methods and (non-factory) functions.  In the few cases where this
  has meant a change to the documented API, the old name has been
  retained as an alias.


Changes in 0.51
---------------

* In 0.50 `processing.Value()` and `processing.sharedctypes.Value()`
  were related but had different signatures, which was rather
  confusing.

  Now `processing.sharedctypes.Value()` has been renamed 
  `processing.sharedctypes.RawValue()` and
  `processing.sharedctypes.Value()` is the same as `processing.Value()`.

* In version 0.50 `sendfd()` and `recvfd()` apparently did not work on
  64bit Linux.  This has been fixed by reverting to using the CMSG_*
  macros as was done in 0.40.  

  However, this means that systems without all the necessary CMSG_*
  macros (such as Solaris 8) will have to disable compilation of
  `sendfd()` and `recvfd()` by setting `macros['HAVE_FD_TRANSFRER'] = 0` 
  in `setup.py`.

* Fixed an authentication error when using a "remote" manager created
  using `BaseManager.from_address()`.

* Fixed a couple of bugs which only affected Python 2.4.


Changes in 0.50
---------------

* `ctypes` is now a prerequisite if you want to use shared memory --
  with Python 2.4 you will need to install it separately.

* `LocalManager()` has been removed.

* Added `processing.Value()` and `processing.Array()`
  which are similar to `LocalManager.SharedValue()` and
  `LocalManager.SharedArray()`.  

* In the `sharedctypes` module `new_value()` and `new_array()` have
  been renamed `Value()` and `Array()`.

* `Process.stop()`, `Process.getStoppable()` and
  `Process.setStoppable()` have been removed.  Use
  `Process.terminate()` instead.

* `procesing.Lock` now matches `threading.Lock` behaviour more
  closely: now a thread can release a lock it does not own, and now
  when a thread tries acquiring a lock it already owns a deadlock
  results instead of an exception.

* On Windows when the main thread is blocking on a method of `Lock`,
  `RLock`, `Semaphore`, `BoundedSemaphore`, `Condition` it will no
  longer ignore Ctrl-C.  (The same was already true on Unix.)  

  This differs from the behaviour of the equivalent objects in
  `threading` which will completely ignore Ctrl-C.

* The `test` sub-package has been replaced by lots of unit tests in a
  `tests` sub-package.  Some of the old test files have been moved
  over to a new `examples` sub-package.

* On Windows it is now possible for a non-console python program
  (i.e. one using `pythonw.exe` instead of `python.exe`) to use
  `processing`.  

  Previously an exception was raised when `subprocess.py` tried to
  duplicate stdin, stdout, stderr.

* Proxy objects should now be thread safe -- they now use thread local
  storage.

* Trying to transfer shared resources such as locks, queues etc
  between processes over a pipe or queue will now raise `RuntimeError`
  with a message saying that the object should only be shared between
  processes using inheritance.

  Previously, this worked unreliably on Windows but would fail with an
  unexplained `AssertionError` on Unix.

* The names of some of the macros used for compiling the extension
  have changed.  See `INSTALL.txt` and `setup.py`.

* A few changes which (hopefully) make compilation possible on Solaris.

* Lots of refactoring of the code.

* Fixed reference leaks so that unit tests pass with "regrtest -R::"
  (at least on Linux).


Changes in 0.40
---------------

* Removed `SimpleQueue` and `PosixQueue` types.  Just use `Queue` instead.

* Previously if you forgot to use the ::

      if __name__ == '__main__':
          freezeSupport()
          ...

  idiom on Windows then processes could be created recursively
  bringing the computer to its knees.  Now `RuntimeError` will be
  raised instead.

* Some refactoring of the code.

* A Unix specific bug meant that a child process might fail to start a
  feeder thread for a queue if its parent process had already started
  its own feeder thread.  Fixed.


Changes in 0.39
---------------

* One can now create one-way pipes by doing 
  `reader, writer = Pipe(duplex=False)`.

* Rewrote code for managing shared memory maps.

* Added a `sharedctypes` module for creating `ctypes` objects allocated
  from shared memory.  On Python 2.4 this requires the installation of
  `ctypes`.

  `ctypes` objects are not protected by any locks so you will need to
  synchronize access to them (such as by using a lock).  However they
  can be much faster to access than equivalent objects allocated using
  a `LocalManager`.

* Rearranged documentation.

* Previously the C extension caused a segfault on 64 bit machines with
  Python 2.5 because it used `int` instead of `Py_ssize_t` in certain
  places.  This is now fixed.  Thanks to Alexy Khrabrov for the report.

* A fix for `Pool.terminate()`.

* A fix for cleanup behaviour of `Queue`.


Changes in 0.38
---------------

* Have revamped the queue types.  Now the queue types are
  `Queue`, `SimpleQueue` and (on systems which support it)
  `PosixQueue`.

  Now `Queue` should behave just like Python's normal `Queue.Queue`
  class except that `qsize()`, `task_done()` and `join()` are not
  implemented.  In particular, if no maximum size was specified when
  the queue was created then `put()` will always succeed without
  blocking.

  A `SimpleQueue` instance is really just a pipe protected by a couple
  of locks.  It has `get()`, `put()` and `empty()` methods but does
  not not support timeouts or non-blocking.

  `BufferedPipeQueue()` and `PipeQueue()` remain as deprecated
  aliases of `Queue()` but `BufferedPosixQueue()` has been removed.
  (Not sure if we really need to keep `PosixQueue()`...)

* Previously the `Pool.shutdown()` method was a little dodgy -- it
  could block indefinitely if `map()` or `imap*()` were used and did
  not try to terminate workers while they were doing a task.

  Now there are three new methods `close()`, `terminate()` and
  `join()` -- `shutdown()` is retained as a deprecated alias of
  `terminate()`.  Thanks to Gerald John M. Manipon for feature
  request/suggested patch to `shutdown()`.

* `Pool.imap()` and `Pool.imap_unordered()` has gained a `chunksize`
  argument which allows the iterable to be submitted to the pool in
  chunks.  Choosing `chunksize` appropriately makes `Pool.imap()`
  almost as fast as `Pool.map()` even for long iterables and cheap
  functions.

* Previously on Windows when the cleanup code for a `LocalManager`
  attempts to unlink the name of the file which backs the shared
  memory map an exception is raised if a child process still exists
  which has a handle open for that mmap.  This is likely to happen if
  a daemon process inherits a `LocalManager` instance.

  Now the parent process will remember the filename and attempt to
  unlink the file name again once all the child processes have been
  joined or terminated.  Reported by Paul Rudin.

* `types.MethodType` is registered with `copy_reg` so now instance
  methods and class methods should be picklable.  (Unfortunately there is
  no obvious way of supporting the pickling of staticmethods since
  they are not marked with the class in which they were defined.)

  This means that on Windows it is now possible to use an instance
  method or class method as the target callable of a Process object.

* On Windows `reduction.fromfd()` now returns true instances of
  `_socket.socket`, so there is no more need for the
  `_processing.falsesocket` type.


Changes in 0.37
---------------

* Updated metadata and documentation because the project is now hosted
  at `developer.berlios.de/projects/pyprocessing`.

* The `Pool.join()` method has been removed.  `Pool.shutdown()` will
  now join the worker processes automatically.

* A pool object no longer participates in a reference cycle so
  `Pool.shutdown()` should get called as soon as its reference count
  falls to zero.
  
* On Windows if `enableLogging()` was used at module scope then the
  logger used by a child process would often get two copies of the
  same handler.  To fix this, now specifiying a handler type in
  `enableLogging()` will cause any previous handlers used by the
  logger to be discarded.


Changes in 0.36
---------------

* In recent versions on Unix the finalizers in a manager process were
  never given a chance to run before `os._exit()` was called, so old
  unlinked AF_UNIX sockets could accumulate in '/tmp'.  Fixed.

* The shutting down of managers has been cleaned up.

* In previous versions on Windows trying to acquire a lock owned by a
  different thread of the current process would raise an exception.
  Fixed.

* In previous versions on Windows trying to use an event object for
  synchronization between two threads of the same process was likely
  to raise an exception.  (This was caused by the bug described
  above.)  Fixed.

* Previously the arguments to `processing.Semaphore()` and
  `processing.BoundedSemaphore()` did not have any defaults.  The
  defaults should be 1 to match `threading`.  Fixed.

* It should now be possible for a Windows Service created by using
  `pywin32` to spawn processes using the `processing` package.

  Note that `pywin32` apparently has a bug meaning that `Py_Finalize()` 
  is never called when the service exits so functions registered with
  `atexit` never get a chance to run.  Therefore it is advisable to
  explicitly call `sys.exitfunc()` or `atexit._run_exitfuncs()` at the
  end of `ServiceFramework.DoSvcRun()`.  Otherwise child processes are
  liable to survive the service when it is stopped.  Thanks to Charlie
  Hull for the report.

* Added `getLogger()` and `enableLogging()` to support logging.


Changes in 0.35
---------------

* By default processes are no longer be stoppable using the `stop()`
  method: one must call `setStoppable(True)` before `start()` in order
  to use the `stop()` method.  (Note that `terminate()` will work
  regardless of whether the process is marked as being "stoppable".)

  The reason for this is that on Windows getting `stop()` to work
  involves starting a new console for the child process and installing
  a signal handler for the `SIGBREAK` signal.  This unfortunately
  means that Ctrl-Break cannot not be used to kill all processes of
  the program.

* Added `setStoppable()` and `getStoppable()` methods -- see above.

* Added `BufferedQueue`/`BufferedPipeQueue`/`BufferedPosixQueue`.
  Putting an object on a buffered queue will always succeed without
  blocking (just like with `Queue.Queue` if no maximum size is
  specified).  This makes them potentially safer than the normal queue
  types provided by `processing` which have finite capacity and may
  cause deadlocks if they fill.

  `test/test_worker.py` has been updated to use `BufferedQueue` for
  the task queue instead of explicitly spawning a thread to feed tasks
  to the queue without risking a deadlock.

* Now when the NO_SEM_TIMED macro is set polling will be used to get
  around the lack of `sem_timedwait()`.  This means that
  `Condition.wait()` and `Queue.get()` should now work with timeouts
  on Mac OS X.

* Added a `callback` argument to `Pool.apply_async()`.

* Added `test/test_httpserverpool.py` which runs a pool of http
  servers which share a single listening socket.

* Previously on Windows the process object was passed to the child
  process on the commandline (after pickling and hex encoding it).
  This caused errors when the pickled string was too large.  Now if
  the pickled string is large then it will be passed to the child
  over a pipe or socket.

* Fixed bug in the iterator returned by `Pool.imap()`.

* Fixed bug in `Condition.__repr__()`.

* Fixed a handle/file descriptor leak when sockets or connections are
  unpickled.


Changes in 0.34
---------------

* Although version 0.33 the C extension would compile on Mac OSX
  trying to import it failed with "undefined symbol: _sem_timedwait".
  Unfortunately the `ImportError` exception was silently swallowed.
  
  This is now fixed by using the `NO_SEM_TIMED` macro.  Unfortunately
  this means that some methods like `Condition.wait()` and
  `Queue.get()` will not work with timeouts on Mac OS X.  If you
  really need to be able to use timeouts then you can always use the
  equivalent objects created with a manager.  Thanks to Doug Hellmann
  for report and testing.
  
* Added a `terminate()` method to process objects which is more
  forceful than `stop()`.

* Fixed bug in the cleanup function registered with `atexit` which on
  Windows could cause a process which is shutting down to deadlock
  waiting for a manager to exit.  Thanks to Dominique Wahli for report
  and testing.
  
* Added `test/test_workers.py` which gives an example of how to create
  a collection of worker processes which execute tasks from one queue
  and return results on another.

* Added `processing.Pool()` which returns a process pool object.  This
  allows one to execute functions asynchronously.  It also has a
  parallel implementation of the `map()` builtin.  This is still
  *experimental* and undocumented --- see `test/test_pool.py` for
  example usage.


Changes in 0.33
---------------

* Added a `recvbytes_into()` method for receiving byte data into
  objects with the writable buffer interface.  Also renamed the
  `_recv_string()` and `_send_string()` methods of connection objects
  to `recvbytes()` and `sendbytes()`.

* Some optimizations for the transferring of large blocks of data
  using connection objects.

* On Unix `os.sysconf()` is now used by default to determine whether
  to compile in support for posix semaphores or posix message queues.
  
  By using the `NO_SEM_TIMED` and `NO_MQ_TIMED` macros (see
  `INSTALL.txt`) it should now also be possible to compile in
  (partial) semaphore or queue support on Unix systems which lack the
  timeout functions `sem_timedwait()` or `mq_timedreceive()` and
  `mq_timesend()`.

* `gettimeofday()` is now used instead of `clock_gettime()` making
  compilation of the C extension (hopefully) possible on Mac OSX.  No
  modificaton of `setup.py` should be necessary.  Thanks to Michele
  Bertoldi for report and proposed patch.

* `cpuCount()` function added which returns the number of CPUs
  in the system.

* Bugfixes to `PosixQueue` class.


Changes in 0.32
---------------

* Refactored and simplified `_nonforking` module -- info about
  `sys.modules` of parent process is no longer passed on to child
  process.  Also `pkgutil` is no longer used.

* Allocated space from an mmap used by `LocalManager` will now be
  recycled.

* Better tests for `LocalManager`.

* Fixed bug in `managers.py` concerning refcounting of shared objects.
  Bug affects the case where the callable used to create a shared
  object does not return a unique object each time it is called.
  Thanks to Alexey Akimov for the report.

* Added a `freezeSupport()` function. Calling this at the appropriate
  point in the main module is necessary when freezing a multiprocess
  program to produce a Windows executable.  (Has been tested with
  `py2exe`, `PyInstaller` and `cx_Freeze`.)


Changes in 0.31
---------------

* Fixed one line bug in `localmanager.py` which caused shared memory maps
  not to be resized properly.

* Added tests for shared values/structs/arrays to `test/test_processing`.


Changes in 0.30
----------------

* Process objects now support the complete API of thread objects.

  In particular `isAlive()`, `isDaemon()`, `setDaemon()` have been
  added and `join()` now supports the `timeout` paramater.

  There are also new methods `stop()`, `getPid()` and `getExitCode()`.
  
* Implemented synchronization primitives based on the Windows mutexes
  and semaphores and posix named semaphores.  

* Added support for sharing simple objects between processes by using
  a shared memory map and the `struct` or `array` modules.

* An `activeChildren()` function has been added to `processing` which
  returns a list of the child processes which are still alive.

* A `Pipe()` function has been added which returns a pair of
  connection objects representing the ends of a duplex connection over
  which picklable objects can be sent.

* socket objects etc are now picklable and can be transferred between
  processes.  (Requires compilation of the `_processing` extension.)

* Subclasses of `managers.BaseManager` no longer automatically spawn a
  child process when an instance is created: the `start()` method must be
  called explicitly.

* On Windows child processes are now spawned using `subprocess`.

* On Windows the Python 2.5 version of `pkgutil` is now used for
  loading modules by the `_nonforking` module.  On Python 2.4 this
  version of `pkgutil` (which uses the standard Python licence) is
  included in `processing.compat`.

* The arguments to the functions in `processing.connection` have
  changed slightly.

* Connection objects now have a `poll()` method which tests whether
  there is any data available for reading.

* The `test/py2exedemo` folder shows how to get `py2exe` to create a
  Windows executable from a program using the `processing` package.

* More tests.

* Bugfixes.

* Rearrangement of various stuff.


Changes in 0.21
---------------

* By default a proxy is now only able to access those methods of its
  referent which have been explicitly exposed.

* The `connection` sub-package now supports digest authentication.

* Process objects are now given randomly generated 'inheritable'
  authentication keys.

* A manager process will now only accept connections from processes
  using the same authentication key.

* Previously `get_module()` from `_nonforking.py` was seriously messed
  up (though it generally worked).  It is a lot saner now.

* Python 2.4 or higher is now required.


Changes in 0.20
---------------

* The `doc` folder contains HTML documentation.

* `test` is now a subpackage.  Running `processing.test.main()` 
  will run test scripts using both processes and threads.

* `nonforking.py` has been renamed `_nonforking.py`.
  `manager.py` has been renamed `manager.py`.
  `connection.py` has become a sub-package `connection`

* `Listener` and `Client` have been removed from
  `processing`, but still exist in `processing.connection`.

* The package is now *probably* compatible with versions of Python
  earlier than 2.4.

* `set` is no longer a type supported by the default manager type.

* Many more changes.


Changes in 0.12
---------------

* Fixed bug where the arguments to `processing.Manager()` were passed on
  to `processing.manager.DefaultManager()` in the wrong order.

* `processing.dummy` is now a subpackage of `processing`
  instead of a module.

* Rearranged package so that the `test` folder, `README.txt` and
  `CHANGES.txt` are copied when the package is installed.


Changes in 0.11
---------------

* Fixed bug on windows when the full path of `nonforking.py` contains a
  space.

* On unix there is no longer a need to make the arguments to the
  constructor of `Process` be picklable or for and instance of a
  subclass of `Process` to be picklable when you call the start method.

* On unix proxies which a child process inherits from its parent can
  be used by the child without any problem, so there is no longer a
  need to pass them as arguments to `Process`.  (This will never be
  possible on windows.)