deap-0.7.1/ 0000755 0000765 0000024 00000000000 11650301263 012657 5 ustar felix staff 0000000 0000000 deap-0.7.1/CHANGELOG.txt 0000644 0000765 0000024 00000001320 11641072614 014710 0 ustar felix staff 0000000 0000000 2011-06-01 Reorganized modules in a more permanent API.
2011-05-23 Modified structure of the project so that deap include both dtm and
eap.
2011-04-02 Added support for numpy.ndarray in the creator.
2011-03-20 Renamed the nsga2 and spea2 selections to selNSGA2 and selSPEA2.
2011-03-19 Moved all the operators from the toolbox to a new operators module.
Also, this new module contains the other operators that are the
History, Hall of Fame, Stats and Milestone.
2011-03-19 Changed version number in eap/__init__.py file.
2011-03-17 Tagged revision 0.7-a1.
2011-03-17 Added the Hillis coevolution example.
2011-03-17 Added the CHANGELOG file to log every change in DEAP from now. deap-0.7.1/deap/ 0000755 0000765 0000024 00000000000 11650301263 013570 5 ustar felix staff 0000000 0000000 deap-0.7.1/deap/__init__.py 0000644 0000765 0000024 00000006134 11650277637 015726 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""
DEAP (Distributed Evolutionary Algorithms in Python) is a novel
evolutionary computation framework for rapid prototyping and testing of
ideas. Its design departs from most other existing frameworks in that it
seeks to make algorithms explicit and data structures transparent, as
opposed to the more common black box type of frameworks. It also
incorporates easy parallelism where users need not concern themselves with
gory implementation details like synchronization and load balancing, only
functional decomposition.
The five founding hypotheses of DEAP are:
(1) The user knows best. Users should be able to understand the internal
mechanisms of the framework so that they can extend them easily to better
suit their specific needs.
(2) User needs in terms of algorithms and operators are so vast that it would
be unrealistic to think of implementing them all in a single framework.
However, it should be possible to build basic tools and generic mechanisms
that enable easy user implementation of most any EA variant.
(3) Speedy prototyping of ideas is often more precious than speedy execution
of programs. Moreover, code compactness and clarity is also very precious.
(4) Even though interpreted, Python is fast enough to execute EAs. Whenever
execution time becomes critical, compute intensive components can always
be recoded in C. Many efficient numerical libraries are already available
through Python APIs.
(5) Easy parallelism can alleviate slow execution.
And these hypotheses lead to the following objectives:
**Rapid prototyping**
Provide an environment allowing users to quickly implement their own
algorithms without compromise.
**Parallelization made easy**
Allow for straightforward parallelization; users should not be forced to
specify more than the granularity level of their functional decomposition.
**Adaptive load balancing**
The workload should automatically and dynamically be distributed among
available compute units; user intervention should be optional and limited
to hints of relative loads of tasks.
**Preach by examples**
Although the aim of the framework is not to provide ready made solutions,
it should nevertheless come with a substantial set of real-world examples
to guide the apprenticeship of users.
"""
__author__ = "Francois-Michel De Rainville and Felix-Antoine Fortin"
__version__ = "0.7"
__revision__ = "0.7.1"
deap-0.7.1/deap/algorithms.py 0000644 0000765 0000024 00000047151 11641072614 016330 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""The :mod:`algorithms` module is intended to contain some specific algorithms
in order to execute very common evolutionary algorithms. The method used here
are more for convenience than reference as the implementation of every
evolutionary algorithm may vary infinitely. Most of the algorithms in this module
use operators registered in the toolbox. Generaly, the keyword used are
:meth:`mate` for crossover, :meth:`mutate` for mutation, :meth:`~deap.select`
for selection and :meth:`evaluate` for evaluation.
You are encouraged to write your own algorithms in order to make them do what
you really want them to do.
"""
import logging
import random
_logger = logging.getLogger("deap.algorithms")
def varSimple(toolbox, population, cxpb, mutpb):
"""Part of the :func:`~deap.algorithmes.eaSimple` algorithm applying only
the variation part (crossover followed by mutation). The modified
individuals have their fitness invalidated. The individuals are not cloned
so there can be twice a reference to the same individual.
This function expects :meth:`toolbox.mate` and :meth:`toolbox.mutate`
aliases to be registered in the toolbox.
"""
# Apply crossover and mutation on the offspring
for ind1, ind2 in zip(population[::2], population[1::2]):
if random.random() < cxpb:
toolbox.mate(ind1, ind2)
del ind1.fitness.values, ind2.fitness.values
for ind in population:
if random.random() < mutpb:
toolbox.mutate(ind)
del ind.fitness.values
return population
def varAnd(toolbox, population, cxpb, mutpb):
"""Part of an evolutionary algorithm applying only the variation part
(crossover **and** mutation). The modified individuals have their
fitness invalidated. The individuals are cloned so returned population is
independent of the input population.
The variator goes as follow. First, the parental population
:math:`P_\mathrm{p}` is duplicated using the :meth:`toolbox.clone` method
and the result is put into the offspring population :math:`P_\mathrm{o}`.
A first loop over :math:`P_\mathrm{o}` is executed to mate consecutive
individuals. According to the crossover probability *cxpb*, the
individuals :math:`\mathbf{x}_i` and :math:`\mathbf{x}_{i+1}` are mated
using the :meth:`toolbox.mate` method. The resulting children
:math:`\mathbf{y}_i` and :math:`\mathbf{y}_{i+1}` replace their respective
parents in :math:`P_\mathrm{o}`. A second loop over the resulting
:math:`P_\mathrm{o}` is executed to mutate every individual with a
probability *mutpb*. When an individual is mutated it replaces its not
mutated version in :math:`P_\mathrm{o}`. The resulting
:math:`P_\mathrm{o}` is returned.
This variation is named *And* beceause of its propention to apply both
crossover and mutation on the individuals. Both probabilities should be in
:math:`[0, 1]`.
"""
offspring = [toolbox.clone(ind) for ind in population]
# Apply crossover and mutation on the offspring
for ind1, ind2 in zip(offspring[::2], offspring[1::2]):
if random.random() < cxpb:
toolbox.mate(ind1, ind2)
del ind1.fitness.values, ind2.fitness.values
for ind in offspring:
if random.random() < mutpb:
toolbox.mutate(ind)
del ind.fitness.values
return offspring
def eaSimple(toolbox, population, cxpb, mutpb, ngen, stats=None, halloffame=None):
"""This algorithm reproduce the simplest evolutionary algorithm as
presented in chapter 7 of Back, Fogel and Michalewicz,
"Evolutionary Computation 1 : Basic Algorithms and Operators", 2000.
It uses :math:`\lambda = \kappa = \mu` and goes as follow.
It first initializes the population (:math:`P(0)`) by evaluating
every individual presenting an invalid fitness. Then, it enters the
evolution loop that begins by the selection of the :math:`P(g+1)`
population. Then the crossover operator is applied on a proportion of
:math:`P(g+1)` according to the *cxpb* probability, the resulting and the
untouched individuals are placed in :math:`P'(g+1)`. Thereafter, a
proportion of :math:`P'(g+1)`, determined by *mutpb*, is
mutated and placed in :math:`P''(g+1)`, the untouched individuals are
transferred :math:`P''(g+1)`. Finally, those new individuals are evaluated
and the evolution loop continues until *ngen* generations are completed.
Briefly, the operators are applied in the following order ::
evaluate(population)
for i in range(ngen):
offspring = select(population)
offspring = mate(offspring)
offspring = mutate(offspring)
evaluate(offspring)
population = offspring
This function expects :meth:`toolbox.mate`, :meth:`toolbox.mutate`,
:meth:`toolbox.select` and :meth:`toolbox.evaluate` aliases to be
registered in the toolbox.
"""
_logger.info("Start of evolution")
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in population if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
if halloffame is not None:
halloffame.update(population)
if stats is not None:
stats.update(population)
# Begin the generational process
for gen in range(ngen):
_logger.info("Evolving generation %i", gen)
# Select and clone the next generation individuals
offsprings = toolbox.select(population, len(population))
offsprings = map(toolbox.clone, offsprings)
# Variate the pool of individuals
offsprings = varSimple(toolbox, offsprings, cxpb, mutpb)
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in offsprings if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
# Update the hall of fame with the generated individuals
if halloffame is not None:
halloffame.update(offsprings)
_logger.debug("Evaluated %i individuals", len(invalid_ind))
# Replace the current population by the offsprings
population[:] = offsprings
# Update the statistics with the new population
if stats is not None:
stats.update(population)
# Log statistics on the current generation
if stats is not None:
print stats
_logger.info("End of (successful) evolution")
return population
def varOr(toolbox, population, lambda_, cxpb, mutpb):
"""Part of an evolutionary algorithm applying only the variation part
(crossover, mutation **or** reproduction). The modified individuals have
their fitness invalidated. The individuals are cloned so returned
population is independent of the input population.
The variator goes as follow. On each of the *lambda_* iteration, it
selects one of the three operations; crossover, mutation or reproduction.
In the case of a crossover, two individuals are selected at random from
the parental population :math:`P_\mathrm{p}`, those individuals are cloned
using the :meth:`toolbox.clone` method and then mated using the
:meth:`toolbox.mate` method. Only the first child is appended to the
offspring population :math:`P_\mathrm{o}`, the second child is discarded.
In the case of a mutation, one individual is selected at random from
:math:`P_\mathrm{p}`, it is cloned and then mutated using using the
:meth:`toolbox.mutate` method. The resulting mutant is appended to
:math:`P_\mathrm{o}`. In the case of a reproduction, one individual is
selected at random from :math:`P_\mathrm{p}`, cloned and appended to
:math:`P_\mathrm{o}`.
This variation is named *Or* beceause an offspring will never result from
both operations crossover and mutation. The sum of both probabilities
shall be in :math:`[0, 1]`, the reproduction probability is
1 - *cxpb* - *mutpb*.
"""
assert (cxpb + mutpb) <= 1.0, ("The sum of the crossover and mutation "
"probabilities must be smaller or equal to 1.0.")
offsprings = []
for _ in xrange(lambda_):
op_choice = random.random()
if op_choice < cxpb: # Apply crossover
ind1, ind2 = [toolbox.clone(ind) for ind in random.sample(population, 2)]
toolbox.mate(ind1, ind2)
del ind1.fitness.values
offsprings.append(ind1)
elif op_choice < cxpb + mutpb: # Apply mutation
ind = toolbox.clone(random.choice(population))
toolbox.mutate(ind)
del ind.fitness.values
offsprings.append(ind)
else: # Apply reproduction
offsprings.append(random.choice(population))
return offsprings
def varLambda(toolbox, population, lambda_, cxpb, mutpb):
"""Part of the :func:`~deap.algorithms.eaMuPlusLambda` and
:func:`~deap.algorithms.eaMuCommaLambda` algorithms that produce the
lambda new individuals. The modified individuals have their fitness
invalidated. The individuals are not cloned so there can be twice a
reference to the same individual.
This function expects :meth:`toolbox.mate` and :meth:`toolbox.mutate`
aliases to be registered in the toolbox.
"""
assert (cxpb + mutpb) <= 1.0, ("The sum of the crossover and mutation "
"probabilities must be smaller or equal to 1.0.")
offsprings = []
nb_offsprings = 0
while nb_offsprings < lambda_:
op_choice = random.random()
if op_choice < cxpb: # Apply crossover
ind1, ind2 = random.sample(population, 2)
ind1 = toolbox.clone(ind1)
ind2 = toolbox.clone(ind2)
toolbox.mate(ind1, ind2)
del ind1.fitness.values, ind2.fitness.values
offsprings.append(ind1)
offsprings.append(ind2)
nb_offsprings += 2
elif op_choice < cxpb + mutpb: # Apply mutation
ind = random.choice(population) # select
ind = toolbox.clone(ind) # clone
toolbox.mutate(ind)
del ind.fitness.values
offsprings.append(ind)
nb_offsprings += 1
else: # Apply reproduction
offsprings.append(random.choice(population))
nb_offsprings += 1
# Remove the exedant of offsprings
if nb_offsprings > lambda_:
del offsprings[lambda_:]
return offsprings
def eaMuPlusLambda(toolbox, population, mu, lambda_, cxpb, mutpb, ngen, stats=None, halloffame=None):
"""This is the :math:`(\mu + \lambda)` evolutionary algorithm. First,
the individuals having an invalid fitness are evaluated. Then, the
evolutionary loop begins by producing *lambda* offspring from the
population, the offspring are generated by a crossover, a mutation or a
reproduction proportionally to the probabilities *cxpb*, *mutpb* and
1 - (cxpb + mutpb). The offspring are then evaluated and the next
generation population is selected from both the offspring **and** the
population. Briefly, the operators are applied as following ::
evaluate(population)
for i in range(ngen):
offspring = generate_offspring(population)
evaluate(offspring)
population = select(population + offspring)
This function expects :meth:`toolbox.mate`, :meth:`toolbox.mutate`,
:meth:`toolbox.select` and :meth:`toolbox.evaluate` aliases to be
registered in the toolbox.
.. note::
Both produced individuals from a crossover are put in the offspring
pool.
"""
_logger.info("Start of evolution")
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in population if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
_logger.debug("Evaluated %i individuals", len(invalid_ind))
if halloffame is not None:
halloffame.update(population)
if stats is not None:
stats.update(population)
# Begin the generational process
for gen in range(ngen):
_logger.info("Evolving generation %i", gen)
# Variate the population
offsprings = varLambda(toolbox, population, lambda_, cxpb, mutpb)
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in offsprings if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
_logger.debug("Evaluated %i individuals", len(invalid_ind))
# Update the hall of fame with the generated individuals
if halloffame is not None:
halloffame.update(offsprings)
# Select the next generation population
population[:] = toolbox.select(population + offsprings, mu)
# Update the statistics with the new population
if stats is not None:
stats.update(population)
# Log statistics on the current generation
if stats is not None:
_logger.debug(stats)
_logger.info("End of (successful) evolution")
return population
def eaMuCommaLambda(toolbox, population, mu, lambda_, cxpb, mutpb, ngen, stats=None, halloffame=None):
"""This is the :math:`(\mu~,~\lambda)` evolutionary algorithm. First,
the individuals having an invalid fitness are evaluated. Then, the
evolutionary loop begins by producing *lambda* offspring from the
population, the offspring are generated by a crossover, a mutation or a
reproduction proportionally to the probabilities *cxpb*, *mutpb* and
1 - (cxpb + mutpb). The offspring are then evaluated and the next
generation population is selected **only** from the offspring. Briefly,
the operators are applied as following ::
evaluate(population)
for i in range(ngen):
offspring = generate_offspring(population)
evaluate(offspring)
population = select(offspring)
This function expects :meth:`toolbox.mate`, :meth:`toolbox.mutate`,
:meth:`toolbox.select` and :meth:`toolbox.evaluate` aliases to be
registered in the toolbox.
.. note::
Both produced individuals from the crossover are put in the offspring
pool.
"""
assert lambda_ >= mu, "lambda must be greater or equal to mu."
_logger.info("Start of evolution")
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in population if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
_logger.debug("Evaluated %i individuals", len(invalid_ind))
if halloffame is not None:
halloffame.update(population)
if stats is not None:
stats.update(population)
# Begin the generational process
for gen in range(ngen):
_logger.info("Evolving generation %i", gen)
# Variate the population
offsprings = varLambda(toolbox, population, lambda_, cxpb, mutpb)
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in offsprings if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
_logger.debug("Evaluated %i individuals", len(invalid_ind))
# Update the hall of fame with the generated individuals
if halloffame is not None:
halloffame.update(offsprings)
# Select the next generation population
population[:] = toolbox.select(offsprings, mu)
# Update the statistics with the new population
if stats is not None:
stats.update(population)
# Log statistics on the current generation
if stats is not None:
_logger.debug(stats)
_logger.info("End of (successful) evolution")
return population
def varSteadyState(toolbox, population):
"""Part of the :func:`~deap.algorithms.eaSteadyState` algorithm
that produce the new individual by crossover of two randomly selected
parents and mutation on one randomly selected child. The modified
individual has its fitness invalidated. The individuals are not cloned so
there can be twice a reference to the same individual.
This function expects :meth:`toolbox.mate`, :meth:`toolbox.mutate` and
:meth:`toolbox.select` aliases to be
registered in the toolbox.
"""
# Select two individuals for crossover
p1, p2 = random.sample(population, 2)
p1 = toolbox.clone(p1)
p2 = toolbox.clone(p2)
toolbox.mate(p1, p2)
# Randomly choose amongst the offsprings the returned child and mutate it
child = random.choice((p1, p2))
toolbox.mutate(child)
return child,
def eaSteadyState(toolbox, population, ngen, stats=None, halloffame=None):
"""The steady-state evolutionary algorithm. Every generation, a single new
individual is produced and put in the population producing a population of
size :math:`lambda+1`, then :math:`lambda` individuals are kept according
to the selection operator present in the toolbox.
This function expects :meth:`toolbox.mate`, :meth:`toolbox.mutate`,
:meth:`toolbox.select` and :meth:`toolbox.evaluate` aliases to be
registered in the toolbox.
"""
_logger.info("Start of evolution")
# Evaluate the individuals with an invalid fitness
invalid_ind = [ind for ind in population if not ind.fitness.valid]
fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
for ind, fit in zip(invalid_ind, fitnesses):
ind.fitness.values = fit
if halloffame is not None:
halloffame.update(population)
# Begin the generational process
for gen in range(ngen):
_logger.info("Evolving generation %i", gen)
# Variate the population
child, = varSteadyState(toolbox, population)
# Evaluate the produced child
child.fitness.values = toolbox.evaluate(child)
# Update the hall of fame
if halloffame is not None:
halloffame.update((child,))
# Select the next generation population
population[:] = toolbox.select(population + [child], len(population))
# Update the statistics with the new population
if stats is not None:
stats.update(population)
# Log statistics on the current generation
if stats is not None:
_logger.debug(stats)
_logger.info("End of (successful) evolution")
return population
deap-0.7.1/deap/base.py 0000644 0000765 0000024 00000042065 11641072614 015070 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""The :mod:`~deap.base` module provides basic structures to build evolutionary
algorithms. It contains only two simple containers that are a basic N-ary
:class:`~deap.base.Tree`, usefull for implementing genetic programing, and a
virtual :class:`~deap.base.Fitness` class used as base class, for the fitness
member of any individual.
"""
import copy
import operator
import functools
from collections import deque
from itertools import izip, repeat, count
class Toolbox(object):
"""A toolbox for evolution that contains the evolutionary operators.
At first the toolbox contains two simple methods. The first method
:meth:`~deap.toolbox.clone` duplicates any element it is passed as
argument, this method defaults to the :func:`copy.deepcopy` function.
The second method :meth:`~deap.toolbox.map` applies the function given
as first argument to every items of the iterables given as next
arguments, this method defaults to the :func:`map` function. You may
populate the toolbox with any other function by using the
:meth:`~deap.base.Toolbox.register` method.
"""
def __init__(self):
self.register("clone", copy.deepcopy)
self.register("map", map)
def register(self, alias, method, *args, **kargs):
"""Register a *method* in the toolbox under the name *alias*. You
may provide default arguments that will be passed automatically when
calling the registered method. Fixed arguments can then be overriden
at function call time. The following code block is a example of how
the toolbox is used. ::
>>> def func(a, b, c=3):
... print a, b, c
...
>>> tools = Toolbox()
>>> tools.register("myFunc", func, 2, c=4)
>>> tools.myFunc(3)
2 3 4
"""
pfunc = functools.partial(method, *args, **kargs)
pfunc.__name__ = alias
setattr(self, alias, pfunc)
def unregister(self, alias):
"""Unregister *alias* from the toolbox."""
delattr(self, alias)
def decorate(self, alias, *decorators):
"""Decorate *alias* with the specified *decorators*, *alias*
has to be a registered function in the current toolbox. Decorate uses
the signature preserving decoration function
:func:`~deap.tools.decorate`.
"""
from tools import decorate
partial_func = getattr(self, alias)
method = partial_func.func
args = partial_func.args
kargs = partial_func.keywords
for decorator in decorators:
method = decorate(decorator)(method)
setattr(self, alias, functools.partial(method, *args, **kargs))
class Fitness(object):
"""The fitness is a measure of quality of a solution. If *values* are
provided as a tuple, the fitness is initalized using those values,
otherwise it is empty (or invalid).
Fitnesses may be compared using the ``>``, ``<``, ``>=``, ``<=``, ``==``,
``!=``. The comparison of those operators is made lexicographically.
Maximization and minimization are taken care off by a multiplication
between the :attr:`weights` and the fitness :attr:`values`. The comparison
can be made between fitnesses of different size, if the fitnesses are
equal until the extra elements, the longer fitness will be superior to the
shorter.
.. note::
When comparing fitness values that are **minimized**, ``a > b`` will
return :data:`True` if *a* is **smaller** than *b*.
"""
weights = None
"""The weights are used in the fitness comparison. They are shared among
all fitnesses of the same type. When subclassing ``Fitness``, ``weights``
must be defined as a tuple where each element is associated to an
objective. A negative weight element corresponds to the minimization of the
associated objective and positive weight to the maximization.
.. note::
If weights is not defined during subclassing, the following error will
occur at instantiation of a subclass fitness object:
``TypeError: Can't instantiate abstract with
abstract attribute weights.``
"""
wvalues = ()
"""Contains the weighted values of the fitness, the multiplication with the
weights is made when the values are set via the property :attr:`values`.
Multiplication is made on setting of the values for efficiency.
Generally it is unnecessary to manipulate *wvalues* as it is an internal
attribute of the fitness used in the comparison operators.
"""
def __init__(self, values=()):
if self.weights is None:
raise TypeError("Can't instantiate abstract %r with abstract "
"attribute weights." % (self.__class__))
if len(values) > 0:
self.values = values
def getValues(self):
return tuple(map(operator.div, self.wvalues, self.weights))
def setValues(self, values):
self.wvalues = tuple(map(operator.mul, values, self.weights))
def delValues(self):
self.wvalues = ()
values = property(getValues, setValues, delValues,
("Fitness values. Use directly ``individual.fitness.values = values`` "
"in order to set the fitness and ``del individual.fitness.values`` "
"in order to clear (invalidate) the fitness. The (unweighted) fitness "
"can be directly accessed via ``individual.fitness.values``."))
@property
def valid(self):
"""Asses if a fitness is valid or not."""
return len(self.wvalues) != 0
def isDominated(self, other):
"""In addition to the comparison operators that are used to sort
lexically the fitnesses, this method returns :data:`True` if this
fitness is dominated by the *other* fitness and :data:`False` otherwise.
The weights are used to compare minimizing and maximizing fitnesses. If
there is more fitness values than weights, the last weight get repeated
until the end of the comparison.
"""
not_equal = False
for self_wvalue, other_wvalue in izip(self.wvalues, other.wvalues):
if self_wvalue > other_wvalue:
return False
elif self_wvalue < other_wvalue:
not_equal = True
return not_equal
def __gt__(self, other):
return not self.__le__(other)
def __ge__(self, other):
return not self.__lt__(other)
def __le__(self, other):
if not other: # Protection against yamling
return False
return self.wvalues <= other.wvalues
def __lt__(self, other):
if not other: # Protection against yamling
return False
return self.wvalues < other.wvalues
def __eq__(self, other):
if not other: # Protection against yamling
return False
return self.wvalues == other.wvalues
def __ne__(self, other):
return not self.__eq__(other)
def __deepcopy__(self, memo):
"""Replace the basic deepcopy function with a faster one.
It assumes that the elements in the :attr:`values` tuple are
immutable and the fitness does not contain any other object
than :attr:`values` and :attr:`weights`.
"""
if len(self.wvalues) > 0:
return self.__class__(self.values)
else:
return self.__class__()
def __str__(self):
"""Return the values of the Fitness object."""
return str(self.values)
def __repr__(self):
"""Return the Python code to build a copy of the object."""
module = self.__module__
name = self.__class__.__name__
return "%s.%s(%r)" % (module, name, self.values)
class Tree(list):
"""Basic N-ary tree class. A tree is initialized from the list `content`.
The first element of the list is the root of the tree, then the
following elements are the nodes. Each node can be either a list or a
single element. In the case of a list, it is considered as a subtree,
otherwise a leaf.
"""
class NodeProxy(object):
__slots__ = ['obj']
def __new__(cls, obj, *args, **kargs):
if isinstance(obj, cls):
return obj
inst = object.__new__(cls)
inst.obj = obj
return inst
def getstate(self):
"""Return the state of the NodeProxy: the proxied object."""
return self.obj
@property
def size(self):
"""Return the size of a leaf: 1."""
return 1
@property
def height(self):
"""Return the height of a leaf: 0."""
return 0
@property
def root(self):
"""Return the root of a leaf: itself."""
return self
def __eq__(self, other):
return self.obj == other.obj
def __getattr__(self, attr):
return getattr(self.obj, attr)
def __call__(self, *args, **kargs):
return self.obj(*args, **kargs)
def __repr__(self):
return self.obj.__repr__()
def __str__(self):
return self.obj.__str__()
@classmethod
def convertNode(cls, node):
"""Convert node into the proper object either a Tree or a Node."""
if isinstance(node, cls):
if len(node) == 1:
return cls.NodeProxy(node[0])
return node
elif isinstance(node, list):
if len(node) > 1:
return cls(node)
else:
return cls.NodeProxy(node[0])
else:
return cls.NodeProxy(node)
def __init__(self, content=None):
"""Initialize a tree with a list `content`.
The first element of the list is the root of the tree, then the
following elements are the nodes. A node could be a list, then
representing a subtree.
"""
for elem in content:
self.append(self.convertNode(elem))
def getstate(self):
"""Return the state of the Tree as a list of arbitrary elements.
It is mainly used for pickling a Tree object.
"""
return [elem.getstate() for elem in self]
def __reduce__(self):
"""Return the class init, the object's state and the object's
dict in a tuple. The function is used to pickle Tree.
"""
return (self.__class__, (self.getstate(),), self.__dict__)
def __deepcopy__(self, memo):
"""Deepcopy a Tree by first converting it back to a list of list."""
new = self.__class__(copy.deepcopy(self.getstate()))
new.__dict__.update(copy.deepcopy(self.__dict__, memo))
return new
def __setitem__(self, key, value):
"""Set the item at `key` with the corresponding `value`."""
list.__setitem__(self, key, self.convertNode(value))
def __setslice__(self, i, j, value):
"""Set the slice at `i` to `j` with the corresponding `value`."""
list.__setslice__(self, i, j, self.convertNode(value))
def __str__(self):
"""Return the tree in its original form, a list, as a string."""
return list.__str__(self)
def __repr__(self):
"""Return the Python code to build a copy of the object."""
module = self.__module__
name = self.__class__.__name__
return "%s.%s(%s)" % (module, name, list.__repr__(self))
@property
def root(self):
"""Return the root element of the tree.
The root node of a tree is the node with no parents. There is at most
one root node in a rooted tree.
"""
return self[0]
@property
def size(self):
"""Return the number of nodes in the tree.
The size of a node is the number of descendants it has including itself.
"""
return sum(elem.size for elem in self)
@property
def height(self):
"""Return the height of the tree.
The height of a tree is the length of the path from the root to the
deepest node in the tree. A (rooted) tree with only one node (the root)
has a height of zero.
"""
try:
return max(elem.height for elem in self[1:])+1
except ValueError:
return 0
@property
def iter(self):
"""Return a generator function that iterates on the element
of the tree in linear time using depth first algorithm.
>>> t = Tree([1,2,3[4,5,[6,7]],8])
>>> [i for i in t.iter]:
[1, 2, 3, 4, 5, 6, 7, 8]
"""
for elem in self:
if isinstance(elem, Tree):
for elem2 in elem.iter:
yield elem2
else:
yield elem
@property
def iter_leaf(self):
"""Return a generator function that iterates on the leaf
of the tree in linear time using depth first
algorithm.
>>> t = Tree([1,2,3,[4,5,[6,7]],8])
>>> [i for i in t.iter_leaf]
[2, 3, 5, 7, 8]
"""
for elem in self[1:]:
if isinstance(elem, Tree):
for elem2 in elem.iter_leaf:
yield elem2
else:
yield elem
@property
def iter_leaf_idx(self):
"""Return a generator function that iterates on the leaf
indices of the tree in linear time using depth first
algorithm.
>>> t = Tree([1,2,3,[4,[5,6,7],[8,9]],[10,11]]);
>>> [i for i in t.iter_leaf_idx]
[1, 2, 5, 6, 8, 10]
"""
def leaf_idx(tree, total):
total[0] += 1
for elem in tree[1:]:
if isinstance(elem, Tree):
for elem2 in leaf_idx(elem, total):
yield total[0]
else:
yield total[0]
total[0] += 1
return leaf_idx(self, [0])
def searchSubtreeDF(self, index):
"""Search the subtree with the corresponding index based on a
depth-first search.
"""
if index == 0:
return self
total = 0
for child in self:
if total == index:
return child
nbr_child = child.size
if nbr_child + total > index:
return child.searchSubtreeDF(index-total)
total += nbr_child
def setSubtreeDF(self, index, subtree):
"""Replace the tree with the corresponding index by subtree based
on a depth-first search.
"""
if index == 0:
try:
self[:] = subtree
except TypeError:
del self[1:]
self[0] = subtree
return
total = 0
for i, child in enumerate(self):
if total == index:
self[i] = subtree
return
nbr_child = child.size
if nbr_child + total > index:
child.setSubtreeDF(index-total, subtree)
return
total += nbr_child
def searchSubtreeBF(self, index):
"""Search the subtree with the corresponding index based on a
breadth-first search.
"""
if index == 0:
return self
queue = deque(self[1:])
for i in xrange(index):
subtree = queue.popleft()
if isinstance(subtree, Tree):
queue.extend(subtree[1:])
return subtree
def setSubtreeBF(self, index, subtree):
"""Replace the subtree with the corresponding index by subtree based
on a breadth-first search.
"""
if index == 0:
try:
self[:] = subtree
except TypeError:
del self[1:]
self[0] = subtree
return
queue = deque(izip(repeat(self, len(self[1:])), count(1)))
for i in xrange(index):
elem = queue.popleft()
parent = elem[0]
child = elem[1]
if isinstance(parent[child], Tree):
tree = parent[child]
queue.extend(izip(repeat(tree, len(tree[1:])), count(1)))
parent[child] = subtree
deap-0.7.1/deap/benchmarks/ 0000755 0000765 0000024 00000000000 11650301263 015705 5 ustar felix staff 0000000 0000000 deap-0.7.1/deap/benchmarks/__init__.py 0000644 0000765 0000024 00000021171 11641072614 020025 0 ustar felix staff 0000000 0000000 # This file is part of EAP.
#
# EAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# EAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with EAP. If not, see .
"""
Regroup typical EC benchmarks functions to import easily and benchmark
examples.
"""
import random
from math import sin, cos, pi, exp, e, sqrt
from operator import mul
from functools import reduce
# Unimodal
def rand(individual):
"""Random test objective function.
:math:`f_{\\text{Rand}}(\mathbf{x}) = \\text{\\texttt{random}}(0,1)`
"""
return random.random(),
def plane(individual):
"""Plane test objective function.
:math:`f_{\\text{Plane}}(\mathbf{x}) = x_0`
"""
return individual[0],
def sphere(individual):
"""Sphere test objective function.
:math:`f_{\\text{Sphere}}(\mathbf{x}) = \sum_{i=1}^Nx_i^2`
"""
return sum(gene * gene for gene in individual),
def cigar(individual):
"""Cigar test objective function.
:math:`f_{\\text{Cigar}}(\mathbf{x}) = x_0^2 + 10^6\\sum_{i=1}^N\,x_i^2`
"""
return individual[0]**2 + 1e6 * sum(gene * gene for gene in individual),
def rosenbrock(individual):
"""Rosenbrock test objective function.
:math:`f_{\\text{Rosenbrock}}(\\mathbf{x}) = \\sum_{i=1}^{N-1} (1-x_i)^2 + 100 (x_{i+1} - x_i^2 )^2`
.. plot:: _scripts/rosenbrock.py
:width: 67 %
"""
return sum(100 * (x * x - y)**2 + (1. - x)**2 \
for x, y in zip(individual[:-1], individual[1:])),
def h1(individual):
""" Simple two-dimensional function containing several local maxima,
H1 has a unique maximum value of 2.0 at the point (8.6998, 6.7665).
From : The Merits of a Parallel Genetic Algorithm in Solving Hard
Optimization Problems, A. J. Knoek van Soest and L. J. R. Richard
Casius, J. Biomech. Eng. 125, 141 (2003)
:math:`f_{\\text{H1}}(x_1, x_2) = \\frac{\sin(x_1 - \\frac{x_2}{8})^2 + \
\\sin(x_2 + \\frac{x_1}{8})^2}{\\sqrt{(x_1 - 8.6998)^2 + \
(x_2 - 6.7665)^2} + 1}`
.. plot:: _scripts/h1.py
:width: 67 %
"""
num = (sin(individual[0] - individual[1] / 8))**2 + (sin(individual[1] + individual[0] / 8))**2
denum = ((individual[0] - 8.6998)**2 + (individual[1] - 6.7665)**2)**0.5 + 1
return num / denum,
#
# Multimodal
def ackley(individual):
"""Ackley test objective function.
:math:`f_{\\text{Ackley}}(\\mathbf{x}) = 20 - 20\cdot\exp\left(-0.2\sqrt{\\frac{1}{N} \
\sum_{i=1}^N x_i^2} \\right)\
+ e - \
\exp\left(\\frac{1}{N}\sum_{i=1}^N \\cos(2\pi x_i) \
\\right)`
.. plot:: _scripts/ackley.py
:width: 67 %
"""
N = len(individual)
return 20 - 20 * exp(-0.2*sqrt(1.0/N * sum(x**2 for x in individual))) \
+ e - exp(1.0/N * sum(cos(2*pi*x) for x in individual)),
def bohachevsky(individual):
"""Bohachevsky test objective function
:math:`f_{\\text{Bohachevsky}}(\mathbf{x}) = \sum_{i=1}^{N-1}(x_i^2 + 2x_{i+1}^2 - \
0.3\cos(3\pi x_i) - 0.4\cos(4\pi x_{i+1}) + 0.7)`
.. plot:: _scripts/bohachevsky.py
:width: 67 %
"""
return sum(x**2 + 2*x1**2 - 0.3*cos(3*pi*x) - 0.4*cos(4*pi*x1) + 0.7
for x, x1 in zip(individual[:-1], individual[1:])),
def griewank(individual):
"""Griewank test objective function
:math:`f_{\\text{Griewank}}(\\mathbf{x}) = \\frac{1}{4000}\\sum_{i=1}^N\,x_i^2 - \
\prod_{i=1}^N\\cos\\left(\\frac{x_i}{\sqrt{i}}\\right) + 1`
.. plot:: _scripts/griewank.py
:width: 67 %
"""
return 1.0/4000.0 * sum(x**2 for x in individual) - \
reduce(mul, (cos(x/sqrt(i+1.0)) for i, x in enumerate(individual)), 1) + 1,
def rastrigin(individual):
"""Rastrigin test objective function.
:math:`f_{\\text{Rastrigin}}(\\mathbf{x}) = 10N \sum_{i=1}^N x_i^2 - 10 \\cos(2\\pi x_i)`
.. plot:: _scripts/rastrigin.py
:width: 67 %
"""
return 10 * len(individual) + sum(gene * gene - 10 * \
cos(2 * pi * gene) for gene in individual),
def rastrigin_scaled(individual):
"""Scaled Rastrigin test objective function
:math:`f_{\\text{RastScaled}}(\mathbf{x}) = 10N + \sum_{i=1}^N \
\left(10^{\left(\\frac{i-1}{N-1}\\right)} x_i \\right)^2 x_i)^2 - \
10\cos\\left(2\\pi 10^{\left(\\frac{i-1}{N-1}\\right)} x_i \\right)`
"""
N = len(individual)
return 10*N + sum((10**(i/(N-1))*x)**2 -
10*cos(2*pi*10**(i/(N-1))*x) for i, x in enumerate(individual)),
def rastrigin_skew(individual):
"""Skewed Rastrigin test objective function
:math:`f_{\\text{RastSkew}}(\mathbf{x}) = 10N \sum_{i=1}^N \left(y_i^2 - 10 \\cos(2\\pi x_i)\\right)`
:math:`\\text{with } y_i = \
\\begin{cases} \
10\\cdot x_i & \\text{ if } x_i > 0,\\\ \
x_i & \\text{ otherwise } \
\\end{cases}`
"""
N = len(individual)
return 10*N + sum((10*x if x > 0 else x)**2
- 10*cos(2*pi*(10*x if x > 0 else x)) for x in individual),
def schaffer(individual):
"""Schaffer test objective function.
:math:`f_{\\text{Schaffer}}(\mathbf{x}) = \sum_{i=1}^{N-1} (x_i^2+x_{i+1}^2)^{0.25} \cdot \
\\left[ \sin^2(50\cdot(x_i^2+x_{i+1}^2)^{0.10}) + 1.0 \
\\right]`
"""
return sum((x**2+x1**2)**0.25 * ((sin(50*(x**2+x1**2)**0.1))**2+1.0)
for x, x1 in zip(individual[:-1], individual[1:])),
def schwefel(individual):
"""Schwefel test objective function.
:math:`f_{\\text{Schwefel}}(\mathbf{x}) = 418.9828872724339\cdot N - \
\sum_{i=1}^N\,x_i\sin\\left(\sqrt{|x_i|}\\right)`
.. plot:: _scripts/schwefel.py
:width: 67 %
"""
N = len(individual)
return 418.9828872724339*N-sum(x*sin(sqrt(abs(x))) for x in individual),
def himmelblau(individual):
"""The Himmelblau's function is multimodal with 4 defined minimums in
:math:`[-6, 6]^2`.
:math:`f_{\\text{Himmelblau}}(x_1, x_2) = (x_1^2 + x_2 - 11)^2 + (x_1 + x_2^2 -7)^2`
.. plot:: _scripts/himmelblau.py
:width: 67 %
"""
return (individual[0] * individual[0] + individual[1] - 11)**2 + \
(individual[0] + individual[1] * individual[1] - 7)**2,
def shekel(individual, a, c):
"""The Shekel multimodal function can have any number of maxima. The number
of maxima is given by the length of any of the arguments *a* or *c*, *a*
is a matrix of size :math:`M\\times N`, where *M* is the number of maxima
and *N* the number of dimensions and *c* is a :math:`M\\times 1` vector.
The matrix :math:`\\mathcal{A}` can be seen as the position of the maxima
and the vector :math:`\\mathbf{c}`, the width of the maxima.
:math:`f_\\text{Shekel}(\mathbf{x}) = \\sum_{i = 1}^{M} \\frac{1}{c_{i} +
\\sum_{j = 1}^{N} (x_{j} - a_{ij})^2 }`
The following figure uses
:math:`\\mathcal{A} = \\begin{bmatrix} 0.5 & 0.5 \\\\ 0.25 & 0.25 \\\\
0.25 & 0.75 \\\\ 0.75 & 0.25 \\\\ 0.75 & 0.75 \\end{bmatrix}` and
:math:`\\mathbf{c} = \\begin{bmatrix} 0.002 \\\\ 0.005 \\\\ 0.005
\\\\ 0.005 \\\\ 0.005 \\end{bmatrix}`, thus defining 5 maximums in
:math:`\\mathbb{R}^2`.
.. plot:: _scripts/shekel.py
:width: 67 %
"""
return sum((1. / (c[i] + sum((x - a[i][j])**2 for j, x in enumerate(individual)))) for i in range(len(c))),
# Multiobjectives
def kursawe(individual):
"""Kursawe multiobjective function.
:math:`f_{\\text{Kursawe}1}(\\mathbf{x}) = \\sum_{i=1}^{N-1} -10 e^{-0.2 \\sqrt{x_i^2 + x_{i+1}^2} }`
:math:`f_{\\text{Kursawe}2}(\\mathbf{x}) = \\sum_{i=1}^{N} |x_i|^{0.8} + 5 \\sin(x_i^3)`
.. plot:: _scripts/kursawe.py
:width: 100 %
"""
f1 = sum(-10 * exp(-0.2 * sqrt(x * x + y * y)) for x, y in zip(individual[:-1], individual[1:]))
f2 = sum(abs(x)**0.8 + 5 * sin(x * x * x) for x in individual)
return f1, f2 deap-0.7.1/deap/cma.py 0000644 0000765 0000024 00000031705 11645601567 014726 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
# Special thanks to Nikolaus Hansen for providing major part of
# this code. The CMA-ES algorithm is provided in many other languages
# and advanced versions at http://www.lri.fr/~hansen/cmaesintro.html.
"""A module that provides support for the Covariance Matrix Adaptation
Evolution Strategy.
"""
import logging
from math import sqrt, log, exp
import numpy
import random # Only used to seed numpy.random
import sys # Used to get maxint
import copy
numpy.random.seed(random.randint(0, sys.maxint))
_logger = logging.getLogger("deap.cma")
def esCMA(toolbox, population, ngen, halloffame=None, statistics=None):
"""The CMA-ES algorithm as described in Hansen, N. (2006). *The CMA
Evolution Strategy: A Comparing Rewiew.*
The provided *population* should be a list of one or more individuals.
"""
_logger.info("Start of evolution")
for g in xrange(ngen):
_logger.info("Evolving generation %i", g)
# Evaluate the individuals
for ind in population:
ind.fitness.values = toolbox.evaluate(ind)
if halloffame is not None:
halloffame.update(population)
# Update the Strategy with the evaluated individuals
toolbox.update(population)
if statistics is not None:
statistics.update(population)
_logger.debug(statistics)
_logger.info("End of (successful) evolution")
class CMAStrategy(object):
"""
Additional configuration may be passed through the *params* argument as a
dictionary,
+----------------+---------------------------+----------------------------+
| Parameter | Default | Details |
+================+===========================+============================+
| ``lambda_`` | ``floor(4 + 3 * log(N))`` | Number of children to |
| | | produce at each generation,|
| | | ``N`` is the individual's |
| | | size. |
+----------------+---------------------------+----------------------------+
| ``mu`` | ``floor(lambda_ / 2)`` | The number of parents to |
| | | keep from the |
| | | lambda children. |
+----------------+---------------------------+----------------------------+
| ``weights`` | ``"superlinear"`` | Decrease speed, can be |
| | | ``"superlinear"``, |
| | | ``"linear"`` or |
| | | ``"equal"``. |
+----------------+---------------------------+----------------------------+
| ``cs`` | ``(mueff + 2) / | Cumulation constant for |
| | (N + mueff + 3)`` | step-size. |
+----------------+---------------------------+----------------------------+
| ``damps`` | ``1 + 2 * max(0, sqrt(( | Damping for step-size. |
| | mueff - 1) / (N + 1)) - 1)| |
| | + cs`` | |
+----------------+---------------------------+----------------------------+
| ``ccum`` | ``4 / (N + 4)`` | Cumulation constant for |
| | | covariance matrix. |
+----------------+---------------------------+----------------------------+
| ``ccov1`` | ``2 / ((N + 1.3)^2 + | Learning rate for rank-one |
| | mueff)`` | update. |
+----------------+---------------------------+----------------------------+
| ``ccovmu`` | ``2 * (mueff - 2 + 1 / | Learning rate for rank-mu |
| | mueff) / ((N + 2)^2 + | update. |
| | mueff)`` | |
+----------------+---------------------------+----------------------------+
"""
def __init__(self, centroid, sigma, **kargs):
self.params = kargs
# Create a centroid as a numpy array
self.centroid = numpy.array(centroid)
self.dim = len(self.centroid)
self.sigma = sigma
self.pc = numpy.zeros(self.dim)
self.ps = numpy.zeros(self.dim)
self.chiN = sqrt(self.dim) * (1 - 1. / (4. * self.dim) + \
1. / (21. * self.dim**2))
self.B = numpy.identity(self.dim)
self.C = numpy.identity(self.dim)
self.diagD = numpy.ones(self.dim)
self.BD = self.B * self.diagD
self.cond = 1
self.lambda_ = self.params.get("lambda_", int(4 + 3 * log(self.dim)))
self.update_count = 0
self.computeParams(self.params)
def generate(self, ind_init):
"""Generate a population from the current strategy using the
centroid individual as parent.
"""
arz = numpy.random.standard_normal((self.lambda_, self.dim))
arz = self.centroid + self.sigma * numpy.dot(arz, self.BD.T)
return [ind_init(arzi) for arzi in arz]
def update(self, population):
"""Update the current covariance matrix strategy.
"""
population.sort(key=lambda ind: ind.fitness, reverse=True)
old_centroid = self.centroid
self.centroid = numpy.dot(self.weights, population[0:self.mu])
c_diff = self.centroid - old_centroid
# Cumulation : update evolution path
self.ps = (1 - self.cs) * self.ps \
+ sqrt(self.cs * (2 - self.cs) * self.mueff) / self.sigma \
* numpy.dot(self.B, (1. / self.diagD) \
* numpy.dot(self.B.T, c_diff))
hsig = float((numpy.linalg.norm(self.ps) /
sqrt(1. - (1. - self.cs)**(2. * (self.update_count + 1.))) / self.chiN
< (1.4 + 2. / (self.dim + 1.))))
self.update_count += 1
self.pc = (1 - self.cc) * self.pc + hsig \
* sqrt(self.cc * (2 - self.cc) * self.mueff) / self.sigma \
* c_diff
# Update covariance matrix
artmp = population[0:self.mu] - old_centroid
self.C = (1 - self.ccov1 - self.ccovmu + (1 - hsig) \
* self.ccov1 * self.cc * (2 - self.cc)) * self.C \
+ self.ccov1 * numpy.outer(self.pc, self.pc) \
+ self.ccovmu * numpy.dot((self.weights * artmp.T), artmp) \
/ self.sigma**2
self.sigma *= numpy.exp((numpy.linalg.norm(self.ps) / self.chiN - 1.) \
* self.cs / self.damps)
self.diagD, self.B = numpy.linalg.eigh(self.C)
indx = numpy.argsort(self.diagD)
self.cond = self.diagD[indx[-1]]/self.diagD[indx[0]]
self.diagD = self.diagD[indx]**0.5
self.B = self.B[:, indx]
self.BD = self.B * self.diagD
arz = numpy.random.standard_normal((self.lambda_, self.dim))
arz = self.centroid + self.sigma * numpy.dot(arz, self.BD.T)
for ind, arzi in zip(population, arz):
del ind[:]
ind.extend(arzi)
def computeParams(self, params):
"""Those parameters depends on lambda and need to computed again if it
changes during evolution.
"""
self.mu = params.get("mu", self.lambda_ / 2)
rweights = params.get("weights", "superlinear")
if rweights == "superlinear":
self.weights = log(self.mu + 0.5) - \
numpy.log(numpy.arange(1, self.mu + 1))
elif rweights == "linear":
self.weights = self.mu + 0.5 - numpy.arange(1, self.mu + 1)
elif rweights == "equal":
self.weights = numpy.ones(self.mu)
else:
pass # Print some warning ?
self.weights /= sum(self.weights)
self.mueff = 1. / sum(self.weights**2)
self.cc = params.get("ccum", 4. / (self.dim + 4.))
self.cs = params.get("cs", (self.mueff + 2.) /
(self.dim + self.mueff + 3.))
self.ccov1 = params.get("ccov1", 2. / ((self.dim + 1.3)**2 + \
self.mueff))
self.ccovmu = params.get("ccovmu", 2. * (self.mueff - 2. + \
1. / self.mueff) / \
((self.dim + 2.)**2 + self.mueff))
self.ccovmu = min(1 - self.ccov1, self.ccovmu)
self.damps = 1. + 2. * max(0, sqrt((self.mueff - 1.) / \
(self.dim + 1.)) - 1.) + self.cs
self.damps = params.get("damps", self.damps)
class CMA1pLStrategy(object):
def __init__(self, parent, sigma, **kargs):
self.parent = parent
self.sigma = sigma
self.dim = len(self.parent)
self.C = numpy.identity(self.dim)
self.A = numpy.identity(self.dim)
self.pc = numpy.zeros(self.dim)
self.computeParams(kargs)
self.psucc = self.ptarg
def computeParams(self, params):
# Selection :
self.lambda_ = params.get("lambda_", 1)
# Step size control :
self.d = params.get("d", 1.0 + self.dim/(2.0*self.lambda_))
self.ptarg = params.get("ptarg", 1.0/(5+sqrt(self.lambda_)/2.0))
self.cp = params.get("cp", self.ptarg*self.lambda_/(2+self.ptarg*self.lambda_))
# Covariance matrix adaptation
self.cc = params.get("cc", 2.0/(self.dim+2.0))
self.ccov = params.get("ccov", 2.0/(self.dim**2 + 6.0))
self.pthresh = params.get("pthresh", 0.44)
def generate(self, ind_init):
# self.y = numpy.dot(self.A, numpy.random.standard_normal(self.dim))
arz = numpy.random.standard_normal((self.lambda_, self.dim))
arz = self.parent + self.sigma * numpy.dot(arz, self.A.T)
return [ind_init(arzi) for arzi in arz]
def update(self, population):
population.sort(key=lambda ind: ind.fitness, reverse=True)
lambda_succ = sum(self.parent.fitness <= ind.fitness for ind in population)
p_succ = float(lambda_succ) / self.lambda_
self.psucc = (1-self.cp)*self.psucc + self.cp*p_succ
if self.parent.fitness <= population[0].fitness:
x_step = (population[0] - numpy.array(self.parent)) / self.sigma
self.parent = copy.deepcopy(population[0])
if self.psucc < self.pthresh:
self.pc = (1 - self.cc)*self.pc + sqrt(self.cc * (2 - self.cc)) * x_step
self.C = (1-self.ccov)*self.C + self.ccov * numpy.dot(self.pc, self.pc.T)
else:
self.pc = (1 - self.cc)*self.pc
self.C = (1-self.ccov)*self.C + self.ccov * (numpy.dot(self.pc, self.pc.T) + self.cc*(2-self.cc)*self.C)
self.sigma = self.sigma * exp(1.0/self.d * (self.psucc - self.ptarg)/(1.0-self.ptarg))
# We use Cholesky since for now we have no use of eigen decomposition
# Basically, Cholesky returns a matrix A as C = A*A.T
# Eigen decomposition returns two matrix B and D^2 as C = B*D^2*B.T = B*D*D*B.T
# So A == B*D
# To compute the new individual we need to multiply each vector z by A
# as y = centroid + sigma * A*z
# So the Cholesky is more straightforward as we don't need to compute
# the squareroot of D^2, and multiply B and D in order to get A, we directly get A.
# This can't be done (without cost) with the standard CMA-ES as the eigen decomposition is used
# to compute covariance matrix inverse in the step-size evolutionary path computation.
self.A = numpy.linalg.cholesky(self.C)
arz = numpy.random.standard_normal((self.lambda_, self.dim))
arz = self.parent + self.sigma * numpy.dot(arz, self.A.T)
for ind, arzi in zip(population, arz):
del ind[:]
ind.extend(arzi)
deap-0.7.1/deap/creator.py 0000644 0000765 0000024 00000014757 11641072614 015624 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""The :mod:`~deap.creator` module is the heart and soul of DEAP, it allows to
create, at runtime, classes that will fulfill the needs of your evolutionary
algorithms. This module follows the meta-factory paradigm by allowing to
create new classes via both composition and inheritance. Attributes both datas
and functions are added to existing types in order to create new types
empowered with user specific evolutionary computation capabilities. In effect,
new classes can be built from any imaginable type, from :class:`list` to
:class:`set`, :class:`dict`, :class:`~deap.base.Tree` and more,
providing the possibility to implement genetic algorithms, genetic
programming, evolution strategies, particle swarm optimizers, and many more.
"""
import array
import copy
class_replacers = {}
"""Some classes in Python's standard library as well as third party library
may be in part incompatible with the logic used in DEAP. In order to palliate
to this problem, the method :func:`create` uses the dictionary
`class_replacers` to identify if the base type provided is problematic, and if
so the new class inherits from the replacement class instead of the
original base class.
`class_replacers` keys are classes to be replaced and the values are the
replacing classes.
"""
try:
import numpy
(numpy.ndarray, numpy.array)
except ImportError:
# Numpy is not present, skip the definition of the replacement class.
pass
except AttributeError:
# Numpy is present, but there is either no ndarray or array in numpy,
# also skip the definition of the replacement class.
pass
else:
class _numpy_array(numpy.ndarray):
def __getslice__(self, i, j):
"""Overrides the getslice from numpy.ndarray that returns a shallow
copy of the slice.
"""
return numpy.ndarray.__getslice__(self, i, j).copy()
def __deepcopy__(self, memo):
"""Overrides the deepcopy from numpy.ndarray that does not copy
the object's attributes.
"""
copy_ = numpy.ndarray.__deepcopy__(self, memo)
copy_.__dict__.update(copy.deepcopy(self.__dict__, memo))
return copy_
@staticmethod
def __new__(cls, iterable):
"""Creates a new instance of a numpy.ndarray from a function call"""
return numpy.array(list(iterable)).view(cls)
def __array_finalize__(self, obj):
# __init__ will reinitialize every member of the subclass.
# this might not be desirable for example in the case of an ES.
self.__init__()
# Instead, e could use the following that will simply deepcopy
# every member that is present in the original class
# This is significantly slower.
#if self.__class__ == obj.__class__:
# self.__dict__.update(copy.deepcopy(obj.__dict__))
class_replacers[numpy.ndarray] = _numpy_array
class _array(array.array):
@staticmethod
def __new__(cls, seq=()):
return super(_array, cls).__new__(cls, cls.typecode, seq)
def __deepcopy__(self, memo):
"""Overrides the deepcopy from array.array that does not copy
the object's attributes and class type.
"""
cls = self.__class__
copy_ = cls.__new__(cls, self)
memo[id(self)] = copy_
copy_.__dict__.update(copy.deepcopy(self.__dict__, memo))
return copy_
def __reduce__(self):
return (self.__class__, (list(self),), self.__dict__)
class_replacers[array.array] = _array
def create(name, base, **kargs):
"""Creates a new class named *name* inheriting from *base* in the
:mod:`~deap.creator` module. The new class can have attributes defined by
the subsequent keyword arguments passed to the function create. If the
argument is a class (without the parenthesis), the __init__ function is
called in the initialization of an instance of the new object and the
returned instance is added as an attribute of the class' instance.
Otherwise, if the argument is not a class, (for example an :class:`int`),
it is added as a "static" attribute of the class.
The following is used to create a class :class:`Foo` inheriting from the
standard :class:`list` and having an attribute :attr:`bar` being an empty
dictionary and a static attribute :attr:`spam` initialized to 1. ::
create("Foo", list, bar=dict, spam=1)
This above line is exactly the same as defining in the :mod:`creator`
module something like the following. ::
def Foo(list):
spam = 1
def __init__(self):
self.bar = dict()
"""
dict_inst = {}
dict_cls = {}
for obj_name, obj in kargs.iteritems():
if hasattr(obj, "__call__"):
dict_inst[obj_name] = obj
else:
dict_cls[obj_name] = obj
# Check if the base class has to be replaced
if base in class_replacers:
base = class_replacers[base]
# A DeprecationWarning is raised when the object inherits from the
# class "object" which leave the option of passing arguments, but
# raise a warning stating that it will eventually stop permitting
# this option. Usually this happens when the base class does not
# override the __init__ method from object.
def initType(self, *args, **kargs):
"""Replace the __init__ function of the new type, in order to
add attributes that were defined with **kargs to the instance.
"""
for obj_name, obj in dict_inst.iteritems():
setattr(self, obj_name, obj())
if base.__init__ is not object.__init__:
base.__init__(self, *args, **kargs)
else:
base.__init__(self)
objtype = type(name, (base,), dict_cls)
objtype.__init__ = initType
globals()[name] = objtype
deap-0.7.1/deap/dtm/ 0000755 0000765 0000024 00000000000 11650301263 014354 5 ustar felix staff 0000000 0000000 deap-0.7.1/deap/dtm/__init__.py 0000644 0000765 0000024 00000002426 11641072614 016476 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
# DTM : Distributed Task Manager
# Alpha version
__author__ = "Marc-Andre Gardner"
__version__ = "0.2"
__revision__ = "0.2.1"
from deap.dtm.manager import Control
_dtmObj = Control()
setOptions = _dtmObj.setOptions
start = _dtmObj.start
map = _dtmObj.map
map_async = _dtmObj.map_async
apply = _dtmObj.apply
apply_async = _dtmObj.apply_async
imap = _dtmObj.imap
imap_unordered = _dtmObj.imap_unordered
filter = _dtmObj.filter
repeat = _dtmObj.repeat
waitForAll = _dtmObj.waitForAll
testAllAsync = _dtmObj.testAllAsync
getWorkerId = _dtmObj.getWorkerId
# Control shall not be imported that way
del Control deap-0.7.1/deap/dtm/abstractCommManager.py 0000644 0000765 0000024 00000005340 11641072614 020647 0 ustar felix staff 0000000 0000000 import threading
from abc import ABCMeta, abstractmethod, abstractproperty
class AbstractCommThread(threading.Thread):
__metaclass__ = ABCMeta
def __init__(self, recvQ, sendQ, mainThreadEvent, exitEvent, commReadyEvent, randomGenerator, cmdlineArgs):
threading.Thread.__init__(self)
self.recvQ = recvQ
self.sendQ = sendQ
self.exitStatus = exitEvent
self.msgSendTag = 2
self.wakeUpMainThread = mainThreadEvent
self.random = randomGenerator
self.commReadyEvent = commReadyEvent
self.cmdArgs = cmdlineArgs
self.traceMode = False
self.traceTo = None
@abstractproperty
def poolSize(self):
"""
Return the number of effective workers (for instance, with MPI, this
is the number of slots asked with the -n or -np option)
"""
pass
@abstractproperty
def workerId(self):
"""
Return an ID for this worker such as each worker gets a different ID.
This must be an immutable type (int, string, tuple, etc.)
"""
pass
@abstractproperty
def isRootWorker(self):
"""
Return True if this worker is the "root worker", that is the worker
which will start the main task. The position of this worker in the
hosts is not important, but one and only one worker should be
designated as the root worker (the others should receive False).
"""
pass
@abstractproperty
def isLaunchProcess(self):
"""
If this function returns True, the main thread will wait for the
termination of this thread, and then exit without executing any
task. This may be useful for the backends which have to launch
themselves the remote DTM processes.
"""
pass
@abstractmethod
def setTraceModeOn(self, xmlLogger):
"""
Used for logging purposes. The xmlLogger arg is an xml.etree object
which can be use by the backend to log some informations. The log
format is not currently specified, and the backend may choose to
ignore this call (and not log anything).
"""
pass
@abstractmethod
def iterOverIDs(self):
"""
Return an iterable over all the worker IDs. For instance, if your
workers IDs are integers between 0 and 63, it should then return
range(0,64).
"""
pass
@abstractmethod
def run(self):
"""
The main method, which will be call to start the communication backend.
This is the place where you should import your special modules, and
insert the communication loop.
"""
pass
deap-0.7.1/deap/dtm/commManagerMpi4py.py 0000644 0000765 0000024 00000015214 11641072614 020267 0 ustar felix staff 0000000 0000000 try:
import Queue
except ImportError:
import queue as Queue
import time
import threading
try:
import cPickle
except ImportError:
import pickle as cPickle
import array
import copy
import logging
try:
from lxml import etree
except ImportError:
try:
import xml.etree.cElementTree as etree
except ImportError:
# Python 2.5
import xml.etree.ElementTree as etree
from deap.dtm.dtmTypes import *
from deap.dtm.abstractCommManager import AbstractCommThread
_logger = logging.getLogger("dtm.communication")
DTM_MPI_MIN_LATENCY = 0.005
DTM_MPI_MAX_LATENCY = 0.01
DTM_CONCURRENT_RECV_LIMIT = 1000
DTM_CONCURRENT_SEND_LIMIT = 1000
class CommThread(AbstractCommThread):
def __init__(self, recvQ, sendQ, mainThreadEvent, exitEvent, commReadyEvent, randomGenerator, cmdlineArgs):
AbstractCommThread.__init__(self, recvQ, sendQ, mainThreadEvent, exitEvent, commReadyEvent, randomGenerator, cmdlineArgs)
@property
def poolSize(self):
return self.pSize
@property
def workerId(self):
return self.currentId
@property
def isRootWorker(self):
return self.currentId == 0
@property
def isLaunchProcess(self):
return False
def setTraceModeOn(self, xmlLogger):
self.traceMode = True
self.traceTo = xmlLogger
def iterOverIDs(self):
return range(self.pSize)
def run(self):
from mpi4py import MPI
def mpiSend(msg, dest):
# Pickle and send over MPI
arrayBuf = array.array('b')
arrayBuf.fromstring(cPickle.dumps(msg, cPickle.HIGHEST_PROTOCOL))
b = MPI.COMM_WORLD.Isend([arrayBuf, MPI.CHAR], dest=dest, tag=self.msgSendTag)
if self.traceMode:
etree.SubElement(self.traceTo, "msg", {"direc" : "out", "type" : str(msg.msgType), "otherWorker" : str(dest), "msgtag" : str(self.msgSendTag), "time" : repr(time.time())})
self.msgSendTag += 1
return b, arrayBuf
assert MPI.Is_initialized(), "Error in MPI Init!"
self.pSize = MPI.COMM_WORLD.Get_size()
self.currentId = MPI.COMM_WORLD.Get_rank()
self.commReadyEvent.set() # Notify the main thread that we are ready
if self.currentId == 0 and MPI.Query_thread() > 0:
# Warn only once
_logger.warning("MPI was initialized with a thread level of %i, which is higher than MPI_THREAD_SINGLE."
" The current MPI implementations do not always handle well the MPI_THREAD_MULTIPLE or MPI_THREAD_SERIALIZED modes."
" As DTM was designed to work with the base, safe mode (MPI_THREAD_SINGLE), it is strongly suggested to change"
" the 'thread_level' variable or your mpi4py settings in 'site-packages/mpi4py/rc.py', unless you have strong"
" motivations to keep that setting. This may bring both stability and performance improvements.", MPI.Query_thread())
lRecvWaiting = []
lSendWaiting = []
countSend = 0
countRecv = 0
lMessageStatus = MPI.Status()
working = True
countRecvNotTransmit = 0
countRecvTimeInit = time.time()
while working:
recvSomething = False
sendSomething = False
if self.exitStatus.is_set(): # Exiting
# Warning : the communication thread MUST clear the sendQ
# BEFORE leaving (the exiting orders must be send)
working = False
while len(lRecvWaiting) < DTM_CONCURRENT_RECV_LIMIT and MPI.COMM_WORLD.Iprobe(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG, status=lMessageStatus):
# We received something
lBuf = array.array('b', (0,))
lBuf = lBuf * lMessageStatus.Get_elements(MPI.CHAR)
lRecvWaiting.append((lBuf, MPI.COMM_WORLD.Irecv([lBuf, MPI.CHAR], source=lMessageStatus.Get_source(), tag=lMessageStatus.Get_tag()), lMessageStatus.Get_tag()))
lMessageStatus = MPI.Status()
recvSomething = True
for i, reqTuple in enumerate(lRecvWaiting):
if reqTuple[1].Test():
countRecv += 1
dataS = cPickle.loads(reqTuple[0].tostring())
if self.traceMode:
etree.SubElement(self.traceTo, "msg", {"direc" : "in", "type" : str(dataS.msgType), "otherWorker" : str(dataS.senderWid), "msgtag" : str(reqTuple[2]), "time" : repr(time.time())})
self.recvQ.put(dataS)
lRecvWaiting[i] = None
recvSomething = True
# Wake up the main thread if there's a sufficient number
# of pending receives
countRecvNotTransmit += 1
if countRecvNotTransmit > 50 or (time.time() - countRecvTimeInit > 0.1 and countRecvNotTransmit > 0):
countRecvNotTransmit = 0
countRecvTimeInit = time.time()
self.wakeUpMainThread.set()
lRecvWaiting = filter(lambda d: not d is None, lRecvWaiting)
if not isinstance(lRecvWaiting, list):
lRecvWaiting = list(lRecvWaiting)
while len(lSendWaiting) < DTM_CONCURRENT_SEND_LIMIT:
# Send all pending sends, under the limit of
# DTM_CONCURRENT_SEND_LIMIT
try:
sendMsg = self.sendQ.get_nowait()
countSend += 1
sendMsg.sendTime = time.time()
commA, buf1 = mpiSend(sendMsg, sendMsg.receiverWid)
lSendWaiting.append((commA, buf1))
sendSomething = True
except Queue.Empty:
break
lSendWaiting = filter(lambda d: not d[0].Test(), lSendWaiting)
if not isinstance(lSendWaiting, list): # Python 3
lSendWaiting = list(lSendWaiting)
if not recvSomething:
time.sleep(self.random.uniform(DTM_MPI_MIN_LATENCY, DTM_MPI_MAX_LATENCY))
while len(lSendWaiting) > 0:
# Send the lasts messages before shutdown
lSendWaiting = filter(lambda d: not d[0].Test(), lSendWaiting)
if not isinstance(lSendWaiting, list): # Python 3
lSendWaiting = list(lSendWaiting)
time.sleep(self.random.uniform(DTM_MPI_MIN_LATENCY, DTM_MPI_MAX_LATENCY))
del lSendWaiting
del lRecvWaiting
deap-0.7.1/deap/dtm/commManagerTCP.py 0000644 0000765 0000024 00000033453 11641072614 017540 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
from multiprocessing.connection import Client,Listener
import Queue
import time
import threading
import sys
import os
import socket
import subprocess
from deap.dtm.abstractCommManager import AbstractCommThread
DTM_TCP_MIN_LATENCY = 0.005
DTM_TCP_MAX_LATENCY = 0.01
DTM_CONCURRENT_RECV_LIMIT = 1000
DTM_CONCURRENT_SEND_LIMIT = 1000
BASE_COMM_PORT = 10011
class CommThread(AbstractCommThread):
def __init__(self, recvQ, sendQ, mainThreadEvent, exitEvent, commReadyEvent, randomGenerator, cmdlineArgs):
AbstractCommThread.__init__(self, recvQ, sendQ, mainThreadEvent, exitEvent, commReadyEvent, randomGenerator, cmdlineArgs)
self.props = dict(dtmTCPGlobalLaunch = True,
dtmTCPLocalLaunch = False,
dtmTCPLineNo = -1,
dtmTCPhosts = '',
dtmTCPstartPort = BASE_COMM_PORT,
dtmTCPnPortsByHost = -1,
dtmTCPnbrWorkers = -1,
dtmTCPmaxWorkersByHost = -1,
dtmTCPisRootWorker = False)
self.userArgs = ""
indexArg = 1
while indexArg < len(cmdlineArgs):
keyDataF = cmdlineArgs[indexArg][2:].split('=')
if keyDataF[0] in self.props.keys():
self.props[keyDataF[0]] = keyDataF[1]
else:
self.userArgs += cmdlineArgs[indexArg] + " "
indexArg += 1
if self.props['dtmTCPhosts'] == '':
cmdlineArgs = cmdlineArgs[1].split(' ')
self.userArgs = ""
indexArg = 1
while indexArg < len(cmdlineArgs):
keyDataF = cmdlineArgs[indexArg][2:].split('=')
if keyDataF[0] in self.props.keys():
self.props[keyDataF[0]] = keyDataF[1]
else:
self.userArgs += cmdlineArgs[indexArg] + " "
indexArg += 1
@property
def poolSize(self):
return len(self.tabConn)+1 # +1 because tabConn does not include ourselves
@property
def workerId(self):
return (self.workerHost, self.workerHostId)
@property
def isRootWorker(self):
return bool(int(self.props['dtmTCPisRootWorker']))
@property
def isLaunchProcess(self):
return bool(int(self.props['dtmTCPGlobalLaunch'])) or bool(int(self.props['dtmTCPLocalLaunch']))
def iterOverIDs(self):
return self.tabConn.keys() + [self.workerId]
def setTraceModeOn(self, xmlLog):
# Profiling is not currently enabled with TCP
return
def run(self):
# Looking if we are in "launch mode" or "execute mode"
if int(self.props['dtmTCPGlobalLaunch']):
# Global launch. Create a SSH connection with each node, and
# execute a local launch on them. Then wait for termination
sshWait = []
lineTab = []
maxWorkersByHost = 1
totalWorkers = 0
with open(self.props['dtmTCPhosts']) as hostFile:
for line in hostFile:
if line[0] == '#' or line[0] == '\n' or line[0] == '\r':
continue
lineTab.append(line.replace("\n", "").replace("\r", "").split(" "))
totalWorkers += int(lineTab[-1][1].split("=")[1])
if int(lineTab[-1][1].split("=")[1]) > maxWorkersByHost:
maxWorkersByHost = int(lineTab[-1][1].split("=")[1])
for lineCount,hostH in enumerate(lineTab):
if hostH[0] == 'localhost' or hostH[0] == '127.0.0.1' or hostH[0] == socket.gethostname():
# Local launch
if len(lineTab) > 1:
sys.stderr.write("Warning : 'localhost' used as a global hostname!\nDTM will probably hang.\nReplace localhost by the fully qualified name or run only on this host.")
sys.stderr.flush()
sshWait.append(subprocess.Popen(["python"+str(sys.version_info[0])+"."+str(sys.version_info[1]), self.cmdArgs[0],
" --dtmTCPGlobalLaunch=0 --dtmTCPLocalLaunch=1 --dtmTCPLineNo="+str(lineCount) +\
" --dtmTCPhosts=" + str(self.props['dtmTCPhosts']) + " --dtmTCPnPortsByHost=" + str(maxWorkersByHost*(totalWorkers-1)) +\
" --dtmTCPstartPort="+ str(self.props['dtmTCPstartPort']) + " --dtmTCPnbrWorkers="+str(hostH[1].split("=")[1])+\
" --dtmTCPmaxWorkersByHost="+ str(maxWorkersByHost)+" "+ self.userArgs], stdout=sys.stdout, stderr=sys.stderr))
else:
# Remote launch
sshWait.append(subprocess.Popen(["/usr/bin/ssh", "-x", hostH[0], "cd "+str(os.getcwd()) +";", "python"+\
str(sys.version_info[0])+"."+str(sys.version_info[1])+" " + self.cmdArgs[0] +\
" --dtmTCPGlobalLaunch=0 --dtmTCPLocalLaunch=1 --dtmTCPLineNo="+str(lineCount) +\
" --dtmTCPhosts=" + str(self.props['dtmTCPhosts']) + " --dtmTCPnPortsByHost=" + str(maxWorkersByHost*(totalWorkers-1)) +\
" --dtmTCPstartPort="+ str(self.props['dtmTCPstartPort']) + " --dtmTCPnbrWorkers="+str(hostH[1].split("=")[1])+\
" --dtmTCPmaxWorkersByHost="+ str(maxWorkersByHost)+" "+ self.userArgs], stdout=sys.stdout, stderr=sys.stderr))
for connJob in sshWait:
connJob.wait()
self.commReadyEvent.set()
return
elif int(self.props['dtmTCPLocalLaunch']):
# Local launch. Makes use of subprocess.popen to launch DTM workers
# After that, wait until termination
dtmLocalWorkersList = []
for i in range(int(self.props['dtmTCPnbrWorkers'])):
if i == 0 and int(self.props['dtmTCPLineNo']) == 0:
dtmLocalWorkersList.append(subprocess.Popen(["python"+str(sys.version_info[0])+"."+str(sys.version_info[1]), self.cmdArgs[0],
" --dtmTCPGlobalLaunch=0 --dtmTCPLocalLaunch=0 --dtmTCPLineNo="+str(self.props['dtmTCPLineNo']) +\
" --dtmTCPhosts=" + str(self.props['dtmTCPhosts']) + " --dtmTCPnPortsByHost=" + str(self.props['dtmTCPnPortsByHost']) +\
" --dtmTCPstartPort="+ str(self.props['dtmTCPstartPort']) + " --dtmTCPnbrWorkers="+str(i)+\
" --dtmTCPmaxWorkersByHost="+ str(self.props['dtmTCPmaxWorkersByHost']) +" --dtmTCPisRootWorker=1", self.userArgs]))
else:
dtmLocalWorkersList.append(subprocess.Popen(["python"+str(sys.version_info[0])+"."+str(sys.version_info[1]), self.cmdArgs[0],
" --dtmTCPGlobalLaunch=0 --dtmTCPLocalLaunch=0 --dtmTCPLineNo="+str(self.props['dtmTCPLineNo']) +\
" --dtmTCPhosts=" + str(self.props['dtmTCPhosts']) + " --dtmTCPnPortsByHost=" + str(self.props['dtmTCPnPortsByHost']) +\
" --dtmTCPstartPort="+ str(self.props['dtmTCPstartPort']) + " --dtmTCPnbrWorkers="+str(i)+\
" --dtmTCPmaxWorkersByHost="+ str(self.props['dtmTCPmaxWorkersByHost']), self.userArgs]))
for dtmW in dtmLocalWorkersList:
dtmW.wait()
self.commReadyEvent.set()
return
# Here, we are ready to start DTM
# We have to open the TCP ports and create the communication sockets
# This is somewhat tricky, because two workers on the same node
# must not have the same port number, and we do not want to bind
# n^2 different ports (which will be the trivial way).
#
# Each worker as an ID which is a tuple (host, unique number)
self.tabConn = {}
listListeners = {}
lineTab = []
self.workerHostId = int(self.props['dtmTCPnbrWorkers'])
maxWorkersByHost = int(self.props['dtmTCPmaxWorkersByHost'])
fullyhost = "boumbadaboum"
with open(self.props['dtmTCPhosts']) as hostFile:
for hostline in hostFile:
if hostline[0] == '#' or hostline[0] == '\n' or hostline[0] == '\r':
continue
lineTab.append(hostline.replace("\n", "").replace("\r", "").split(" "))
if len(lineTab)-1 == int(self.props['dtmTCPLineNo']):
fullyhost = lineTab[-1][0]
# If our line number is GREATER than the other OR our uniq ID is
# GREATER than the ID of another worker on our node,
# we create the Client which will attempt to connect to the
# listener opened by the other.
for lineC,hostInfos in enumerate(lineTab):
remoteIp = socket.gethostbyname(hostInfos[0])
if lineC > int(self.props['dtmTCPLineNo']):
for i in range(0, int(hostInfos[1].split("=")[1])):
try:
listListeners[(hostInfos[0], i)] = Listener((fullyhost, int(self.props['dtmTCPstartPort']) +\
lineC*maxWorkersByHost + i + self.workerHostId*maxWorkersByHost*len(lineTab)))
except Exception as ex:
sys.stderr.write(str(("ERROR IN : ", "Listener", self.props['dtmTCPLineNo'], self.workerHostId, lineC, i, maxWorkersByHost, len(lineTab), " < ", remoteIp,
lineC*maxWorkersByHost + i + self.workerHostId*maxWorkersByHost*len(lineTab))) + "\n\n")
raise ex
elif lineC == int(self.props['dtmTCPLineNo']):
self.workerHost = hostInfos[0]
for i in range(0, int(hostInfos[1].split("=")[1])):
if i == self.workerHostId:
continue # Pas de connexion avec nous-memes
if i > self.workerHostId:
try:
listListeners[(hostInfos[0], i)] = Listener(('127.0.0.1', int(self.props['dtmTCPstartPort']) + lineC*maxWorkersByHost + i + self.workerHostId*maxWorkersByHost*len(lineTab)))
except Exception as ex:
sys.stderr.write(str(("ERROR IN : ", "Listener", self.props['dtmTCPLineNo'], self.workerHostId, lineC, i, maxWorkersByHost, len(lineTab), " => ", remoteIp,
int(self.props['dtmTCPLineNo'])*maxWorkersByHost + i + self.workerHostId*maxWorkersByHost*len(lineTab))) +"\n\n\n")
raise ex
else:
self.tabConn[(hostInfos[0], i)] = Client(('127.0.0.1', int(self.props['dtmTCPstartPort']) + lineC*maxWorkersByHost + self.workerHostId + i*maxWorkersByHost*len(lineTab)))
else:
# lineC < int(self.props['dtmTCPLineNo'])
for i in range(0, int(hostInfos[1].split("=")[1])):
self.tabConn[(hostInfos[0], i)] = Client((remoteIp, int(self.props['dtmTCPstartPort']) +\
int(self.props['dtmTCPLineNo'])*maxWorkersByHost + self.workerHostId + i*maxWorkersByHost*len(lineTab)))
for le in listListeners.keys():
conn = listListeners[le].accept()
self.tabConn[le] = conn
self.commReadyEvent.set()
working = True
countRecvNotTransmit = 0
countRecvTimeInit = time.time()
while working:
recvSomething = False
sendSomething = False
if self.exitStatus.is_set(): # Exit
# Warning : the communication thread MUST clear the sendQ
# BEFORE leaving (the exiting orders must be send)
working = False
# Check for incoming messages
for conn in self.tabConn.values():
if conn.poll():
# We received something
try:
self.recvQ.put(conn.recv())
countRecvNotTransmit += 1
except EOFError:
working = False
recvSomething = True
if countRecvNotTransmit > 50 or (time.time() - countRecvTimeInit > 0.1 and countRecvNotTransmit > 0):
countRecvNotTransmit = 0
countRecvTimeInit = time.time()
self.wakeUpMainThread.set()
while True:
# Send all our messages
try:
sendMsg = self.sendQ.get_nowait()
self.tabConn[sendMsg.receiverWid].send(sendMsg)
sendSomething = True
except Queue.Empty:
break
if not recvSomething:
time.sleep(self.random.uniform(DTM_TCP_MIN_LATENCY, DTM_TCP_MAX_LATENCY))
deap-0.7.1/deap/dtm/dtmTypes.py 0000644 0000765 0000024 00000015356 11641072614 016556 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
DTM_TASK_STATE_IDLE = 0
DTM_TASK_STATE_RUNNING = 1
DTM_TASK_STATE_WAITING = 2
DTM_TASK_STATE_FINISH = 3
DTM_MSG_EXIT = 0
DTM_MSG_TASK = 1
DTM_MSG_REQUEST_TASK = 2
DTM_MSG_RESULT = 3
DTM_MSG_ACK_RECEIVED_TASK = 4
DTM_WAIT_NONE = 0
DTM_WAIT_SOME = 1
DTM_WAIT_ALL = 2
DTM_WAIT_ANY = 3
class TaskContainer(object):
"""
Contains all the information about a task (running or not)
"""
__slots__ = ('tid', # Unique ID of the task
'creatorWid', # ID of the worker who creates this task
'creatorTid', # ID of the task who creates this task (parent)
'taskIndex', # Position into the parents task childs list
'taskRoute', # Worker path followed by the job before begin
'creationTime', # Time at the job creation
'target', # Target function (or callable object) of the task
'args', # Arguments (list)
'kwargs', # Key-worded arguments (dictionnary)
'threadObject', # Python representation of the thread
'taskState') # State of the task (DTM_TASK_*)
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
def __lt__(self, other):
return self.creationTime < other.creationTime
class ResultContainer(object):
"""
Used to store the result of a task so it can be sent
"""
__slots__ = ('tid', # ID of the task which produced these results
'parentTid', # Parent ID (waiting for these results)
'taskIndex', # Position into the parents task childs list
'execTime', # Total execution time (NOT waiting time)
'success', # False if an exception occured
'result') # The result; if an exception occured, contains it
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
class ExceptedResultContainer(object):
"""
Keep the information of a result waited on the task creator side
"""
__slots__ = ('tids', # List of IDs of the tasks which produce these results
'waitingOn', # Is the parent task waiting on this result?
'finished', # Boolean : is the task finished (i.e. result received)?
'success', # Boolean : False if unfinished or if an exception occured
'callbackFunc', # Callback function, FOR USE IN DTM, NO ARGUMENTS PASSED, or None
'result') # Result, or the exception occured
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
class WaitInfoContainer(object):
"""
Keep a track on the pending child tasks of a parent task.
"""
__slots__ = ('threadObject', # Python representation of the thread
'eventObject', # threading.Event flag (to wake up the thread)
'waitBeginningTime', # Time when the thread started waiting (0 if not)
'tasksWaitingCount', # How many tasks are we waiting on
'waitingMode', # DTM_WAIT_* : waiting mode (None, One, Any, All)
'rWaitingDict') # List of ExceptedResultContainer, key : the first task ID
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
class StatsContainer(object):
"""
Contains stats about a target
"""
__slots__ = ('rAvg', # RELATIVE average execution time
'rStdDev', # RELATIVE standard deviation of the exec time
'rSquareSum', # Square sum of the RELATIVE exec times
'execCount') # Number of executions
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
class MessageContainer(object):
"""
Generic message container
If msgType == DTM_MSG_EXIT:
msg = (Exit code, exit message)
If msgType == DTM_MSG_TASK:
msg = [TaskContainer, TaskContainer, TaskContainer, ...]
If msgType == DTM_MSG_REQUEST_TASK:
msg = None
If msgType == DTM_MSG_RESULT:
msg = [ResultContainer, ResultContainer, ResultContainer, ...]
If msgType == DTM_MSG_ACK_RECEIVED_TASK:
msg = AckId
"""
__slots__ = ('msgType', # Message type (DTM_MSG_*)
'senderWid', # Worker id of the sender
'receiverWid', # Worker id of the receiver
'loadsDict', # Load dictionnary of the sender
'targetsStats', # Stats on the tasks of the sender
'prepTime', # Time when it was ready to send
'sendTime', # Time when sent
'ackNbr', # ACK number (optionnal for some operations)
'msg') # Message (varies with msgType)
def __init__(self, **kwargs):
self.__setstate__(kwargs)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self, state):
for t in state:
self.__setattr__(t, state[t])
deap-0.7.1/deap/dtm/loadBalancerPDB.py 0000644 0000765 0000024 00000025131 11641072614 017632 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
import threading
try:
import Queue
except ImportError:
import queue as Queue
import time
import copy
import logging
try:
from lxml import etree
except ImportError:
try:
import xml.etree.cElementTree as etree
except ImportError:
# Python 2.5
import xml.etree.ElementTree as etree
_logger = logging.getLogger("dtm.loadBalancing")
DTM_ASK_FOR_TASK_DELAY = 2.5
DTM_SEND_TASK_DELAY = 0.001
DTM_RESTART_QUEUE_BLOCKING_FROM = 1.
class LoadInfoContainer(object):
"""
Contains load information of a worker
"""
__slots__ = ('loadCurrentExec', # Load of the current exec task
'loadExecQ', # Load of the exec queue
'loadWaitingRestartQ', # Load of the waiting for restart queue
'loadWaitingQ', # Load of the waiting queue
'seqNbr', # Sequence number, only ++ by the worker which is its info
'infoUpToDate', # Boolean : is the dictionnary of the OTHER worker up to date with our info?
'ackWaiting') # List of ACKs (messages transmitted)
def __init__(self,args):
self.__setstate__(args)
def __getstate__(self):
d = {}
for a in self.__slots__:
d[a] = self.__getattribute__(a)
return d
def __setstate__(self,state):
for t in state:
self.__setattr__(t,state[t])
def sumRealLoad(self):
return self.loadCurrentExec+self.loadExecQ+self.loadWaitingRestartQ
class LoadBalancer(object):
"""
"""
def __init__(self, workersIterator, workerId, execQueue, randomizer):
self.wid = workerId
self.ws = {}
self.execQ = execQueue # Les autres queues ne sont pas necessaires
self.wIter = workersIterator
self.dLock = threading.Lock()
self.tmpRecvJob = False
self.localRandom = randomizer
self.xmlLogger = None
for w in workersIterator:
self.ws[w] = LoadInfoContainer({'loadCurrentExec' : 0.,
'loadExecQ' : 0.,
'loadWaitingRestartQ' : 0.,
'loadWaitingQ' : 0.,
'seqNbr' : 0.,
'infoUpToDate' : False,
'ackWaiting' : []})
self.totalExecLoad, self.totalEQueueLoad, self.totalWaitingRQueueLoad, self.totalWaitingQueueLoad = 0., 0., 0., 0.
self.sendDelayMultiplier = DTM_SEND_TASK_DELAY*float(len(self.ws))**0.5
self.lastTaskSend = time.time() - self.sendDelayMultiplier
self.lastQuerySend = time.time()
def setTraceModeOn(self, xmlLogger):
self.xmlLogger = xmlLogger
def getNodesDict(self):
return self.ws
def updateSelfStatus(self, statusTuple):
self.ws[self.wid].loadCurrentExec = statusTuple[0]
self.ws[self.wid].loadExecQ = statusTuple[1]
self.ws[self.wid].loadWaitingRestartQ = statusTuple[2]
self.ws[self.wid].loadWaitingQ = statusTuple[3]
self.ws[self.wid].seqNbr += 1
def mergeNodeStatus(self, otherDict):
for wId in otherDict:
if len(self.ws[wId].ackWaiting) == 0 and otherDict[wId].seqNbr > self.ws[wId].seqNbr and wId != self.wid:
remInfoUpToDate = self.ws[wId].infoUpToDate
self.ws[wId] = otherDict[wId] # Les deux dernieres infos sont "personnelles"
self.ws[wId].infoUpToDate = remInfoUpToDate
self.ws[wId].ackWaiting = []
def notifyTaskReceivedFrom(self, fromId):
self.ws[fromId].infoUpToDate = True
self.tmpRecvJob = True
def acked(self, fromWorker, ackN):
try:
self.ws[fromWorker].ackWaiting.remove(ackN)
except ValueError:
print("ERROR : Tentative to delete an already received or non-existant ACK!", self.ws[fromWorker].ackWaiting, ackN)
def takeDecision(self):
MAX_PROB = 1.
MIN_PROB = 0.00
sendTasksList = []
sendNotifList = []
if not self.xmlLogger is None:
decisionLog = etree.SubElement(self.xmlLogger, "decision", {"time" : repr(time.time()),
"selfLoad" : repr(self.ws[self.wid].loadCurrentExec)+","+repr(self.ws[self.wid].loadExecQ)+","+repr(self.ws[self.wid].loadWaitingRestartQ)+","+repr(self.ws[self.wid].loadWaitingQ)})
for workerId in self.ws:
etree.SubElement(decisionLog, "workerKnownState", {"load" : repr(self.ws[workerId].loadCurrentExec)+","+repr(self.ws[workerId].loadExecQ)+","+repr(self.ws[workerId].loadWaitingRestartQ)+","+repr(self.ws[workerId].loadWaitingQ)})
listLoads = self.ws.values()
self.totalExecLoad, self.totalEQueueLoad, self.totalWaitingRQueueLoad, self.totalWaitingQueueLoad = 0., 0., 0., 0.
totalSum2 = 0.
for r in listLoads:
self.totalExecLoad += r.loadCurrentExec
self.totalEQueueLoad += r.loadExecQ
self.totalWaitingRQueueLoad += r.loadWaitingRestartQ
self.totalWaitingQueueLoad += r.loadWaitingQ
totalSum2 += (r.loadCurrentExec+r.loadExecQ+r.loadWaitingRestartQ)**2
avgLoad = (self.totalExecLoad + self.totalEQueueLoad + self.totalWaitingRQueueLoad) / float(len(self.ws))
stdDevLoad = (totalSum2/float(len(self.ws)) - avgLoad**2)**0.5
selfLoad = self.ws[self.wid].sumRealLoad()
diffLoad = selfLoad - avgLoad
listPossibleSendTo = list(filter(lambda d: d[1].infoUpToDate and d[1].sumRealLoad() > avgLoad, self.ws.items()))
if selfLoad == 0 and len(listPossibleSendTo) > 0 and avgLoad != 0 and self.ws[self.wid].loadWaitingRestartQ < DTM_RESTART_QUEUE_BLOCKING_FROM:
# Algorithme d'envoi de demandes de taches
self.lastQuerySend = time.time()
txtList = ""
for worker in listPossibleSendTo:
sendNotifList.append(worker[0])
txtList +=str(worker[0])+","
self.ws[worker[0]].infoUpToDate = False
if not self.xmlLogger is None:
etree.SubElement(decisionLog, "action", {"time" : repr(time.time()), "type" : "sendrequest", "destination":txtList})
if self.ws[self.wid].loadExecQ > 0 and diffLoad > -stdDevLoad and avgLoad != 0 and stdDevLoad != 0:
# Algorithme d'envoi de taches
def scoreFunc(loadi):
if loadi < (avgLoad-2*stdDevLoad) or loadi == 0:
return MAX_PROB # Si le load du worker est vraiment tres bas, forte probabilite de lui envoyer des taches
elif loadi > (avgLoad + stdDevLoad):
return MIN_PROB # Si le load du worker est tres haut, tres faible probabilite de lui envoyer des taches
else:
a = (MIN_PROB-MAX_PROB)/(3*stdDevLoad)
b = MIN_PROB - a*(avgLoad + stdDevLoad)
return a*loadi + b # Lineaire entre Avg-2*stdDev et Avg+stdDev
scores = [(None,0)] * (len(self.ws)-1) # Gagne-t-on vraiment du temps en prechargeant la liste?
i = 0
for worker in self.ws:
if worker == self.wid:
continue
scores[i] = (worker, scoreFunc(self.ws[worker].sumRealLoad()))
i += 1
if not self.xmlLogger is None:
etree.SubElement(decisionLog, "action", {"time" : repr(time.time()), "type" : "checkavail", "destination":str(scores)})
while diffLoad > 0.00000001 and len(scores) > 0 and self.ws[self.wid].loadExecQ > 0.:
selectedIndex = self.localRandom.randint(0,len(scores)-1)
if self.localRandom.random() > scores[selectedIndex][1]:
del scores[selectedIndex]
continue
widToSend = scores[selectedIndex][0]
loadForeign = self.ws[widToSend]
diffLoadForeign = loadForeign.sumRealLoad() - avgLoad
sendT = 0.
if diffLoadForeign < 0: # On veut lui envoyer assez de taches pour que son load = loadAvg
sendT = diffLoadForeign*-1 if diffLoadForeign*-1 < self.ws[self.wid].loadExecQ else self.ws[self.wid].loadExecQ
elif diffLoadForeign < stdDevLoad: # On veut lui envoyer assez de taches pour son load = loadAvg + stdDev
sendT = stdDevLoad - diffLoadForeign if stdDevLoad - diffLoadForeign < self.ws[self.wid].loadExecQ else self.ws[self.wid].loadExecQ
else: # On envoie une seule tache
sendT = 0.
tasksIDs, retiredTime = self.execQ.getTasksIDsWithExecTime(sendT)
tasksObj = []
for tID in tasksIDs:
t = self.execQ.getSpecificTask(tID)
if not t is None:
tasksObj.append(t)
if len(tasksObj) > 0:
diffLoad -= retiredTime
self.ws[self.wid].loadExecQ -= retiredTime
self.ws[widToSend].loadExecQ += retiredTime
ackNbr = len(self.ws[widToSend].ackWaiting)
self.ws[widToSend].ackWaiting.append(ackNbr)
sendTasksList.append((widToSend, tasksObj, ackNbr))
del scores[selectedIndex] # On s'assure de ne pas reprendre le meme worker
if not self.xmlLogger is None:
etree.SubElement(decisionLog, "action", {"time" : repr(time.time()), "type" : "sendtasks", "destination":str([b[0] for b in sendTasksList]), "tasksinfo" : str([len(b[1]) for b in sendTasksList])})
self.lastTaskSend = time.time()
return sendNotifList, sendTasksList deap-0.7.1/deap/dtm/manager.py 0000644 0000765 0000024 00000175442 11641072614 016362 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
import threading
import time
import math
import random
try:
import Queue
except ImportError:
import queue as Queue
import copy
import pickle
import logging
import sys
import os
PRETTY_PRINT_SUPPORT = False
try:
from lxml import etree
PRETTY_PRINT_SUPPORT = True
except ImportError:
try:
import xml.etree.cElementTree as etree
except ImportError:
# Python 2.5
import xml.etree.ElementTree as etree
from deap.dtm.dtmTypes import *
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout)
_logger = logging.getLogger("dtm.control")
# Constantes
DTM_CONTROL_THREAD_LATENCY = 0.05
MSG_COMM_TYPE = 0
MSG_SENDER_INFO = 1
MSG_NODES_INFOS = 2
DTM_LOGDIR_DEFAULT_NAME = "DtmLog"
try:
from math import erf
except ImportError:
def erf(x):
# See http://stackoverflow.com/questions/457408/is-there-an-easily-available-implementation-of-erf-for-python
# save the sign of x
sign = 1
if x < 0:
sign = -1
x = abs(x)
# constants
a1 = 0.254829592
a2 = -0.284496736
a3 = 1.421413741
a4 = -1.453152027
a5 = 1.061405429
p = 0.3275911
# A&S formula 7.1.26
t = 1.0 / (1.0 + p * x)
y = 1.0 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * math.exp(-x * x)
return sign * y # erf(-x) = -erf(x)
class TaskIdGenerator(object):
"""
A thread-safe ID generator.
A DTM task ID is a tuple looking like (workerId, uniqId)
With MPI, workerId is an integer (rank) and the uniqIds start from 0,
but workerId can be virtually any immutable object (as it is used as
a dictionnary key)
"""
def __init__(self, rank, initId=0):
self.r = rank
self.wtid = 0
self.generatorLock = threading.Lock()
@property
def tid(self):
self.generatorLock.acquire()
newId = self.wtid
self.wtid += 1
self.generatorLock.release()
return (self.r, newId)
class ExecInfo(object):
"""
Contains the information about the current task
"""
def __init__(self, tStats, tStatsLock):
self.eLock = threading.Lock()
self.tStats = tStats
self.tStatsLock = tStatsLock
self.eCurrent = None
self.mainThread = threading.currentThread()
self.piSqrt = math.sqrt(math.pi)
self.sqrt2 = 2 ** 0.5
def _execTimeRemaining(self, mean, stdDev, alreadyDone):
if alreadyDone == 0.:
# If not started yet, return mean
return mean
#
# Else compute the mass center of the remaining part of the gaussian
# For instance, if we have a gaussian (mu, sigma) = (3, 4)
# and that the task is executing for 3 seconds, then we may estimate
# that it will probably finished at :
# int(x*gaussian) / int(gaussian) over [3, +inf[
# (where int is integral)
#
# We get 6.192 seconds, that is an expected remaining time of
# 6.192 - 3 = 3.192 seconds
# Evaluate primitive at point 'alreadyDone'
commonPart = erf(self.sqrt2 * (alreadyDone - mean) / (stdDev * 2))
areaPart1 = 0.5 * commonPart
massCenterPart1 = (self.sqrt2 / (4 * self.piSqrt)) * (-2 * stdDev * math.exp(-0.5 * ((alreadyDone - mean) ** 2) / (stdDev ** 2)) + mean * (self.piSqrt) * (self.sqrt2) * commonPart)
# Evaluate primitive at the infinite
# erf(+inf) = 1, so
areaPart2 = 0.5
# exp(-inf) = 0, and erf(+inf) = 1, so
massCenterPart2 = mean / 2.
if areaPart1 == areaPart2:
# So far from the mean that erf(alreadyDone - moyenne) = 1
previsionPoint = alreadyDone + 0.5 # Hack : lets assume that there's 0.5 sec left...
else:
previsionPoint = (massCenterPart2 - massCenterPart1) / (areaPart2 - areaPart1)
return previsionPoint - alreadyDone
def acquire(self, forTask, blocking=True):
if self.eLock.acquire(blocking):
self.eCurrent = forTask
return True
else:
return False
def isLocked(self):
if self.eLock.acquire(False):
self.eLock.release()
return False
return True
def release(self):
self.eCurrent = None
self.eLock.release()
def getLoad(self):
try:
self.tStatsLock.acquire()
tInfo = self.tStats.get(self.eCurrent.target, StatsContainer(rAvg=1., rStdDev=1., rSquareSum=0., execCount=0))
val = self._execTimeRemaining(tInfo.rAvg, tInfo.rStdDev, self.eCurrent.threadObject.timeExec)
self.tStatsLock.release()
return val
except AttributeError:
self.tStatsLock.release()
return 0.
class TaskQueue(object):
"""
"""
def __init__(self, tasksStatsStruct, tasksStatsLock):
self.tStats = tasksStatsStruct
self.tStatsLock = tasksStatsLock
self.previousLoad = 0.
self.piSqrt = math.sqrt(math.pi)
self.sqrt2 = 2 ** 0.5
self._tQueue = Queue.PriorityQueue()
self._tDict = {}
self._actionLock = threading.Lock()
self.changed = True
def _getTimeInfoAbout(self, task):
if task.threadObject is None:
timeDone = 0.
else:
timeDone = task.threadObject.timeExec
self.tStatsLock.acquire()
tInfo = self.tStats.get(task.target, StatsContainer(rAvg=1., rStdDev=1., rSquareSum=0., execCount=0))
self.tStatsLock.release()
return self._execTimeRemaining(tInfo.rAvg, tInfo.rStdDev, timeDone)
def _execTimeRemaining(self, mean, stdDev, alreadyDone):
if alreadyDone == 0. or stdDev == 0:
return mean
#
# Else compute the mass center of the remaining part of the gaussian
# For instance, if we have a gaussian (mu, sigma) = (3, 4)
# and that the task is executing for 3 seconds, then we may estimate
# that it will probably finished at :
# int(x*gaussian) / int(gaussian) over [3, +inf[
# (where int is integral)
#
# We get 6.192 seconds, that is an expected remaining time of
# 6.192 - 3 = 3.192 seconds
# Evaluate primitive at point 'alreadyDone'
commonPart = erf(self.sqrt2 * (alreadyDone - mean) / (stdDev * 2))
areaPart1 = 0.5 * commonPart
massCenterPart1 = (self.sqrt2 / (4 * self.piSqrt)) * (-2 * stdDev * math.exp(-0.5 * ((alreadyDone - mean) ** 2) / (stdDev ** 2)) + mean * (self.piSqrt) * (self.sqrt2) * commonPart)
# Evaluate primitive at the infinite
# erf(+inf) = 1, so
areaPart2 = 0.5
# exp(-inf) = 0, and erf(+inf) = 1, so
massCenterPart2 = mean / 2.
if areaPart1 == areaPart2:
# So far from the mean that erf(alreadyDone - moyenne) = 1
previsionPoint = alreadyDone + 0.5 # Hack : lets assume that there's 0.5 sec left...
else:
previsionPoint = (massCenterPart2 - massCenterPart1) / (areaPart2 - areaPart1)
return previsionPoint - alreadyDone
def put(self, taskObject):
self.putList((taskObject,))
def putList(self, tasksList):
self._actionLock.acquire()
for taskObject in tasksList:
self._tDict[taskObject.tid] = taskObject
self._tQueue.put(taskObject)
self.changed = True
self._actionLock.release()
def getTask(self):
self._actionLock.acquire()
while True:
try:
taskObject = self._tQueue.get_nowait()
except Queue.Empty:
self._actionLock.release()
raise Queue.Empty
if taskObject.tid in self._tDict:
del self._tDict[taskObject.tid]
self.changed = True
self._actionLock.release()
return taskObject
def isTaskIn(self, taskId):
return taskId in self._tDict
def _getApproximateLoad(self):
return self.previousLoad
def getLen(self):
return len(self._tDict)
def getLoad(self):
tmpLoad = 0.
self._actionLock.acquire()
if not self.changed:
self._actionLock.release()
return self.previousLoad
for tid, tObj in self._tDict.items():
tmpLoad += self._getTimeInfoAbout(tObj)
self.previousLoad = tmpLoad
self.changed = False
self._actionLock.release()
return self.previousLoad
def getSpecificTask(self, taskId):
self._actionLock.acquire()
if taskId in self._tDict:
task = self._tDict[taskId]
del self._tDict[taskId]
self.changed = True
else:
task = None
self._actionLock.release()
return task
def getTaskByExecTime(self, execTimeWanted, maxDiff= -1.):
# maxDiff is the max difference wanted
# If there is no task with the execTimeWanted +- maxDiff,
# return None
# maxDiff = -1 means that there is no max difference
mostClose = None
self._actionLock.acquire()
for tid, tObj in self._tDict.items():
timeDiff = math.abs(execTimeWanted - self._getTimeInfoAbout(tObj))
if mostClose is None or mostClose[1] > timeDiff:
mostClose = (tid, timeDiff)
self._actionLock.release()
if mostClose is None or (maxDiff >= 0 and mostClose[1] > maxDiff):
return None
else:
return self.getSpecificTask(mostClose[0])
def getTasksIDsWithExecTime(self, execTimeWanted, maxDiff= -1.):
# Return task list containing tasks which the durations approximately
# sum to execTimeWanted
returnList = []
totalTime = 0.
self._actionLock.acquire()
for tid, tObj in self._tDict.items():
timeInfo = self._getTimeInfoAbout(tObj)
if totalTime + timeInfo <= execTimeWanted:
returnList.append(tid)
totalTime += timeInfo
#
# Hack to avoid the starting bug (few jobs but very long)
#
if len(returnList) == 0 and maxDiff == -1 and len(self._tDict) > 1:
it = list(filter(lambda x: self._getTimeInfoAbout(x[1]) != 0, self._tDict.items()))
if len(it) != 0:
returnList.append(it[0][0])
totalTime += self._getTimeInfoAbout(it[0][1])
self._actionLock.release()
return returnList, totalTime
class Control(object):
"""
Control is the main DTM class. The dtm object you receive when you use ``from deap import dtm``
is a proxy over an instance of this class.
Most of its methods are used by your program, in the execution tasks; however, two of thems (start() and setOptions()) MUST be called
in the MainThread (i.e. the thread started by the Python interpreter).
As this class is instancied directly in the module, initializer takes no arguments.
"""
def __init__(self):
self.sTime = time.time()
# Key : Target, Value : StatsContainer
self.tasksStats = {}
self.tasksStatsLock = threading.Lock()
# Global execution lock
self.dtmExecLock = ExecInfo(self.tasksStats, self.tasksStatsLock)
self.waitingThreadsQueue = TaskQueue(self.tasksStats, self.tasksStatsLock)
# Key : task ID, Value : WaitInfoContainer
self.waitingThreads = {}
self.waitingThreadsLock = threading.Lock()
self.launchTaskLock = threading.Lock()
self.waitingForRestartQueue = TaskQueue(self.tasksStats, self.tasksStatsLock)
self.execQueue = TaskQueue(self.tasksStats, self.tasksStatsLock)
self.recvQueue = Queue.Queue() # Contains MessageContainer
self.sendQueue = Queue.Queue() # Contains MessageContainer
self.exitStatus = threading.Event()
self.commExitNotification = threading.Event()
self.commReadyEvent = threading.Event()
self.exitState = (None, None)
self.exitSetHere = False
# Will stop the main thread until an event occurs
self.runningFlag = threading.Event()
self.runningFlag.set()
self.commManagerType = "deap.dtm.commManagerMpi4py"
self.loadBalancerType = "deap.dtm.loadBalancerPDB"
self.printExecSummary = True
self.isStarted = False
self.traceMode = False
self.traceRoot = None # Root element of the XML log
self.traceTasks = None # XML elements
self.traceComm = None # defined if traceMode == True
self.traceLoadB = None
self.traceLock = threading.Lock()
self.refTime = 1.
self.loadBalancer = None
self.lastRetValue = None
self.dtmRandom = random.Random()
def _doCleanUp(self):
"""
Clean up function, called at this end of the execution.
Should NOT be called by the user.
"""
#if self.printExecSummary:
#_logger.info("[%s] did %i tasks", str(self.workerId), self._DEBUG_COUNT_EXEC_TASKS)
if self.exitSetHere:
for n in self.commThread.iterOverIDs():
if n == self.workerId:
continue
self.sendQueue.put(MessageContainer(msgType = DTM_MSG_EXIT,
senderWid = self.workerId,
receiverWid = n,
loadsDict = None,
targetsStats = None,
prepTime = time.time(),
sendTime = 0,
ackNbr = -1,
msg = (0, "Exit with success")))
self.commExitNotification.set()
self.commThread.join()
del self.execQueue
del self.sendQueue
del self.recvQueue
countThreads = sum([1 for th in threading.enumerate() if not th.daemon])
if countThreads > 1:
_logger.warning("[%s] There's more than 1 active thread (%i) at the exit.", str(self.workerId), threading.activeCount())
if self.commThread.isRootWorker:
if self.printExecSummary:
msgT = " Tasks execution times summary :\n"
for target, times in self.tasksStats.items():
msgT += "\t" + str(target) + " : Avg=" + str(times.rAvg * self.refTime) + ", StdDev=" + str(times.rStdDev * self.refTime) + "\n\n"
_logger.info(msgT)
_logger.info("DTM execution ended (no more tasks)")
_logger.info("Total execution time : %s", str(time.time() - self.sTime))
if self.traceMode:
_logger.info("Writing log to the log files...")
self.traceLock.acquire()
fstat = etree.SubElement(self.traceRoot, "finalStats")
for target, times in self.tasksStats.items():
etree.SubElement(fstat, "taskinfo", {"target" : str(target), "avg" : repr(times.rAvg * self.refTime), "stddev" : repr(times.rStdDev * self.refTime), "execcount" : str(times.execCount)})
flog = open(DTM_LOGDIR_DEFAULT_NAME + "/log" + str(self.workerId) + ".xml", 'w')
if PRETTY_PRINT_SUPPORT:
flog.write(etree.tostring(self.traceRoot, pretty_print=True))
else:
flog.write(etree.tostring(self.traceRoot))
flog.close()
self.traceLock.release()
if self.exitSetHere:
if self.lastRetValue[0]:
return self.lastRetValue[1]
else:
raise self.lastRetValue[1]
def _addTaskStat(self, taskKey, timeExec):
# The execution time is based on the calibration ref time
comparableLoad = timeExec / self.refTime
# We do not keep up all the execution times, but
# update the mean and stddev realtime
self.tasksStatsLock.acquire()
if not taskKey in self.tasksStats:
self.tasksStats[taskKey] = StatsContainer(rAvg = timeExec,
rStdDev = 0.,
rSquareSum = timeExec * timeExec,
execCount = 1)
else:
oldAvg = self.tasksStats[taskKey].rAvg
oldStdDev = self.tasksStats[taskKey].rStdDev
oldSum2 = self.tasksStats[taskKey].rSquareSum
oldExecCount = self.tasksStats[taskKey].execCount
self.tasksStats[taskKey].rAvg = (timeExec + oldAvg * oldExecCount) / (oldExecCount + 1)
self.tasksStats[taskKey].rSquareSum = oldSum2 + timeExec * timeExec
self.tasksStats[taskKey].rStdDev = abs(self.tasksStats[taskKey].rSquareSum / (oldExecCount + 1) - self.tasksStats[taskKey].rAvg ** 2) ** 0.5
self.tasksStats[taskKey].execCount = oldExecCount + 1
self.tasksStatsLock.release()
def _calibrateExecTime(self, runsN=3):
"""
Small calibration test, run at startup
Should not be called by user
"""
timesList = []
for r in range(runsN):
timeS = time.clock()
a = 0.
for i in range(10000):
a = math.sqrt(self.dtmRandom.random() / (self.dtmRandom.uniform(0, i) + 1))
strT = ""
for i in range(5000):
strT += str(self.dtmRandom.randint(0, 9999))
for i in range(500):
pickStr = pickle.dumps(strT)
strT = pickle.loads(pickStr)
timesList.append(time.clock() - timeS)
return sorted(timesList)[int(runsN / 2)]
def _getLoadTuple(self):
return (self.dtmExecLock.getLoad(), self.execQueue.getLoad(), self.waitingForRestartQueue.getLoad(), self.waitingThreadsQueue.getLoad())
def _returnResult(self, idToReturn, resultInfo):
"""
Called by the execution threads when they have to return a result
Should NOT be called explicitly by the user
"""
if idToReturn == self.workerId:
self._dispatchResults((resultInfo,))
else:
self.sendQueue.put(MessageContainer(msgType = DTM_MSG_RESULT,
senderWid = self.workerId,
receiverWid = idToReturn,
loadsDict = self.loadBalancer.getNodesDict(),
targetsStats = self.tasksStats,
prepTime = time.time(),
sendTime = 0,
ackNbr = -1,
msg = (resultInfo,)))
def _updateStats(self, msg):
"""
Called by the control thread to update its dictionnary
Should NOT be called explicitly by the user
"""
self.loadBalancer.mergeNodeStatus(msg.loadsDict)
self.tasksStatsLock.acquire()
for key, val in msg.targetsStats.items():
if not key in self.tasksStats or val.execCount > self.tasksStats[key].execCount:
self.tasksStats[key] = val
self.tasksStatsLock.release()
def _dispatchResults(self, resultsList):
"""
Called by the control thread when a message is received;
Dispatch it to the task waiting for it.
Should NOT be called explicitly by the user
"""
for result in resultsList:
self.waitingThreadsLock.acquire()
# We look for the task waiting for each result
foundKey = None
for taskKey in self.waitingThreads[result.parentTid].rWaitingDict:
try:
self.waitingThreads[result.parentTid].rWaitingDict[taskKey].tids.remove(result.tid)
except ValueError:
# The ID is not in this waiting list, continue with other worker
continue
foundKey = taskKey
if isinstance(self.waitingThreads[result.parentTid].rWaitingDict[taskKey].result, list):
if not result.success:
# Exception occured
self.waitingThreads[result.parentTid].rWaitingDict[taskKey].result = result.result
else:
self.waitingThreads[result.parentTid].rWaitingDict[taskKey].result[result.taskIndex] = result.result
break
assert not foundKey is None, "Parent task not found for result dispatch!" # Debug
if len(self.waitingThreads[result.parentTid].rWaitingDict[foundKey].tids) == 0:
# All tasks done
self.waitingThreads[result.parentTid].rWaitingDict[foundKey].finished = True
if isinstance(self.waitingThreads[result.parentTid].rWaitingDict[foundKey].result, list):
self.waitingThreads[result.parentTid].rWaitingDict[foundKey].success = True
canRestart = False
if self.waitingThreads[result.parentTid].rWaitingDict[foundKey].waitingOn == True or self.waitingThreads[result.parentTid].waitingMode == DTM_WAIT_ALL:
if (self.waitingThreads[result.parentTid].waitingMode == DTM_WAIT_ALL and len(self.waitingThreads[result.parentTid].rWaitingDict) == 1)\
or self.waitingThreads[result.parentTid].waitingMode == DTM_WAIT_ANY:
canRestart = True
elif self.waitingThreads[result.parentTid].waitingMode == DTM_WAIT_SOME:
canRestart = True
for rKey in self.waitingThreads[result.parentTid].rWaitingDict:
if self.waitingThreads[result.parentTid].rWaitingDict[rKey].waitingOn and rKey != foundKey:
canRestart = False
if not self.waitingThreads[result.parentTid].rWaitingDict[taskKey].callbackFunc is None:
self.waitingThreads[result.parentTid].rWaitingDict[taskKey].callbackFunc()
if canRestart:
wTask = self.waitingThreadsQueue.getSpecificTask(result.parentTid)
assert not wTask is None
self.waitingForRestartQueue.put(wTask)
self.waitingThreadsLock.release()
def _startNewTask(self):
"""
Start a new task (if there's one available)
Return True if so
Should NOT be called explicitly by the user
"""
taskLauched = False
self.launchTaskLock.acquire()
if not self.dtmExecLock.isLocked():
try:
wTask = self.waitingForRestartQueue.getTask()
self.dtmExecLock.acquire(wTask)
wTask.threadObject.waitingFlag.set()
taskLauched = True
except Queue.Empty:
pass
if not taskLauched:
try:
newTask = self.execQueue.getTask()
if self.traceMode:
self.traceLock.acquire()
newTaskElem = etree.SubElement(self.traceTasks, "task",
{"id" : str(newTask.tid),
"creatorTid" : str(newTask.creatorTid),
"creatorWid" : str(newTask.creatorWid),
"taskIndex" : str(newTask.taskIndex),
"creationTime" : repr(newTask.creationTime)})
try:
newTaskTarget = etree.SubElement(newTaskElem, "target",
{"name" : str(newTask.target.__name__)})
except AttributeError:
newTaskTarget = etree.SubElement(newTaskElem, "target",
{"name" : str(newTask.target)})
for i, a in enumerate(newTask.args):
newTaskTarget.set("arg" + str(i), str(a))
for k in newTask.kwargs:
newTaskTarget.set("kwarg_" + str(k), str(newTask.kwargs[k]))
newTaskPath = etree.SubElement(newTaskElem, "path", {"data" : str(newTask.taskRoute)})
self.traceLock.release()
newThread = DtmThread(newTask, self, newTaskElem)
else:
newThread = DtmThread(newTask, self)
self.dtmExecLock.acquire(newTask)
newThread.start()
taskLauched = True
except Queue.Empty:
pass
self.launchTaskLock.release()
return taskLauched
def _main(self):
"""
Main loop of the control thread
Should NOT be called explicitly by the user
"""
timeBegin = time.time()
while True:
self.runningFlag.wait() # WARNING, may deadlock on very specific conditions
self.runningFlag.clear() # (if the _last_ task do a set() between those 2 lines, nothing will wake up the main thread)
while True:
try:
recvMsg = self.recvQueue.get_nowait()
if recvMsg.msgType == DTM_MSG_EXIT:
self.exitStatus.set()
self.exitState = (recvMsg.msg[1], recvMsg.msg[0])
break
elif recvMsg.msgType == DTM_MSG_TASK:
self.execQueue.putList(recvMsg.msg)
self.loadBalancer.updateSelfStatus(self._getLoadTuple())
self.sendQueue.put(MessageContainer(msgType = DTM_MSG_ACK_RECEIVED_TASK,
senderWid = self.workerId,
receiverWid = recvMsg.senderWid,
loadsDict = self.loadBalancer.getNodesDict(),
targetsStats = self.tasksStats,
prepTime = time.time(),
sendTime = 0,
ackNbr = -1,
msg = recvMsg.ackNbr))
self._updateStats(recvMsg)
elif recvMsg.msgType == DTM_MSG_RESULT:
self._dispatchResults(recvMsg.msg)
self._updateStats(recvMsg)
elif recvMsg.msgType == DTM_MSG_REQUEST_TASK:
self._updateStats(recvMsg)
elif recvMsg.msgType == DTM_MSG_ACK_RECEIVED_TASK:
self.loadBalancer.acked(recvMsg.senderWid, recvMsg.msg)
self._updateStats(recvMsg)
else:
_logger.warning("[%s] Unknown message type %s received will be ignored.", str(self.workerId), str(recvMsg.msgType))
except Queue.Empty:
break
if self.exitStatus.is_set():
break
currentNodeStatus = self._getLoadTuple()
self.loadBalancer.updateSelfStatus(currentNodeStatus)
sendUpdateList, sendTasksList = self.loadBalancer.takeDecision()
self.tasksStatsLock.acquire()
for sendInfo in sendTasksList:
self.sendQueue.put(MessageContainer(msgType = DTM_MSG_TASK,
senderWid = self.workerId,
receiverWid = sendInfo[0],
loadsDict = self.loadBalancer.getNodesDict(),
targetsStats = self.tasksStats,
prepTime = time.time(),
sendTime = 0,
ackNbr = sendInfo[2],
msg = sendInfo[1]))
for updateTo in sendUpdateList:
self.sendQueue.put(DtmMessageContainer(msgType = DTM_MSG_REQUEST_TASK,
senderWid = self.workerId,
receiverWid = updateTo,
loadsDict = self.loadBalancer.getNodesDict(),
targetsStats = self.tasksStats,
prepTime = time.time(),
sendTime = 0,
ackNbr = -1,
msg = None))
self.tasksStatsLock.release()
self._startNewTask()
return self._doCleanUp()
def setOptions(self, *args, **kwargs):
"""
Set a DTM global option.
.. warning::
This function must be called BEFORE ``start()``. It is also the user responsability to ensure that the same option is set on every worker.
Currently, the supported options are :
* **communicationManager** : can be *deap.dtm.mpi4py* (default) or *deap.dtm.commManagerTCP*.
* **loadBalancer** : currently only the default *PDB* is available.
* **printSummary** : if set, DTM will print a task execution summary at the end (mean execution time of each tasks, how many tasks did each worker do, ...)
* **setTraceMode** : if set, will enable a special DTM tracemode. In this mode, DTM logs all its activity in XML files (one by worker). Mainly for DTM debug purpose, but can also be used for profiling.
This function can be called more than once. Any unknown parameter will have no effect.
"""
if self.isStarted:
if self.commThread.isRootWorker:
_logger.warning("dtm.setOptions() was called after dtm.start(); options will not be considered")
return
for opt in kwargs:
if opt == "communicationManager":
self.commManagerType = kwargs[opt]
elif opt == "loadBalancer":
self.loadBalancerType = kwargs[opt]
elif opt == "printSummary":
self.printExecSummary = kwargs[opt]
elif opt == "setTraceMode":
self.traceMode = kwargs[opt]
elif self.commThread.isRootWorker:
_logger.warning("Unknown option '%s'", opt)
def start(self, initialTarget, *args, **kwargs):
"""
Start the execution with the target `initialTarget`.
Calling this function create and launch the first task on the root worker
(defined by the communication manager, for instance, with MPI, the root worker is the worker with rank 0.).
.. warning::
This function must be called only ONCE, and after the target has been parsed by the Python interpreter.
"""
self.isStarted = True
try:
tmpImport = __import__(self.commManagerType, globals(), locals(), ['CommThread'], 0)
if not hasattr(tmpImport, 'CommThread'):
raise ImportError
CommThread = tmpImport.CommThread
except ImportError:
_logger.warning("Warning : %s is not a suitable communication manager. Default to commManagerMpi4py.", self.commManagerType)
tmpImport = __import__('deap.dtm.commManagerMpi4py', globals(), locals(), ['CommThread'], 0)
CommThread = tmpImport.CommThread
try:
tmpImport = __import__(self.loadBalancerType, globals(), locals(), ['LoadBalancer'], 0)
if not hasattr(tmpImport, 'LoadBalancer'):
raise ImportError
LoadBalancer = tmpImport.LoadBalancer
except ImportError:
_logger.warning("Warning : %s is not a suitable load balancer. Default to loadBalancerPDB.", self.loadBalancerType)
tmpImport = __import__('deap.dtm.loadBalancerPDB', globals(), locals(), ['LoadBalancer'], 0)
LoadBalancer = tmpImport.LoadBalancer
self.commThread = CommThread(self.recvQueue, self.sendQueue, self.runningFlag, self.commExitNotification, self.commReadyEvent, self.dtmRandom, sys.argv)
self.commThread.start()
self.refTime = self._calibrateExecTime()
self.commReadyEvent.wait()
if self.commThread.isLaunchProcess:
sys.exit()
self.poolSize = self.commThread.poolSize
self.workerId = self.commThread.workerId
self.idGenerator = TaskIdGenerator(self.workerId)
self.loadBalancer = LoadBalancer(self.commThread.iterOverIDs(), self.workerId, self.execQueue, self.dtmRandom)
if self.traceMode:
self.traceLock.acquire()
self.traceRoot = etree.Element("dtm", {"version" : str(0.7), "workerId" : str(self.workerId), "timeBegin" : repr(self.sTime)})
self.traceTasks = etree.SubElement(self.traceRoot, "tasksLog")
self.traceComm = etree.SubElement(self.traceRoot, "commLog")
self.traceLoadB = etree.SubElement(self.traceRoot, "loadBalancerLog")
self.traceLock.release()
self.commThread.setTraceModeOn(self.traceComm)
self.loadBalancer.setTraceModeOn(self.traceLoadB)
if self.commThread.isRootWorker:
if self.traceMode:
# Create the log folder
try:
os.mkdir(DTM_LOGDIR_DEFAULT_NAME)
except OSError:
_logger.warning("Log folder '" + DTM_LOGDIR_DEFAULT_NAME + "' already exists!")
_logger.info("DTM started with %i workers", self.poolSize)
_logger.info("DTM load balancer is %s, and communication manager is %s", self.loadBalancerType, self.commManagerType)
initTask = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = None,
taskIndex = 0,
taskRoute = [self.workerId],
creationTime = time.time(),
target = initialTarget,
args = args,
kwargs = kwargs,
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
self.execQueue.put(initTask)
return self._main()
# The following methods are NOT called by the control thread, but by the EXECUTION THREADS
# All the non-local objects used MUST be thread-safe
def map(self, function, *iterables, **kwargs):
"""
A parallel equivalent of the :func:`map` built-in function. It blocks till the result is ready.
This method chops the iterables into a number of chunks determined by DTM in order to get the most efficient use of the workers.
It takes any number of iterables (though it will shrink all of them to the len of the smallest one), and any others kwargs that will be
transmitted as is to the *function* target.
"""
cThread = threading.currentThread()
currentId = cThread.tid
zipIterable = list(zip(*iterables))
listResults = [None] * len(zipIterable)
listTasks = []
listTids = []
for index, elem in enumerate(zipIterable):
task = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = currentId,
taskIndex = index,
taskRoute = [self.workerId],
creationTime = time.time(),
target = function,
args = elem,
kwargs = kwargs,
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
listTasks.append(task)
listTids.append(task.tid)
if self.traceMode:
self.traceLock.acquire()
newTaskElem = etree.SubElement(cThread.xmlTrace, "event",
{"type" : "map",
"time" : repr(time.time()),
"target" : str(function.__name__),
"childTasks" : str(listTids)})
self.traceLock.release()
self.waitingThreadsLock.acquire()
if currentId not in self.waitingThreads.keys():
self.waitingThreads[currentId] = WaitInfoContainer(threadObject = cThread,
eventObject = cThread.waitingFlag,
waitBeginningTime = 0,
tasksWaitingCount = 0,
waitingMode = DTM_WAIT_NONE,
rWaitingDict = {})
resultKey = listTids[0]
self.waitingThreads[currentId].rWaitingDict[resultKey] = ExceptedResultContainer(tids = listTids,
waitingOn = True,
finished = False,
success = False,
callbackFunc = None,
result = listResults)
self.waitingThreads[currentId].tasksWaitingCount += len(listTasks)
self.waitingThreads[currentId].waitingMode = DTM_WAIT_SOME
self.waitingThreadsQueue.put(cThread.taskInfo)
self.waitingThreads[currentId].waitBeginningTime = time.time()
self.waitingThreadsLock.release()
self.execQueue.putList(listTasks)
cThread.waitForResult()
self.waitingThreadsLock.acquire()
ret = self.waitingThreads[currentId].rWaitingDict[resultKey].result
if self.waitingThreads[currentId].rWaitingDict[resultKey].success == False:
# Exception occured
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
raise ret
else:
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
return ret
def map_async(self, function, iterable, callback=None):
"""
A non-blocking variant of the :func:`~deap.dtm.taskmanager.Control.map` method which returns a :class:`~deap.dtm.taskmanager.AsyncResult` object.
.. note::
As on version 0.2, callback is not implemented.
"""
cThread = threading.currentThread()
currentId = cThread.tid
listResults = [None] * len(iterable)
listTasks = []
listTids = []
for index, elem in enumerate(iterable):
task = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = currentId,
taskIndex = index,
taskRoute = [self.workerId],
creationTime = time.time(),
target = function,
args = (elem,),
kwargs = {},
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
listTasks.append(task)
listTids.append(task.tid)
resultKey = listTids[0]
if self.traceMode:
newTaskElem = etree.SubElement(cThread.xmlTrace, "event",
{"type" : "map_async",
"time" : repr(time.time()),
"target" : str(function.__name__),
"childTasks" : str(listTids)})
self.waitingThreadsLock.acquire()
if currentId not in self.waitingThreads.keys():
self.waitingThreads[currentId] = WaitInfoContainer(threadObject = cThread,
eventObject = cThread.waitingFlag,
waitBeginningTime = 0,
tasksWaitingCount = 0,
waitingMode = DTM_WAIT_NONE,
rWaitingDict = {})
self.waitingThreads[currentId].rWaitingDict[resultKey] = ExceptedResultContainer(tids = listTids,
waitingOn = False,
finished = False,
success = False,
callbackFunc = None,
result = listResults)
self.waitingThreads[currentId].waitingMode = DTM_WAIT_NONE
asyncRequest = AsyncResult(self, self.waitingThreads[currentId], resultKey)
self.waitingThreads[currentId].rWaitingDict[resultKey].callbackFunc = asyncRequest._dtmCallback
self.waitingThreadsLock.release()
self.execQueue.putList(listTasks)
self.runningFlag.set()
return asyncRequest
def apply(self, function, *args, **kwargs):
"""
Equivalent of the :func:`apply` built-in function. It blocks till the result is ready.
Given this blocks, :func:`~deap.dtm.taskmanager.Control.apply_async()` is better suited for performing work in parallel.
Additionally, the passed in function is only executed in one of the workers of the pool.
"""
cThread = threading.currentThread()
currentId = cThread.tid
task = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = currentId,
taskIndex = 0,
taskRoute = [self.workerId],
creationTime = time.time(),
target = function,
args = args,
kwargs = kwargs,
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
if self.traceMode:
newTaskElem = etree.SubElement(cThread.xmlTrace, "event",
{"type" : "apply",
"time" : repr(time.time()),
"target" : str(function.__name__),
"childTasks" : str([task.tid])})
self.waitingThreadsLock.acquire()
if currentId not in self.waitingThreads.keys():
self.waitingThreads[currentId] = WaitInfoContainer(threadObject = cThread,
eventObject = cThread.waitingFlag,
waitBeginningTime = 0,
tasksWaitingCount = 0,
waitingMode = DTM_WAIT_NONE,
rWaitingDict = {})
resultKey = task.tid
self.waitingThreads[currentId].rWaitingDict[resultKey] = ExceptedResultContainer(tids = [task.tid],
waitingOn = True,
finished = False,
success = False,
callbackFunc = None,
result = [None])
self.waitingThreads[currentId].tasksWaitingCount += 1
self.waitingThreads[currentId].waitingMode = DTM_WAIT_SOME
self.waitingThreadsQueue.put(cThread.taskInfo)
self.waitingThreads[currentId].waitBeginningTime = time.time()
self.waitingThreadsLock.release()
self.execQueue.put(task)
cThread.waitForResult()
self.waitingThreadsLock.acquire()
ret = self.waitingThreads[currentId].rWaitingDict[resultKey].result
if self.waitingThreads[currentId].rWaitingDict[resultKey].success == False:
# Exception occured
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
raise ret
else:
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
return ret[0]
def apply_async(self, function, *args, **kwargs):
"""
A non-blocking variant of the :func:`~deap.dtm.taskmanager.Control.apply` method which returns a :class:`~deap.dtm.taskmanager.AsyncResult` object.
"""
cThread = threading.currentThread()
currentId = cThread.tid
task = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = currentId,
taskIndex = 0,
taskRoute = [self.workerId],
creationTime = time.time(),
target = function,
args = args,
kwargs = kwargs,
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
if self.traceMode:
newTaskElem = etree.SubElement(cThread.xmlTrace, "event",
{"type" : "apply_async",
"time" : repr(time.time()),
"target" : str(function.__name__),
"childTasks" : str([task.tid])})
self.waitingThreadsLock.acquire()
if currentId not in self.waitingThreads.keys():
self.waitingThreads[currentId] = WaitInfoContainer(threadObject = cThread,
eventObject = cThread.waitingFlag,
waitBeginningTime = 0,
tasksWaitingCount = 0,
waitingMode = DTM_WAIT_NONE,
rWaitingDict = {})
resultKey = task.tid
self.waitingThreads[currentId].rWaitingDict[resultKey] = ExceptedResultContainer(tids = [task.tid],
waitingOn = False,
finished = False,
success = False,
callbackFunc = None,
result = [None])
self.waitingThreads[currentId].waitingMode = DTM_WAIT_NONE
asyncRequest = AsyncResult(self, self.waitingThreads[currentId], resultKey)
self.waitingThreads[currentId].rWaitingDict[resultKey].callbackFunc = asyncRequest._dtmCallback
self.waitingThreadsLock.release()
self.execQueue.put(task)
self.runningFlag.set()
return asyncRequest
def imap(self, function, iterable, chunksize=1):
"""
An equivalent of :func:`itertools.imap`.
The chunksize argument can be used to tell DTM how many elements should be computed at the same time.
For very long iterables using a large value for chunksize can make make the job complete much faster than using the default value of 1.
"""
currentIndex = 0
while currentIndex < len(iterable):
maxIndex = currentIndex + chunksize if currentIndex + chunksize < len(iterable) else len(iterable)
asyncResults = [None] * (maxIndex - currentIndex)
for i in range(currentIndex, maxIndex):
asyncResults[i % chunksize] = self.apply_async(function, iterable[i])
for result in asyncResults:
ret = result.get()
yield ret
currentIndex = maxIndex
def imap_unordered(self, function, iterable, chunksize=1):
"""
Not implemented yet.
"""
raise NotImplementedError
def filter(self, function, iterable):
"""
Same behavior as the built-in :func:`filter`. The filtering is done localy, but the computation is distributed.
"""
results = self.map(function, iterable)
return [item for result, item in zip(results, iterable) if result]
def repeat(self, function, n, *args, **kwargs):
"""
Repeat the function *function* *n* times, with given args and keyworded args.
Return a list containing the results.
"""
cThread = threading.currentThread()
currentId = cThread.tid
listResults = [None] * n
listTasks = []
listTids = []
for index in range(n):
task = TaskContainer(tid = self.idGenerator.tid,
creatorWid = self.workerId,
creatorTid = currentId,
taskIndex = index,
taskRoute = [self.workerId],
creationTime = time.time(),
target = function,
args = args,
kwargs = kwargs,
threadObject = None,
taskState = DTM_TASK_STATE_IDLE)
listTasks.append(task)
listTids.append(task.tid)
self.waitingThreadsLock.acquire()
if currentId not in self.waitingThreads.keys():
self.waitingThreads[currentId] = WaitInfoContainer(threadObject = cThread,
eventObject = cThread.waitingFlag,
waitBeginningTime = 0,
tasksWaitingCount = 0,
waitingMode = DTM_WAIT_NONE,
rWaitingDict = {})
resultKey = listTids[0]
self.waitingThreads[currentId].rWaitingDict[resultKey] = ExceptedResultContainer(tids = listTids,
waitingOn = True,
finished = False,
success = False,
callbackFunc = None,
result = listResults)
self.waitingThreads[currentId].tasksWaitingCount += len(listTasks)
self.waitingThreads[currentId].waitingMode = DTM_WAIT_SOME
self.waitingThreadsQueue.put(cThread.taskInfo)
self.waitingThreads[currentId].waitBeginningTime = time.time()
self.waitingThreadsLock.release()
self.execQueue.putList(listTasks)
cThread.waitForResult()
self.waitingThreadsLock.acquire()
ret = self.waitingThreads[currentId].rWaitingDict[resultKey].result
if self.waitingThreads[currentId].rWaitingDict[resultKey].success == False:
# Exception occured
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
raise ret
else:
del self.waitingThreads[currentId].rWaitingDict[resultKey]
self.waitingThreadsLock.release()
return ret
def waitForAll(self):
"""
Wait for all pending asynchronous results. When this function returns,
DTM guarantees that all ready() call on asynchronous tasks will
return true.
"""
threadId = threading.currentThread().tid
self.waitingThreadsLock.acquire()
if threadId in self.waitingThreads and len(self.waitingThreads[threadId].rWaitingDict) > 0:
self.waitingThreads[threadId].waitingMode = DTM_WAIT_ALL
self.waitingThreadsQueue.put(threading.currentThread().taskInfo)
self.waitingThreadsLock.release()
threading.currentThread().waitForResult()
self.waitingThreadsLock.acquire()
self.waitingThreads[threadId].waitingMode = DTM_WAIT_NONE
self.waitingThreadsLock.release()
else:
self.waitingThreadsLock.release()
return
return None
def testAllAsync(self):
"""
Check whether all pending asynchronous tasks are done.
It does not lock if it is not the case, but returns false.
"""
threadId = threading.currentThread().tid
self.waitingThreadsLock.acquire()
if threadId in self.waitingThreads:
ret = len(self.waitingThreads[threadId].rWaitingDict)
self.waitingThreadsLock.release()
return False
else:
self.waitingThreadsLock.release()
return True
def getWorkerId(self):
"""
Return a unique ID for the current worker. Depending of the
communication manager type, it can be virtually any Python
immutable type.
.. note::
With MPI, the value returned is the MPI slot number.
"""
return self.workerId
class DtmThread(threading.Thread):
"""
DTM execution threads. Those are one of the main parts of DTM.
They should not be created or called directly by the user.
"""
def __init__(self, structInfo, controlThread, xmlTrace=None):
threading.Thread.__init__(self)
self.taskInfo = structInfo # TaskContainer
self.taskInfo.threadObject = self # Remind that we are the exec thread
self.tid = structInfo.tid
self.t = structInfo.target
self.control = controlThread
self.waitingFlag = threading.Event()
self.waitingFlag.clear()
self.timeExec = 0
self.timeBegin = 0
if structInfo.creatorTid is None:
self.isRootTask = True
else:
self.isRootTask = False
self.xmlTrace = xmlTrace
def run(self):
# The lock is already acquired for us
self.taskInfo.taskState = DTM_TASK_STATE_RUNNING
success = True
self.timeBegin = time.time()
if not self.xmlTrace is None:
# Debug output in xml object
self.control.traceLock.acquire()
etree.SubElement(self.xmlTrace, "event", {"type" : "begin", "worker" : str(self.control.workerId), "time" : repr(self.timeBegin)})
self.control.traceLock.release()
try:
returnedR = self.t(*self.taskInfo.args, **self.taskInfo.kwargs)
except Exception as expc:
returnedR = expc
strWarn = "An exception of type " + str(type(expc)) + " occured on worker " + str(self.control.workerId) + " while processing task " + str(self.tid)
_logger.warning(strWarn)
_logger.warning("This will be transfered to the parent task.")
_logger.warning("Exception details : " + str(expc))
success = False
self.timeExec += time.time() - self.timeBegin
self.control.dtmExecLock.release()
if not self.xmlTrace is None:
# Debug output in xml object
self.control.traceLock.acquire()
etree.SubElement(self.xmlTrace, "event", {"type" : "end", "worker" : str(self.control.workerId), "time" : repr(time.time()), "execTime" : repr(self.timeExec), "success" : str(success)})
self.control.traceLock.release()
if success:
try:
self.control._addTaskStat(self.t.__name__, self.timeExec)
except AttributeError:
self.control._addTaskStat(str(self.t), self.timeExec)
if self.isRootTask:
# Is this task the root task (launch by dtm.start)? If so, we quit
self.control.lastRetValue = (success, returnedR)
self.control.exitSetHere = True
self.control.exitStatus.set()
else:
# Else, tell the communication thread to return the result
resultStruct = ResultContainer(tid = self.tid,
parentTid = self.taskInfo.creatorTid,
taskIndex = self.taskInfo.taskIndex,
execTime = self.timeExec,
success = success,
result = returnedR)
self.control._returnResult(self.taskInfo.creatorWid, resultStruct)
# Tell the control thread that something happened
self.control._startNewTask()
if self.isRootTask:
self.control.runningFlag.set()
self.control.waitingThreadsLock.acquire()
if self.tid in self.control.waitingThreads.keys():
del self.control.waitingThreads[self.tid]
self.control.waitingThreadsLock.release()
self.taskInfo.taskState = DTM_TASK_STATE_FINISH
def waitForResult(self):
# Clear the execution lock, and sleep
beginTimeWait = time.time()
self.timeExec += beginTimeWait - self.timeBegin
self.control.dtmExecLock.release()
self.taskInfo.taskState = DTM_TASK_STATE_WAITING
if not self.xmlTrace is None:
# Debug output in xml object
self.control.traceLock.acquire()
etree.SubElement(self.xmlTrace, "event", {"type" : "sleep", "worker" : str(self.control.workerId), "time" : repr(beginTimeWait)})
self.control.traceLock.release()
self.control._startNewTask()
self.control.runningFlag.set()
self.waitingFlag.wait()
self.waitingFlag.clear()
# At this point, we already have acquired the execution lock
self.taskInfo.taskState = DTM_TASK_STATE_RUNNING
self.timeBegin = time.time()
if not self.xmlTrace is None:
# Debug output in xml object
self.control.traceLock.acquire()
etree.SubElement(self.xmlTrace, "event", {"type" : "wakeUp", "worker" : str(self.control.workerId), "time" : repr(self.timeBegin)})
self.control.traceLock.release()
class AsyncResult(object):
"""
The class of the result returned by :func:`~deap.dtm.taskmanager.Control.map_async()` and :func:`~deap.dtm.taskmanager.Control.apply_async()`.
"""
def __init__(self, control, waitingInfo, taskKey):
self.control = control
self.resultReturned = False
self.resultSuccess = False
self.resultVal = None
self.taskKey = taskKey
self.dictWaitingInfo = waitingInfo
def _dtmCallback(self):
# Used by DTM to inform the object that the job is done
self.resultSuccess = self.dictWaitingInfo.rWaitingDict[self.taskKey].success
self.resultVal = self.dictWaitingInfo.rWaitingDict[self.taskKey].result
self.resultReturned = True
del self.dictWaitingInfo.rWaitingDict[self.taskKey]
def get(self):
"""
Return the result when it arrives.
.. note::
This is a blocking call : caller will wait in this function until the result is ready.
To check for the avaibility of the result, use :func:`~deap.dtm.taskmanager.AsyncResult.ready()`.
"""
if not self.resultReturned:
self.wait()
if self.resultSuccess:
return self.resultVal
else:
raise self.resultVal
def wait(self):
"""
Wait until the result is available
"""
self.control.waitingThreadsLock.acquire()
if self.ready():
# This test MUST be protected by the mutex on waitingThreads
self.control.waitingThreadsLock.release()
return
self.control.waitingThreads[threading.currentThread().tid].waitingMode = DTM_WAIT_SOME
self.control.waitingThreads[threading.currentThread().tid].rWaitingDict[self.taskKey].waitingOn = True
#self.dictWaitingInfo.waitingMode = DTM_WAIT_SOME
#self.dictWaitingInfo.rWaitingDict[self.taskKey].waitingOn = True
self.control.waitingThreadsQueue.put(threading.currentThread().taskInfo)
self.control.waitingThreadsLock.release()
threading.currentThread().waitForResult()
def ready(self):
"""
Return whether the asynchronous task has completed.
"""
return self.resultReturned
def successful(self):
"""
Return whether the task completed without error. Will raise AssertionError if the result is not ready.
"""
if not self.resultReturned:
raise AssertionError("Call AsyncResult.successful() before the results were ready!")
return self.resultSuccess
deap-0.7.1/deap/gp.py 0000644 0000765 0000024 00000064032 11641072614 014562 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""The :mod:`gp` module provides the methods and classes to perform
Genetic Programming with DEAP. It essentially contains the classes to
build a Genetic Program Tree, and the functions to evaluate it.
This module support both strongly and loosely typed GP.
"""
import copy
import random
import base
from itertools import repeat
from collections import defaultdict
# Define the name of type for any types.
__type__ = None
## GP Tree utility functions
def evaluate(expr, pset=None):
"""Evaluate the expression *expr* into a string if *pset* is None
or into Python code if *pset* is not None.
"""
def _stringify(expr):
try:
func = expr[0]
return str(func(*[_stringify(value) for value in expr[1:]]))
except TypeError:
return str(expr)
if not pset is None:
try:
return eval(_stringify(expr), pset.func_dict)
except MemoryError:
raise MemoryError,("DEAP : Error in tree evaluation :"
" Python cannot evaluate a tree with a height bigger than 90. "
"To avoid this problem, you should use bloat control on your "
"operators. See the DEAP documentation for more information. "
"DEAP will now abort.")
else:
return _stringify(expr)
def evaluateADF(seq):
"""Evaluate a list of ADF and return a dict mapping the ADF name with its
lambda function.
"""
adfdict = {}
for i, expr in enumerate(reversed(seq[1:])):
func = lambdify(expr.pset, expr)
adfdict.update({expr.pset.__name__ : func})
for expr2 in reversed(seq[1:i+1]):
expr2.pset.func_dict.update(adfdict)
return adfdict
def lambdify(pset, expr):
"""Return a lambda function of the expression *expr*.
.. note::
This function is a stripped version of the lambdify
function of sympy0.6.6.
"""
expr = evaluate(expr)
args = ",".join(a for a in pset.arguments)
lstr = "lambda %s: %s" % (args, expr)
try:
return eval(lstr, dict(pset.func_dict))
except MemoryError:
raise MemoryError,("DEAP : Error in tree evaluation :"
" Python cannot evaluate a tree with a height bigger than 90. "
"To avoid this problem, you should use bloat control on your "
"operators. See the DEAP documentation for more information. "
"DEAP will now abort.")
def lambdifyList(expr):
"""Return a lambda function created from a list of trees. The first
element of the list is the main tree, and the following elements are
automatically defined functions (ADF) that can be called by the first
tree.
"""
adfdict = evaluateADF(expr)
expr[0].pset.func_dict.update(adfdict)
return lambdify(expr[0].pset, expr[0])
## Loosely + Strongly Typed GP
class Primitive(object):
"""Class that encapsulates a primitive and when called with arguments it
returns the Python code to call the primitive with the arguments.
>>> import operator
>>> pr = Primitive(operator.mul, (int, int), int)
>>> pr("1", "2")
'mul(1, 2)'
"""
def __init__(self, primitive, args, ret = __type__):
self.name = primitive.__name__
self.arity = len(args)
self.args = args
self.ret = ret
args = ", ".join(repeat("%s", self.arity))
self.seq = "%s(%s)" % (self.name, args)
def __call__(self, *args):
return self.seq % args
def __repr__(self):
return self.name
class Operator(Primitive):
"""Class that encapsulates an operator and when called with arguments it
returns the Python code to call the operator with the arguments. It acts
as the Primitive class, but instead of returning a function and its
arguments, it returns an operator and its operands.
>>> import operator
>>> op = Operator(operator.mul, (int, int), int)
>>> op("1", "2")
'(1 * 2)'
>>> op2 = Operator(operator.neg, (int,), int)
>>> op2(1)
'-(1)'
"""
symbols = {"add" : "+", "sub" : "-", "mul" : "*", "div" : "/", "neg" : "-",
"and_" : "and", "or_" : "or", "not_" : "not",
"lt" : "<", "eq" : "==", "gt" : ">", "geq" : ">=", "leq" : "<="}
def __init__(self, operator, args, ret = __type__):
Primitive.__init__(self, operator, args, ret)
if len(args) == 1:
self.seq = "%s(%s)" % (self.symbols[self.name], "%s")
elif len(args) == 2:
self.seq = "(%s %s %s)" % ("%s", self.symbols[self.name], "%s")
else:
raise ValueError("Operator arity can be either 1 or 2.")
class Terminal(object):
"""Class that encapsulates terminal primitive in expression. Terminals can
be symbols, values, or 0-arity functions.
"""
def __init__(self, terminal, ret = __type__):
self.ret = ret
try:
self.value = terminal.__name__
except AttributeError:
self.value = terminal
def __call__(self):
return self
def __repr__(self):
return str(self.value)
class Ephemeral(Terminal):
"""Class that encapsulates a terminal which value is set at run-time.
The value of the `Ephemeral` can be regenerated with the method `regen`.
"""
def __init__(self, func, ret = __type__):
self.func = func
Terminal.__init__(self, self.func(), ret)
def regen(self):
"""Regenerate the ephemeral value."""
self.value = self.func()
class EphemeralGenerator(object):
"""Class that generates `Ephemeral` to be added to an expression."""
def __init__(self, ephemeral, ret = __type__):
self.ret = ret
self.name = ephemeral.__name__
self.func = ephemeral
def __call__(self):
return Ephemeral(self.func, self.ret)
def __repr__(self):
return self.name
class PrimitiveSetTyped(object):
"""Class that contains the primitives that can be used to solve a
Strongly Typed GP problem. The set also defined the researched
function return type, and input arguments type and number.
"""
def __init__(self, name, in_types, ret_type, prefix = "ARG"):
self.terminals = defaultdict(list)
self.primitives = defaultdict(list)
self.arguments = []
self.func_dict = dict()
self.terms_count = 0
self.prims_count = 0
self.adfs_count = 0
self.__name__ = name
self.ret = ret_type
self.ins = in_types
for i, type_ in enumerate(in_types):
self.arguments.append(prefix + ("%s" % i))
PrimitiveSetTyped.addTerminal(self, self.arguments[-1], type_)
def renameArguments(self, new_args):
"""Rename function arguments with new arguments name *new_args*.
"""
for i, argument in enumerate(self.arguments):
if new_args.has_key(argument):
self.arguments[i] = new_args[argument]
for terminals in self.terminals.values():
for terminal in terminals:
if ( isinstance(terminal, Terminal) and
new_args.has_key(terminal.value) ):
terminal.value = new_args[terminal.value]
def addPrimitive(self, primitive, in_types, ret_type):
"""Add a primitive to the set.
*primitive* is a callable object or a function.
*in_types* is a list of argument's types the primitive takes.
*ret_type* is the type returned by the primitive.
"""
try:
prim = Operator(primitive, in_types, ret_type)
except (KeyError, ValueError):
prim = Primitive(primitive, in_types, ret_type)
self.primitives[ret_type].append(prim)
self.func_dict[primitive.__name__] = primitive
self.prims_count += 1
def addTerminal(self, terminal, ret_type):
"""Add a terminal to the set.
*terminal* is an object, or a function with no arguments.
*ret_type* is the type of the terminal.
"""
if callable(terminal):
self.func_dict[terminal.__name__] = terminal
prim = Terminal(terminal, ret_type)
self.terminals[ret_type].append(prim)
self.terms_count += 1
def addEphemeralConstant(self, ephemeral, ret_type):
"""Add an ephemeral constant to the set. An ephemeral constant
is a no argument function that returns a random value. The value
of the constant is constant for a Tree, but may differ from one
Tree to another.
*ephemeral* function with no arguments that returns a random value.
*ret_type* is the type of the object returned by the function.
"""
prim = EphemeralGenerator(ephemeral, ret_type)
self.terminals[ret_type].append(prim)
self.terms_count += 1
def addADF(self, adfset):
"""Add an Automatically Defined Function (ADF) to the set.
*adfset* is a PrimitiveSetTyped containing the primitives with which
the ADF can be built.
"""
prim = Primitive(adfset, adfset.ins, adfset.ret)
self.primitives[adfset.ret].append(prim)
self.prims_count += 1
@property
def terminalRatio(self):
"""Return the ratio of the number of terminals on the number of all
kind of primitives.
"""
return self.terms_count / float(self.terms_count + self.prims_count)
class PrimitiveSet(PrimitiveSetTyped):
"""Class same as :class:`~deap.gp.PrimitiveSetTyped`, except there is no
definition of type.
"""
def __init__(self, name, arity, prefix="ARG"):
args = [__type__]*arity
PrimitiveSetTyped.__init__(self, name, args, __type__, prefix)
def addPrimitive(self, primitive, arity):
"""Add primitive *primitive* with arity *arity* to the set."""
assert arity > 0, "arity should be >= 1"
args = [__type__] * arity
PrimitiveSetTyped.addPrimitive(self, primitive, args, __type__)
def addTerminal(self, terminal):
"""Add a terminal to the set."""
PrimitiveSetTyped.addTerminal(self, terminal, __type__)
def addEphemeralConstant(self, ephemeral):
"""Add an ephemeral constant to the set."""
PrimitiveSetTyped.addEphemeralConstant(self, ephemeral, __type__)
class PrimitiveTree(base.Tree):
"""Tree class faster than base Tree, optimized for Primitives."""
pset = None
def _getstate(self):
state = []
for elem in self:
try:
state.append(elem._getstate())
except AttributeError:
state.append(elem)
return state
def __deepcopy__(self, memo):
"""Deepcopy a Tree by first converting it back to a list of list.
This deepcopy is faster than the default implementation. From
quick testing, up to 1.6 times faster, and at least 2 times less
function calls.
"""
new = self.__class__(self._getstate())
new.__dict__.update(copy.deepcopy(self.__dict__, memo))
return new
# Expression generation functions
def genFull(pset, min_, max_, type_=__type__):
"""Generate an expression where each leaf has a the same depth
between *min* and *max*.
"""
def condition(max_depth):
"""Expression generation stops when the depth is zero."""
return max_depth == 0
return _generate(pset, min_, max_, condition, type_)
def genGrow(pset, min_, max_, type_=__type__):
"""Generate an expression where each leaf might have a different depth
between *min* and *max*.
"""
def condition(max_depth):
"""Expression generation stops when the depth is zero or when
it is randomly determined that a a node should be a terminal.
"""
return max_depth == 0 or random.random() < pset.terminalRatio
return _generate(pset, min_, max_, condition, type_)
def genRamped(pset, min_, max_, type_=__type__):
"""Generate an expression with a PrimitiveSet *pset*.
Half the time, the expression is generated with :func:`~deap.gp.genGrow`,
the other half, the expression is generated with :func:`~deap.gp.genFull`.
"""
method = random.choice((genGrow, genFull))
return method(pset, min_, max_, type_)
def _generate(pset, min_, max_, condition, type_=__type__):
def genExpr(max_depth, type_):
if condition(max_depth):
term = random.choice(pset.terminals[type_])
expr = term()
else:
prim = random.choice(pset.primitives[type_])
expr = [prim]
args = (genExpr(max_depth-1, arg) for arg in prim.args)
expr.extend(args)
return expr
max_depth = random.randint(min_, max_)
expr = genExpr(max_depth, type_)
if not isinstance(expr, list):
expr = [expr]
return expr
######################################
# GP Crossovers #
######################################
def cxUniformOnePoint(ind1, ind2):
"""Randomly select in each individual and exchange each subtree with the
point as root between each individual.
"""
try:
index1 = random.randint(1, ind1.size-1)
index2 = random.randint(1, ind2.size-1)
except ValueError:
return ind1, ind2
sub1 = ind1.searchSubtreeDF(index1)
sub2 = ind2.searchSubtreeDF(index2)
ind1.setSubtreeDF(index1, sub2)
ind2.setSubtreeDF(index2, sub1)
return ind1, ind2
## Strongly Typed GP crossovers
def cxTypedOnePoint(ind1, ind2):
"""Randomly select in each individual and exchange each subtree with the
point as root between each individual. Since the node are strongly typed,
the operator then make sure the type of second node correspond to the type
of the first node. If it doesn't, it randomly selects another point in the
second individual and try again. It tries up to *5* times before
giving up on the crossover.
.. note::
This crossover is subject to change for a more effective method
of selecting the crossover points.
"""
# choose the crossover point in each individual
try:
index1 = random.randint(1, ind1.size-1)
index2 = random.randint(1, ind2.size-1)
except ValueError:
return ind1, ind2
subtree1 = ind1.searchSubtreeDF(index1)
subtree2 = ind2.searchSubtreeDF(index2)
type1 = subtree1.root.ret
type2 = subtree2.root.ret
# try to mate the trees
# if no crossover point is found after 5 it gives up trying
# mating individuals.
tries = 0
MAX_TRIES = 5
while not (type1 == type2) and tries < MAX_TRIES:
index2 = random.randint(1, ind2.size-1)
subtree2 = ind2.searchSubtreeDF(index2)
type2 = subtree2.root.ret
tries += 1
if type1 == type2:
sub1 = ind1.searchSubtreeDF(index1)
sub2 = ind2.searchSubtreeDF(index2)
ind1.setSubtreeDF(index1, sub2)
ind2.setSubtreeDF(index2, sub1)
return ind1, ind2
def cxOnePointLeafBiased(ind1, ind2, cxtermpb):
"""Randomly select crossover point in each individual and exchange each
subtree with the point as root between each individual.
This operator takes another parameter *cxtermpb*, which set the probability
to choose between a terminal or non-terminal crossover point.
For instance, as defined by Koza, non-terminal primitives are selected for
90% of the crossover points, and terminals for 10%, so *cxtermpb* should be
set to 0.1.
"""
size1, size2 = ind1.size, ind2.size
if size1 == 1 or size2 == 1:
return ind1, ind2
# Those were not implemented with set because random.choice()
# works only on sequencable iterables (it is not clear whether
# it would be faster to perform the conversion set->list or
# directly use lists)
termsList1 = [termIndex for termIndex in ind1.iter_leaf_idx]
termsList2 = [termIndex for termIndex in ind2.iter_leaf_idx]
primList1 = [i for i in xrange(1,size1) if i not in termsList1]
primList2 = [i for i in xrange(1,size2) if i not in termsList2]
if random.random() < cxtermpb or len(primList1) == 0:
# Choose a terminal from the first parent
index1 = random.choice(termsList1)
subtree1 = ind1.searchSubtreeDF(index1)
else:
# Choose a primitive (non-terminal) from the first parent
index1 = random.choice(primList1)
subtree1 = ind1.searchSubtreeDF(index1)
if random.random() < cxtermpb or len(primList2) == 0:
# Choose a terminal from the second parent
index2 = random.choice(termsList2)
subtree2 = ind2.searchSubtreeDF(index2)
else:
# Choose a primitive (non-terminal) from the second parent
index2 = random.choice(primList2)
subtree2 = ind2.searchSubtreeDF(index2)
ind1.setSubtreeDF(index1, subtree2)
ind2.setSubtreeDF(index2, subtree1)
return ind1, ind2
## Strongly Typed GP crossovers
def cxTypedOnePointLeafBiased(ind1, ind2, cxtermpb):
"""Randomly select crossover point in each individual and exchange each
subtree with the point as root between each individual. Since the node are
strongly typed, the operator then make sure the type of second node
correspond to the type of the first node. If it doesn't, it randomly
selects another point in the second individual and try again. It tries up
to *5* times before giving up on the crossover.
This operator takes another parameter *cxtermpb*, which set the probability
to choose between a terminal or non-terminal crossover point.
For instance, as defined by Koza, non-terminal primitives are selected for
90% of the crossover points, and terminals for 10%, so *cxtermpb* should be
set to 0.1.
.. note::
This crossover is subject to change for a more effective method
of selecting the crossover points.
"""
size1, size2 = ind1.size, ind2.size
if size1 == 1 or size2 == 1:
return ind1, ind2
# Those were not implemented with set because random.choice()
# works only on sequencable iterables (it is not clear whether
# it would be faster to perform the conversion set->list or
# directly use lists)
termsList1 = [termIndex for termIndex in ind1.iter_leaf_idx]
termsList2 = [termIndex for termIndex in ind2.iter_leaf_idx]
primList1 = [i for i in xrange(1,size1) if i not in termsList1]
primList2 = [i for i in xrange(1,size2) if i not in termsList2]
if random.random() < cxtermpb or len(primList1) == 0:
# Choose a terminal from the first parent
index1 = random.choice(termsList1)
subtree1 = ind1.searchSubtreeDF(index1)
else:
# Choose a primitive (non-terminal) from the first parent
index1 = random.choice(primList1)
subtree1 = ind1.searchSubtreeDF(index1)
if random.random() < cxtermpb or len(primList2) == 0:
# Choose a terminal from the second parent
index2 = random.choice(termsList2)
subtree2 = ind2.searchSubtreeDF(index2)
else:
# Choose a primitive (non-terminal) from the second parent
index2 = random.choice(primList2)
subtree2 = ind2.searchSubtreeDF(index2)
type1 = subtree1.root.ret
type2 = subtree2.root.ret
# try to mate the trees
# if no crossover point is found after MAX_CX_TRY
# the children are returned without modifications.
tries = 0
MAX_CX_TRY = 5
while not (type1 is type2) and tries != MAX_CX_TRY:
if random.random() < cxtermpb or len(primList2) == 0:
index2 = random.choice(termsList2)
subtree2 = ind2.searchSubtreeDF(index2)
else:
index2 = random.choice(primList2)
subtree2 = ind2.searchSubtreeDF(index2)
type2 = subtree2.root.ret
tries += 1
if type1 is type2:
ind1.setSubtreeDF(index1, subtree2)
ind2.setSubtreeDF(index2, subtree1)
return ind1, ind2
######################################
# GP Mutations #
######################################
def mutUniform(individual, expr):
"""Randomly select a point in the Tree, then replace the subtree with
the point as a root by a randomly generated expression. The expression
is generated using the method `expr`.
"""
index = random.randint(0, individual.size-1)
individual.setSubtreeDF(index, expr(pset=individual.pset))
return individual,
## Strongly Typed GP mutations
def mutTypedUniform(individual, expr):
"""The mutation of strongly typed GP expression is pretty easy. First,
it finds a subtree. Second, it has to identify the return type of the root
of this subtree. Third, it generates a new subtree with a root's type
corresponding to the original subtree root's type. Finally, the old
subtree is replaced by the new subtree.
"""
index = random.randint(0, individual.size-1)
subtree = individual.searchSubtreeDF(index)
individual.setSubtreeDF(index, expr(pset=individual.pset,
type_=subtree.root.ret))
return individual,
def mutTypedNodeReplacement(individual):
"""This operator mutates the individual *individual* that are subjected to
it. The operator randomly chooses a primitive in the individual
and replaces it with a randomly selected primitive in *pset* that takes
the same number of arguments.
This operator works on strongly typed trees as on normal GP trees.
"""
if individual.size < 2:
return individual,
index = random.randint(1, individual.size-1)
node = individual.searchSubtreeDF(index)
if node.size == 1:
subtree = random.choice(individual.pset.terminals[node.root.ret])()
individual.setSubtreeDF(index, subtree)
else:
# We're going to replace one of the *node* children
index = random.randint(1, len(node) - 1)
if node[index].size > 1:
prim_set = individual.pset.primitives[node[index].root.ret]
repl_node = random.choice(prim_set)
while repl_node.args != node[index].root.args:
repl_node = random.choice(prim_set)
node[index][0] = repl_node
else:
term_set = individual.pset.terminals[node[index].root.ret]
repl_node = random.choice(term_set)()
node[index] = repl_node
return individual,
def mutTypedEphemeral(individual, mode):
"""This operator works on the constants of the tree *ind*.
In *mode* ``"one"``, it will change the value of **one**
of the individual ephemeral constants by calling its generator function.
In *mode* ``"all"``, it will change the value of **all**
the ephemeral constants.
This operator works on strongly typed trees as on normal GP trees.
"""
if mode not in ["one", "all"]:
raise ValueError("Mode must be one of \"one\" or \"all\"")
ephemerals = []
for i in xrange(1, individual.size):
subtree = individual.searchSubtreeDF(i)
if hasattr(subtree.root.obj, 'regen'):
ephemerals.append(i)
if len(ephemerals) > 0:
if mode == "one":
ephemerals = [random.choice(ephemerals)]
elif mode == "all":
pass
for i in ephemerals:
individual.searchSubtreeDF(i).regen()
return individual,
def mutShrink(individual):
"""This operator shrinks the individual *individual* that are subjected to
it. The operator randomly chooses a branch in the individual and replaces
it with one of the branch's arguments (also randomly chosen).
This operator is not usable with STGP.
"""
if individual.size < 3 or individual.height <= 2:
return individual, # We don't want to "shrink" the root
index = random.randint(1, individual.size-2)
# Shrinking a terminal is useless
while individual.searchSubtreeDF(index).size == 1:
index = random.randint(1, individual.size-2)
deleted_node = individual.searchSubtreeDF(index)
repl_subtree_index = random.randint(1, len(deleted_node)-1)
individual.setSubtreeDF(index, deleted_node[repl_subtree_index])
return individual,
def mutTypedInsert(individual):
"""This operator mutate the GP tree of the *individual* passed and the
primitive set *expr*, by inserting a new branch at a random position in a
tree, using the original subtree at this position as one argument,
and if necessary randomly selecting terminal primitives
to complete the arguments of the inserted node.
Note that the original subtree will become one of the children of the new
primitive inserted, but not perforce the first (its position is
randomly selected if the new primitive has more than one child)
This operator works on strongly typed trees as on normal GP trees.
"""
pset = individual.pset
index = random.randint(0, individual.size-1)
node = individual.searchSubtreeDF(index)
if node.size > 1: # We do not need to deepcopy the leafs
node = copy.deepcopy(node)
new_primitive = random.choice(pset.primitives[node.root.ret])
inserted_list = [new_primitive]
for i in xrange(0, new_primitive.arity):
# Why don't we use expr to create the other subtrees?
# Bloat control?
new_child = random.choice(pset.terminals[new_primitive.args[i]])
inserted_list.append(new_child())
inserted_list[random.randint(1, new_primitive.arity)] = node
individual.setSubtreeDF(index, inserted_list)
return individual,
if __name__ == "__main__":
import doctest
doctest.testmod()
deap-0.7.1/deap/tests/ 0000755 0000765 0000024 00000000000 11650301263 014732 5 ustar felix staff 0000000 0000000 deap-0.7.1/deap/tests/__init__.py 0000644 0000765 0000024 00000001265 11641072614 017054 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
deap-0.7.1/deap/tests/test_pickle.py 0000644 0000765 0000024 00000007734 11641072614 017632 0 ustar felix staff 0000000 0000000
import sys
import unittest
import array
import pickle
import operator
from test import test_support
sys.path.append("..")
import creator
import base
import gp
import tools
def func():
return "True"
class Pickling(unittest.TestCase):
def setUp(self):
creator.create("FitnessMax", base.Fitness, weights=(1.0,))
creator.create("IndList", list, fitness=creator.FitnessMax)
creator.create("IndArray", array.array, typecode='f', fitness=creator.FitnessMax)
creator.create("IndTree", base.Tree, fitness=creator.FitnessMax)
self.toolbox = base.Toolbox()
self.toolbox.register("func", func)
self.toolbox.register("lambda_func", lambda: "True")
def test_pickle_fitness(self):
fitness = creator.FitnessMax()
fitness.values = (1.0,)
fitness_s = pickle.dumps(fitness)
fitness_l = pickle.loads(fitness_s)
self.failUnlessEqual(fitness, fitness_l, "Unpickled fitness != pickled fitness")
def test_pickle_ind_list(self):
ind = creator.IndList([1.0, 2.0, 3.0])
ind.fitness.values = (4.0,)
ind_s = pickle.dumps(ind)
ind_l = pickle.loads(ind_s)
self.failUnlessEqual(ind, ind_l, "Unpickled individual list != pickled individual list")
self.failUnlessEqual(ind.fitness, ind_l.fitness, "Unpickled individual fitness != pickled individual fitness")
def test_pickle_ind_array(self):
ind = creator.IndArray([1.0, 2.0, 3.0])
ind.fitness.values = (4.0,)
ind_s = pickle.dumps(ind)
ind_l = pickle.loads(ind_s)
self.failUnlessEqual(ind, ind_l, "Unpickled individual array != pickled individual array")
self.failUnlessEqual(ind.fitness, ind_l.fitness, "Unpickled individual fitness != pickled individual fitness")
def test_pickle_tree(self):
ind = creator.IndTree(["+", 1, 2])
ind.fitness.values = (1.0,)
ind_s = pickle.dumps(ind)
ind_l = pickle.loads(ind_s)
msg = "Unpickled individual %s != pickled individual %s" % (str(ind), str(ind_l))
self.failUnlessEqual(ind, ind_l, msg)
msg = "Unpickled fitness %s != pickled fitness %s" % (str(ind.fitness), str(ind_l.fitness))
self.failUnlessEqual(ind.fitness, ind_l.fitness, msg)
def test_pickle_population(self):
ind1 = creator.IndList([1.0,2.0,3.0])
ind1.fitness.values = (1.0,)
ind2 = creator.IndList([4.0,5.0,6.0])
ind2.fitness.values = (2.0,)
ind3 = creator.IndList([7.0,8.0,9.0])
ind3.fitness.values = (3.0,)
pop = [ind1, ind2, ind3]
pop_s = pickle.dumps(pop)
pop_l = pickle.loads(pop_s)
self.failUnlessEqual(pop[0], pop_l[0], "Unpickled individual list != pickled individual list")
self.failUnlessEqual(pop[0].fitness, pop_l[0].fitness, "Unpickled individual fitness != pickled individual fitness")
self.failUnlessEqual(pop[1], pop_l[1], "Unpickled individual list != pickled individual list")
self.failUnlessEqual(pop[1].fitness, pop_l[1].fitness, "Unpickled individual fitness != pickled individual fitness")
self.failUnlessEqual(pop[2], pop_l[2], "Unpickled individual list != pickled individual list")
self.failUnlessEqual(pop[2].fitness, pop_l[2].fitness, "Unpickled individual fitness != pickled individual fitness")
if not sys.version_info < (2, 7):
def test_pickle_partial(self):
func_s = pickle.dumps(self.toolbox.func)
func_l = pickle.loads(func_s)
self.failUnlessEqual(self.toolbox.func(), func_l())
@unittest.expectedFailure
def test_pickle_lambda(self):
func_s = pickle.dumps(self.toolbox.lambda_func)
func_l = pickle.loads(func_s)
self.failUnlessEqual(self.toolbox.lambda_func(), func_l())
if __name__ == "__main__":
suite = unittest.TestLoader().loadTestsFromTestCase(Pickling)
unittest.TextTestRunner(verbosity=2).run(suite)
deap-0.7.1/deap/tools.py 0000644 0000765 0000024 00000154615 11641072614 015323 0 ustar felix staff 0000000 0000000 # This file is part of DEAP.
#
# DEAP is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as
# published by the Free Software Foundation, either version 3 of
# the License, or (at your option) any later version.
#
# DEAP is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with DEAP. If not, see .
"""The :mod:`~deap.tools` module contains the operators for evolutionary
algorithms. They are used to modify, select and move the individuals in their
environment. The set of operators it contains are readily usable in the
:class:`~deap.base.Toolbox`. In addition to the basic operators this module
also contains utility tools to enhance the basic algorithms with
:class:`Statistics`, :class:`HallOfFame`, :class:`Checkpoint`, and
:class:`History`.
"""
from __future__ import division
import bisect
import copy
import inspect
import math
import random
from itertools import chain
from operator import attrgetter, eq
from collections import defaultdict
from functools import partial
try:
import yaml
CHECKPOINT_USE_YAML = True
try:
from yaml import CDumper as Dumper # CLoader and CDumper are much
from yaml import CLoader as Loader # faster than default ones, but
except ImportError: # requires LibYAML to be compiled
from yaml import Dumper
from yaml import Loader
except ImportError:
CHECKPOINT_USE_YAML = False
# If yaml ain't present, use
try: # pickling to dump
import cPickle as pickle # cPickle is much faster than
except ImportError: # pickle but only present under
import pickle # CPython
def initRepeat(container, func, n):
"""Call the function *container* with a generator function corresponding
to the calling *n* times the function *func*.
This helper function can can be used in conjunction with a Toolbox
to register a generator of filled containers, as individuals or
population.
>>> initRepeat(list, random.random, 2) # doctest: +ELLIPSIS,
... # doctest: +NORMALIZE_WHITESPACE
[0.4761..., 0.6302...]
"""
return container(func() for _ in xrange(n))
def initIterate(container, generator):
"""Call the function *container* with an iterable as
its only argument. The iterable must be returned by
the method or the object *generator*.
This helper function can can be used in conjunction with a Toolbox
to register a generator of filled containers, as individuals or
population.
>>> from random import sample
>>> from functools import partial
>>> gen_idx = partial(sample, range(10), 10)
>>> initIterate(list, gen_idx)
[4, 5, 3, 6, 0, 9, 2, 7, 1, 8]
"""
return container(generator())
def initCycle(container, seq_func, n=1):
"""Call the function *container* with a generator function corresponding
to the calling *n* times the functions present in *seq_func*.
This helper function can can be used in conjunction with a Toolbox
to register a generator of filled containers, as individuals or
population.
>>> func_seq = [lambda:1 , lambda:'a', lambda:3]
>>> initCycle(list, func_seq, 2)
[1, 'a', 3, 1, 'a', 3]
"""
return container(func() for _ in xrange(n) for func in seq_func)
class History(object):
"""The :class:`History` class helps to build a genealogy of all the
individuals produced in the evolution. It contains two attributes,
the :attr:`genealogy_tree` that is a dictionary of lists indexed by
individual, the list contain the indices of the parents. The second
attribute :attr:`genealogy_history` contains every individual indexed
by their individual number as in the genealogy tree.
The produced genealogy tree is compatible with `NetworkX
`_, here is how to plot the genealogy
tree ::
hist = History()
# Do some evolution and fill the history
import matplotlib.pyplot as plt
import networkx as nx
g = nx.DiGraph(hist.genealogy_tree)
nx.draw_springs(g)
plt.show()
.. note::
The genealogy tree might get very big if your population and/or the
number of generation is large.
"""
def __init__(self):
self.genealogy_index = 0
self.genealogy_history = dict()
self.genealogy_tree = dict()
def populate(self, individuals):
"""Populate the history with the initial *individuals*. An attribute
:attr:`history_index` is added to every individual, this index will
help to track the parents and the children through evolution. This
index will be modified by the :meth:`update` method when a child is
produced. Modifying the internal :attr:`genealogy_index` of the
history or the :attr:`history_index` of an individual may lead to
unpredictable results and corruption of the history.
"""
for ind in individuals:
self.genealogy_index += 1
ind.history_index = self.genealogy_index
self.genealogy_history[self.genealogy_index] = copy.deepcopy(ind)
self.genealogy_tree[self.genealogy_index] = list()
def update(self, *individuals):
"""Update the history with the new *individuals*. The index present
in their :attr:`history_index` attribute will be used to locate their
parents and modified to a unique one to keep track of those new
individuals.
"""
parent_indices = [ind.history_index for ind in individuals]
for ind in individuals:
self.genealogy_index += 1
ind.history_index = self.genealogy_index
self.genealogy_history[self.genealogy_index] = copy.deepcopy(ind)
self.genealogy_tree[self.genealogy_index] = parent_indices
@property
def decorator(self):
"""Property that returns an appropriate decorator to enhance the
operators of the toolbox. The returned decorator assumes that the
individuals are returned by the operator. First the decorator calls
the underlying operation and then calls the update function with what
has been returned by the operator as argument. Finally, it returns
the individuals with their history parameters modified according to
the update function.
"""
def decFunc(func):
def wrapFunc(*args, **kargs):
individuals = func(*args, **kargs)
self.update(*individuals)
return individuals
return wrapFunc
return decFunc
class Checkpoint(object):
"""A checkpoint is a file containing the state of any object that has been
hooked. While initializing a checkpoint, add the objects that you want to
be dumped by appending keyword arguments to the initializer or using the
:meth:`add`. By default the checkpoint tries to use the YAML format which
is human readable, if PyYAML is not installed, it uses pickling which is
not readable. You can force the use of pickling by setting the argument
*yaml* to :data:`False`.
In order to use efficiently this module, you must understand properly the
assignment principles in Python. This module use the *pointers* you passed
to dump the object, for example the following won't work as desired ::
>>> my_object = [1, 2, 3]
>>> cp = Checkpoint(obj=my_object)
>>> my_object = [3, 5, 6]
>>> cp.dump("example")
>>> cp.load("example.ems")
>>> cp["obj"]
[1, 2, 3]
In order to dump the new value of ``my_object`` it is needed to change its
internal values directly and not touch the *label*, as in the following ::
>>> my_object = [1, 2, 3]
>>> cp = Checkpoint(obj=my_object)
>>> my_object[:] = [3, 5, 6]
>>> cp.dump("example")
>>> cp.load("example.ems")
>>> cp["obj"]
[3, 5, 6]
"""
def __init__(self, yaml=True, **kargs):
# self.zipped = zipped
self._dict = kargs
if CHECKPOINT_USE_YAML and yaml:
self.use_yaml = True
else:
self.use_yaml = False
def add(self, **kargs):
"""Add objects to the list of objects to be dumped. The object is
added under the name specified by the argument's name. Keyword
arguments are mandatory in this function.
"""
self._dict.update(*kargs)
def remove(self, *args):
"""Remove objects with the specified name from the list of objects to
be dumped.
"""
for element in args:
del self._dict[element]
def __getitem__(self, value):
return self._dict[value]
def dump(self, prefix):
"""Dump the current registered objects in a file named *prefix.ecp*,
the randomizer state is always added to the file and available under
the ``"randomizer_state"`` tag.
"""
# if not self.zipped:
cp_file = open(prefix + ".ecp", "w")
# else:
# file = gzip.open(prefix + ".ems.gz", "w")
cp = self._dict.copy()
cp.update({"randomizer_state" : random.getstate()})
if self.use_yaml:
cp_file.write(yaml.dump(ms, Dumper=Dumper))
else:
pickle.dump(cp, cp_file)
cp_file.close()
def load(self, filename):
"""Load a checkpoint file retrieving the dumped objects, it is not
safe to load a checkpoint file in a checkpoint object that contains
references as all conflicting names will be updated with the new
values.
"""
if self.use_yaml:
self._dict.update(yaml.load(open(filename, "r"), Loader=Loader))
else:
self._dict.update(pickle.load(open(filename, "r")))
def mean(seq):
"""Returns the arithmetic mean of the sequence *seq* =
:math:`\{x_1,\ldots,x_n\}` as :math:`A = \\frac{1}{n} \sum_{i=1}^n x_i`.
"""
return sum(seq) / len(seq)
def median(seq):
"""Returns the median of *seq* - the numeric value separating the higher half
of a sample from the lower half. If there is an even number of elements in
*seq*, it returns the mean of the two middle values.
"""
sseq = sorted(seq)
length = len(seq)
if length % 2 == 1:
return sseq[int((length - 1) / 2)]
else:
return (sseq[int((length - 1) / 2)] + sseq[int(length / 2)]) / 2
def var(seq):
"""Returns the variance :math:`\sigma^2` of *seq* =
:math:`\{x_1,\ldots,x_n\}` as
:math:`\sigma^2 = \\frac{1}{N} \sum_{i=1}^N (x_i - \\mu )^2`,
where :math:`\\mu` is the arithmetic mean of *seq*.
"""
return abs(sum(x*x for x in seq) / len(seq) - mean(seq)**2)
def std(seq):
"""Returns the square root of the variance :math:`\sigma^2` of *seq*.
"""
return var(seq)**0.5
class Statistics(object):
"""A statistics object that holds the required data for as long as it
exists. When created the statistics object receives a *key* argument that
is used to get the required data, if not provided the *key* argument
defaults to the identity function. A statistics object can be represented
as a 4 dimensional matrix. Along the first axis (wich length is given by
the *n* argument) are independent statistics objects that are used on
different collections given this index in the :meth:`update` method. The
second axis is the function it-self, each element along the second axis
(indexed by their name) will represent a different function. The third
axis is the accumulator of statistics. each time the update function is
called the new statistics are added using the registered functions at the
end of this axis. The fourth axis is used when the entered data is an
iterable (for example a multiobjective fitness).
Data can be retrieved by different means in a statistics object. One can
use directly the registered function name with an *index* argument that
represent the first axis of the matrix. This method returns the last
entered data.
::
>>> s = Statistics(n=2)
>>> s.register("Mean", mean)
>>> s.update([1, 2, 3, 4], index=0)
>>> s.update([5, 6, 7, 8], index=1)
>>> s.Mean(0)
[2.5]
>>> s.Mean(1)
[6.5]
An other way to obtain the statistics is to use directly the ``[]``. In
that case all dimensions must be specified. This is how stats that have
been registered earlier in the process can be retrieved.
::
>>> s.update([10, 20, 30, 40], index=0)
>>> s.update([50, 60, 70, 80], index=1)
>>> s[0]["Mean"][0]
[2.5]
>>> s[1]["Mean"][0]
[6.5]
>>> s[0]["Mean"][1]
[25]
>>> s[1]["Mean"][1]
[65]
Finally, the fourth dimension is used when stats are needed on lists of
lists. The stats are computed on the matching indices of each list.
::
>>> s = Statistics()
>>> s.register("Mean", mean)
>>> s.update([[1, 2, 3], [4, 5, 6]])
>>> s.Mean()
[2.5, 3.5, 4.5]
>>> s[0]["Mean"][-1][0]
2.5
"""
class Data(defaultdict):
def __init__(self):
defaultdict.__init__(self, list)
def __str__(self):
return "\n".join("%s %s" % (key, ", ".join(map(str, stat[-1]))) for key, stat in self.iteritems())
def __init__(self, key=lambda x: x, n=1):
self.key = key
self.functions = {}
self.data = tuple(self.Data() for _ in xrange(n))
def __getitem__(self, index):
return self.data[index]
def _getFuncValue(self, name, index=0):
return self.data[index][name][-1]
def register(self, name, function):
"""Register a function *function* that will be apply on the sequence
each time :func:`~deap.tools.Statistics.update` is called.
The function result will be accessible by using the string given by
the argument *name* as a function of the statistics object.
>>> s = Statistics()
>>> s.register("Mean", mean)
>>> s.update([1,2,3,4,5,6,7])
>>> s.Mean()
[4.0]
"""
self.functions[name] = function
setattr(self, name, partial(self._getFuncValue, name))
def update(self, seq, index=0):
"""Apply to the input sequence *seq* each registered function
and store each result in a list specific to the function and
the data index *index*.
>>> s = Statistics()
>>> s.register("Mean", mean)
>>> s.register("Max", max)
>>> s.update([4,5,6,7,8])
>>> s.Max()
[8]
>>> s.Mean()
[6.0]
>>> s.update([1,2,3])
>>> s.Max()
[3]
>>> s[0]["Max"]
[[8], [3]]
>>> s[0]["Mean"]
[[6.0], [2.0]]
"""
# Transpose the values
data = self.data[index]
try:
values = zip(*(self.key(elem) for elem in seq))
except TypeError:
values = zip(*[(self.key(elem),) for elem in seq])
for key, func in self.functions.iteritems():
data[key].append(map(func, values))
def __str__(self):
return "\n".join("%s %s" % (key, ", ".join(map(str, stat[-1]))) for key, stat in self.data[-1].iteritems())
class HallOfFame(object):
"""The hall of fame contains the best individual that ever lived in the
population during the evolution. It is sorted at all time so that the
first element of the hall of fame is the individual that has the best
first fitness value ever seen, according to the weights provided to the
fitness at creation time.
The class :class:`HallOfFame` provides an interface similar to a list
(without being one completely). It is possible to retrieve its length, to
iterate on it forward and backward and to get an item or a slice from it.
"""
def __init__(self, maxsize):
self.maxsize = maxsize
self.keys = list()
self.items = list()
def update(self, population):
"""Update the hall of fame with the *population* by replacing the
worst individuals in it by the best individuals present in
*population* (if they are better). The size of the hall of fame is
kept constant.
"""
if len(self) < self.maxsize:
# Items are sorted with the best fitness first
self.items = sorted(chain(self, population),
key=attrgetter("fitness"),
reverse=True)[:self.maxsize]
self.items = [copy.deepcopy(item) for item in self.items]
# The keys are the fitnesses in reverse order to allow the use
# of the bisection algorithm
self.keys = map(attrgetter("fitness"),
reversed(self.items))
else:
for ind in population:
if ind.fitness > self[-1].fitness:
# Delete the worst individual from the front
self.remove(-1)
# Insert the new individual
self.insert(ind)
def insert(self, item):
"""Insert a new individual in the hall of fame using the
:func:`~bisect.bisect_right` function. The inserted individual is
inserted on the right side of an equal individual. Inserting a new
individual in the hall of fame also preserve the hall of fame's order.
This method **does not** check for the size of the hall of fame, in a
way that inserting a new individual in a full hall of fame will not
remove the worst individual to maintain a constant size.
"""
item = copy.deepcopy(item)
i = bisect.bisect_right(self.keys, item.fitness)
self.items.insert(len(self) - i, item)
self.keys.insert(i, item.fitness)
def remove(self, index):
"""Remove the specified *index* from the hall of fame."""
del self.keys[len(self) - (index % len(self) + 1)]
del self.items[index]
def clear(self):
"""Clear the hall of fame."""
del self.items[:]
del self.keys[:]
def __len__(self):
return len(self.items)
def __getitem__(self, i):
return self.items[i]
def __iter__(self):
return iter(self.items)
def __reversed__(self):
return reversed(self.items)
def __str__(self):
return str(self.items) + "\n" + str(self.keys)
class ParetoFront(HallOfFame):
"""The Pareto front hall of fame contains all the non-dominated individuals
that ever lived in the population. That means that the Pareto front hall of
fame can contain an infinity of different individuals.
The size of the front may become very large if it is used for example on
a continuous function with a continuous domain. In order to limit the number
of individuals, it is possible to specify a similarity function that will
return :data:`True` if the genotype of two individuals are similar. In that
case only one of the two individuals will be added to the hall of fame. By
default the similarity function is :func:`operator.__eq__`.
Since, the Pareto front hall of fame inherits from the :class:`HallOfFame`,
it is sorted lexicographically at every moment.
"""
def __init__(self, similar=eq):
self.similar = similar
HallOfFame.__init__(self, None)
def update(self, population):
"""Update the Pareto front hall of fame with the *population* by adding
the individuals from the population that are not dominated by the hall
of fame. If any individual in the hall of fame is dominated it is
removed.
"""
for ind in population:
is_dominated = False
has_twin = False
to_remove = []
for i, hofer in enumerate(self): # hofer = hall of famer
if ind.fitness.isDominated(hofer.fitness):
is_dominated = True
break
elif hofer.fitness.isDominated(ind.fitness):
to_remove.append(i)
elif ind.fitness == hofer.fitness and self.similar(ind, hofer):
has_twin = True
break
for i in reversed(to_remove): # Remove the dominated hofer
self.remove(i)
if not is_dominated and not has_twin:
self.insert(ind)
######################################
# GA Crossovers #
######################################
def cxTwoPoints(ind1, ind2):
"""Execute a two points crossover on the input individuals. The two
individuals are modified in place. This operation apply on an individual
composed of a list of attributes and act as follow ::
>>> ind1 = [A(1), ..., A(i), ..., A(j), ..., A(m)] #doctest: +SKIP
>>> ind2 = [B(1), ..., B(i), ..., B(j), ..., B(k)]
>>> # Crossover with mating points 1 < i < j <= min(m, k) + 1
>>> cxTwoPoints(ind1, ind2)
>>> print ind1, len(ind1)
[A(1), ..., B(i), ..., B(j-1), A(j), ..., A(m)], m
>>> print ind2, len(ind2)
[B(1), ..., A(i), ..., A(j-1), B(j), ..., B(k)], k
This function use the :func:`~random.randint` function from the python base
:mod:`random` module.
"""
size = min(len(ind1), len(ind2))
cxpoint1 = random.randint(1, size)
cxpoint2 = random.randint(1, size - 1)
if cxpoint2 >= cxpoint1:
cxpoint2 += 1
else: # Swap the two cx points
cxpoint1, cxpoint2 = cxpoint2, cxpoint1
ind1[cxpoint1:cxpoint2], ind2[cxpoint1:cxpoint2] \
= ind2[cxpoint1:cxpoint2], ind1[cxpoint1:cxpoint2]
return ind1, ind2
def cxOnePoint(ind1, ind2):
"""Execute a one point crossover on the input individuals.
The two individuals are modified in place. This operation apply on an
individual composed of a list of attributes
and act as follow ::
>>> ind1 = [A(1), ..., A(n), ..., A(m)] #doctest: +SKIP
>>> ind2 = [B(1), ..., B(n), ..., B(k)]
>>> # Crossover with mating point i, 1 < i <= min(m, k)
>>> cxOnePoint(ind1, ind2)
>>> print ind1, len(ind1)
[A(1), ..., B(i), ..., B(k)], k
>>> print ind2, len(ind2)
[B(1), ..., A(i), ..., A(m)], m
This function use the :func:`~random.randint` function from the
python base :mod:`random` module.
"""
size = min(len(ind1), len(ind2))
cxpoint = random.randint(1, size - 1)
ind1[cxpoint:], ind2[cxpoint:] = ind2[cxpoint:], ind1[cxpoint:]
return ind1, ind2
def cxUniform(ind1, ind2, indpb):
"""Execute a uniform crossover that modify in place the two individuals.
The genes are swapped according to the *indpb* probability.
This function use the :func:`~random.random` function from the python base
:mod:`random` module.
"""
size = min(len(ind1), len(ind2))
for i in xrange(size):
if random.random() < indpb:
ind1[i], ind2[i] = ind2[i], ind1[i]
return ind1, ind2
def cxPartialyMatched(ind1, ind2):
"""Execute a partially matched crossover (PMX) on the input individuals.
The two individuals are modified in place. This crossover expect iterable
individuals of indices, the result for any other type of individuals is
unpredictable.
Moreover, this crossover consists of generating two children by matching
pairs of values in a certain range of the two parents and swapping the values
of those indexes. For more details see Goldberg and Lingel, "Alleles,
loci, and the traveling salesman problem", 1985.
For example, the following parents will produce the two following children
when mated with crossover points ``a = 2`` and ``b = 4``. ::
>>> ind1 = [0, 1, 2, 3, 4]
>>> ind2 = [1, 2, 3, 4, 0]
>>> cxPartialyMatched(ind1, ind2)
>>> print ind1
[0, 2, 3, 1, 4]
>>> print ind2
[2, 3, 1, 4, 0]
This function use the :func:`~random.randint` function from the python base
:mod:`random` module.
"""
size = min(len(ind1), len(ind2))
p1, p2 = [0]*size, [0]*size
# Initialize the position of each indices in the individuals
for i in xrange(size):
p1[ind1[i]] = i
p2[ind2[i]] = i
# Choose crossover points
cxpoint1 = random.randint(0, size)
cxpoint2 = random.randint(0, size - 1)
if cxpoint2 >= cxpoint1:
cxpoint2 += 1
else: # Swap the two cx points
cxpoint1, cxpoint2 = cxpoint2, cxpoint1
# Apply crossover between cx points
for i in xrange(cxpoint1, cxpoint2):
# Keep track of the selected values
temp1 = ind1[i]
temp2 = ind2[i]
# Swap the matched value
ind1[i], ind1[p1[temp2]] = temp2, temp1
ind2[i], ind2[p2[temp1]] = temp1, temp2
# Position bookkeeping
p1[temp1], p1[temp2] = p1[temp2], p1[temp1]
p2[temp1], p2[temp2] = p2[temp2], p2[temp1]
return ind1, ind2
def cxUniformPartialyMatched(ind1, ind2, indpb):
"""Execute a uniform partially matched crossover (UPMX) on the input
individuals. The two individuals are modified in place. This crossover
expect iterable individuals of indices, the result for any other type of
individuals is unpredictable.
Moreover, this crossover consists of generating two children by matching
pairs of values chosen at random with a probability of *indpb* in the two
parents and swapping the values of those indexes. For more details see
Cicirello and Smith, "Modeling GA performance for control parameter
optimization", 2000.
For example, the following parents will produce the two following children
when mated with the chosen points ``[0, 1, 0, 0, 1]``. ::
>>> ind1 = [0, 1, 2, 3, 4] #doctest: +SKIP
>>> ind2 = [1, 2, 3, 4, 0]
>>> cxUniformPartialyMatched(ind1, ind2)
>>> print ind1
[4, 2, 1, 3, 0]
>>> print ind2
[2, 1, 3, 0, 4]
This function use the :func:`~random.random` and :func:`~random.randint`
functions from the python base :mod:`random` module.
"""
size = min(len(ind1), len(ind2))
p1, p2 = [0]*size, [0]*size
# Initialize the position of each indices in the individuals
for i in xrange(size):
p1[ind1[i]] = i
p2[ind2[i]] = i
for i in xrange(size):
if random.random < indpb:
# Keep track of the selected values
temp1 = ind1[i]
temp2 = ind2[i]
# Swap the matched value
ind1[i], ind1[p1[temp2]] = temp2, temp1
ind2[i], ind2[p2[temp1]] = temp1, temp2
# Position bookkeeping
p1[temp1], p1[temp2] = p1[temp2], p1[temp1]
p2[temp1], p2[temp2] = p2[temp2], p2[temp1]
return ind1, ind2
def cxBlend(ind1, ind2, alpha):
"""Executes a blend crossover that modify in-place the input individuals.
The blend crossover expect individuals formed of a list of floating point
numbers.
This function use the :func:`~random.random` function from the python base
:mod:`random` module.
"""
size = min(len(ind1), len(ind2))
for i in xrange(size):
gamma = (1. + 2. * alpha) * random.random() - alpha
x1 = ind1[i]
x2 = ind2[i]
ind1[i] = (1. - gamma) * x1 + gamma * x2
ind2[i] = gamma * x1 + (1. - gamma) * x2
return ind1, ind2
def cxSimulatedBinary(ind1, ind2, nu):
"""Executes a simulated binary crossover that modify in-place the input
individuals. The simulated binary crossover expect individuals formed of
a list of floating point numbers.
This function use the :func:`~random.random` function from the python base
:mod:`random` module.
"""
size = min(len(ind1), len(ind2))
for i in xrange(size):
rand = random.random()
if rand <= 0.5:
beta = 2. * rand
else:
beta = 1. / (2. * (1. - rand))
beta **= 1. / (nu + 1.)
x1 = ind1[i]
x2 = ind2[i]
ind1[i] = 0.5 * (((1 + beta) * x1) + ((1 - beta) * x2))
ind2[i] = 0.5 * (((1 - beta) * x1) + ((1 + beta) * x2))
return ind1, ind2
######################################
# Messy Crossovers #
######################################
def cxMessyOnePoint(ind1, ind2):
"""Execute a one point crossover that will in most cases change the
individuals size. This operation apply on an individual composed
of a list of attributes and act as follow ::
>>> ind1 = [A(1), ..., A(i), ..., A(m)] #doctest: +SKIP
>>> ind2 = [B(1), ..., B(j), ..., B(n)]
>>> # Crossover with mating points i, j, 1 <= i <= m, 1 <= j <= n
>>> cxMessyOnePoint(ind1, ind2)
>>> print ind1, len(ind1)
[A(1), ..., A(i - 1), B(j), ..., B(n)], n + j - i
>>> print ind2, len(ind2)
[B(1), ..., B(j - 1), A(i), ..., A(m)], m + i - j
This function use the :func:`~random.randint` function from the python base
:mod:`random` module.
"""
cxpoint1 = random.randint(0, len(ind1))
cxpoint2 = random.randint(0, len(ind2))
ind1[cxpoint1:], ind2[cxpoint2:] = ind2[cxpoint2:], ind1[cxpoint1:]
return ind1, ind2
######################################
# ES Crossovers #
######################################
def cxESBlend(ind1, ind2, alpha):
"""Execute a blend crossover on both, the individual and the strategy. The
individuals must have a :attr:`strategy` attribute. Adjustement of the
minimal strategy shall be done after the call to this function using a
decorator, for example ::
def checkStrategy(minstrategy):
def decMinStrategy(func):
def wrapMinStrategy(*args, **kargs):
children = func(*args, **kargs)
for child in children:
if child.strategy < minstrategy:
child.strategy = minstrategy
return children
return wrapMinStrategy
return decMinStrategy
toolbox.register("mate", tools.cxEsBlend, alpha=ALPHA)
toolbox.decorate("mate", checkStrategy(minstrategy=0.01))
"""
size = min(len(ind1), len(ind2))
for indx in xrange(size):
# Blend the values
gamma = (1. + 2. * alpha) * random.random() - alpha
x1 = ind1[indx]
x2 = ind2[indx]
ind1[indx] = (1. - gamma) * x1 + gamma * x2
ind2[indx] = gamma * x1 + (1. - gamma) * x2
# Blend the strategies
gamma = (1. + 2. * alpha) * random.random() - alpha
s1 = ind1.strategy[indx]
s2 = ind2.strategy[indx]
ind1.strategy[indx] = (1. - gamma) * s1 + gamma * s2
ind2.strategy[indx] = gamma * s1 + (1. - gamma) * s2
return ind1, ind2
def cxESTwoPoints(ind1, ind2):
"""Execute a classical two points crossover on both the individual and
its strategy. The crossover points for the individual and the strategy
are the same.
"""
size = min(len(ind1), len(ind2))
pt1 = random.randint(1, size)
pt2 = random.randint(1, size - 1)
if pt2 >= pt1:
pt2 += 1
else: # Swap the two cx points
pt1, pt2 = pt2, pt1
ind1[pt1:pt2], ind2[pt1:pt2] = ind2[pt1:pt2], ind1[pt1:pt2]
ind1.strategy[pt1:pt2], ind2.strategy[pt1:pt2] = \
ind2.strategy[pt1:pt2], ind1.strategy[pt1:pt2]
return ind1, ind2
######################################
# GA Mutations #
######################################
def mutGaussian(individual, mu, sigma, indpb):
"""This function applies a gaussian mutation of mean *mu* and standard
deviation *sigma* on the input individual and
returns the mutant. The *individual* is left intact and the mutant is an
independant copy. This mutation expects an iterable individual composed of
real valued attributes. The *mutIndxPb* argument is the probability of each
attribute to be mutated.
.. note::
The mutation is not responsible for constraints checking, because
there is too many possibilities for
resetting the values. Which way is closer to the representation used
is up to you.
One easy way to add constraint checking to an operator is to
use the function decoration in the toolbox. See the multi-objective
example (moga_kursawefct.py) for an explicit example.
This function uses the :func:`~random.random` and :func:`~random.gauss`
functions from the python base :mod:`random` module.
"""
for i in xrange(len(individual)):
if random.random() < indpb:
individual[i] += random.gauss(mu, sigma)
return individual,
def mutShuffleIndexes(individual, indpb):
"""Shuffle the attributes of the input individual and return the mutant.
The *individual* is left intact and the mutant is an independent copy. The
*individual* is expected to be iterable. The *shuffleIndxPb* argument is the
probability of each attribute to be moved.
This function uses the :func:`~random.random` and :func:`~random.randint`
functions from the python base :mod:`random` module.
"""
size = len(individual)
for i in xrange(size):
if random.random() < indpb:
swap_indx = random.randint(0, size - 2)
if swap_indx >= i:
swap_indx += 1
individual[i], individual[swap_indx] = \
individual[swap_indx], individual[i]
return individual,
def mutFlipBit(individual, indpb):
"""Flip the value of the attributes of the input individual and return the
mutant. The *individual* is left intact and the mutant is an independent
copy. The *individual* is expected to be iterable and the values of the
attributes shall stay valid after the ``not`` operator is called on them.
The *flipIndxPb* argument is the probability of each attribute to be
flipped.
This function uses the :func:`~random.random` function from the python base
:mod:`random` module.
"""
for indx in xrange(len(individual)):
if random.random() < indpb:
individual[indx] = not individual[indx]
return individual,
######################################
# ES Mutations #
######################################
def mutESLogNormal(individual, c, indpb):
"""Mutate an evolution strategy according to its :attr:`strategy`
attribute as described in *Beyer and Schwefel, 2002, Evolution strategies
- A Comprehensive Introduction*. The individual is first mutated by a
normal distribution of mean 0 and standard deviation of
:math:`\\boldsymbol{\sigma}_{t-1}` then the strategy is mutated according
to an extended log normal rule,
:math:`\\boldsymbol{\sigma}_t = \\exp(\\tau_0 \mathcal{N}_0(0, 1)) \\left[
\\sigma_{t-1, 1}\\exp(\\tau \mathcal{N}_1(0, 1)), \ldots, \\sigma_{t-1, n}
\\exp(\\tau \mathcal{N}_n(0, 1))\\right]`, with :math:`\\tau_0 =
\\frac{c}{\\sqrt{2n}}` and :math:`\\tau = \\frac{c}{\\sqrt{2\\sqrt{n}}}`.
A recommended choice is :math:`c=1` when using a :math:`(10, 100)`
evolution strategy (Beyer and Schwefel, 2002).
The strategy shall be the same size as the individual. Each index
(strategy and attribute) is mutated with probability *indpb*. In order to
limit the strategy, use a decorator as shown in the :func:`cxESBlend`
function.
"""
size = len(individual)
t = c / math.sqrt(2. * math.sqrt(size))
t0 = c / math.sqrt(2. * size)
n = random.gauss(0, 1)
t0_n = t0 * n
for indx in xrange(size):
if random.random() < indpb:
individual[indx] += individual.strategy[indx] * random.gauss(0, 1)
individual.strategy[indx] *= math.exp(t0_n + t * random.gauss(0, 1))
return individual,
######################################
# Selections #
######################################
def selRandom(individuals, k):
"""Select *k* individuals at random from the input *individuals* with
replacement. The list returned contains references to the input
*individuals*.
This function uses the :func:`~random.choice` function from the
python base :mod:`random` module.
"""
return [random.choice(individuals) for i in xrange(k)]
def selBest(individuals, k):
"""Select the *k* best individuals among the input *individuals*. The
list returned contains references to the input *individuals*.
"""
return sorted(individuals, key=attrgetter("fitness"), reverse=True)[:k]
def selWorst(individuals, k):
"""Select the *k* worst individuals among the input *individuals*. The
list returned contains references to the input *individuals*.
"""
return sorted(individuals, key=attrgetter("fitness"))[:k]
def selTournament(individuals, k, tournsize):
"""Select *k* individuals from the input *individuals* using *k*
tournaments of *tournSize* individuals. The list returned contains
references to the input *individuals*.
This function uses the :func:`~random.choice` function from the python base
:mod:`random` module.
"""
chosen = []
for i in xrange(k):
chosen.append(random.choice(individuals))
for j in xrange(tournsize - 1):
aspirant = random.choice(individuals)
if aspirant.fitness > chosen[i].fitness:
chosen[i] = aspirant
return chosen
def selRoulette(individuals, k):
"""Select *k* individuals from the input *individuals* using *k*
spins of a roulette. The selection is made by looking only at the first
objective of each individual. The list returned contains references to
the input *individuals*.
This function uses the :func:`~random.random` function from the python base
:mod:`random` module.
.. warning::
The roulette selection by definition cannot be used for minimization
or when the fitness can be smaller or equal to 0.
"""
s_inds = sorted(individuals, key=attrgetter("fitness"), reverse=True)
sum_fits = sum(ind.fitness.values[0] for ind in individuals)
chosen = []
for i in xrange(k):
u = random.random() * sum_fits
sum_ = 0
for ind in s_inds:
sum_ += ind.fitness.values[0]
if sum_ > u:
chosen.append(ind)
break
return chosen
######################################
# Non-Dominated Sorting (NSGA-II) #
######################################
def selNSGA2(individuals, k):
"""Apply NSGA-II selection operator on the *individuals*. Usually,
the size of *individuals* will be larger than *k* because any individual
present in *individuals* will appear in the returned list at most once.
Having the size of *individuals* equals to *n* will have no effect other
than sorting the population according to a non-domination scheme. The list
returned contains references to the input *individuals*.
For more details on the NSGA-II operator see Deb, Pratab, Agarwal,
and Meyarivan, "A fast elitist non-dominated sorting genetic algorithm for
multi-objective optimization: NSGA-II", 2002.
"""
pareto_fronts = sortFastND(individuals, k)
chosen = list(chain(*pareto_fronts[:-1]))
k = k - len(chosen)
if k > 0:
chosen.extend(sortCrowdingDist(pareto_fronts[-1], k))
return chosen
def sortFastND(individuals, k, first_front_only=False):
"""Sort the first *k* *individuals* according the the fast non-dominated
sorting algorithm.
"""
N = len(individuals)
pareto_fronts = []
if k == 0:
return pareto_fronts
pareto_fronts.append([])
pareto_sorted = 0
dominating_inds = [0] * N
dominated_inds = [list() for i in xrange(N)]
# Rank first Pareto front
for i in xrange(N):
for j in xrange(i+1, N):
if individuals[j].fitness.isDominated(individuals[i].fitness):
dominating_inds[j] += 1
dominated_inds[i].append(j)
elif individuals[i].fitness.isDominated(individuals[j].fitness):
dominating_inds[i] += 1
dominated_inds[j].append(i)
if dominating_inds[i] == 0:
pareto_fronts[-1].append(i)
pareto_sorted += 1
if not first_front_only:
# Rank the next front until all individuals are sorted or the given
# number of individual are sorted
N = min(N, k)
while pareto_sorted < N:
pareto_fronts.append([])
for indice_p in pareto_fronts[-2]:
for indice_d in dominated_inds[indice_p]:
dominating_inds[indice_d] -= 1
if dominating_inds[indice_d] == 0:
pareto_fronts[-1].append(indice_d)
pareto_sorted += 1
return [[individuals[index] for index in front] for front in pareto_fronts]
def sortCrowdingDist(individuals, k):
"""Sort the individuals according to the crowding distance."""
if len(individuals) == 0:
return []
distances = [0.0] * len(individuals)
crowding = [(ind, i) for i, ind in enumerate(individuals)]
number_objectives = len(individuals[0].fitness.values)
inf = float("inf") # It is four times faster to compare with a local
# variable than create the float("inf") each time
for i in xrange(number_objectives):
crowding.sort(key=lambda element: element[0].fitness.values[i])
distances[crowding[0][1]] = float("inf")
distances[crowding[-1][1]] = float("inf")
for j in xrange(1, len(crowding) - 1):
if distances[crowding[j][1]] < inf:
distances[crowding[j][1]] += \
crowding[j + 1][0].fitness.values[i] - \
crowding[j - 1][0].fitness.values[i]
sorted_dist = sorted([(dist, i) for i, dist in enumerate(distances)],
key=lambda value: value[0], reverse=True)
return (individuals[index] for dist, index in sorted_dist[:k])
######################################
# Strength Pareto (SPEA-II) #
######################################
def selSPEA2(individuals, k):
"""Apply SPEA-II selection operator on the *individuals*. Usually,
the size of *individuals* will be larger than *n* because any individual
present in *individuals* will appear in the returned list at most once.
Having the size of *individuals* equals to *n* will have no effect other
than sorting the population according to a strength Pareto scheme. The list
returned contains references to the input *individuals*.
For more details on the SPEA-II operator see Zitzler, Laumanns and Thiele,
"SPEA 2: Improving the strength Pareto evolutionary algorithm", 2001.
"""
N = len(individuals)
L = len(individuals[0].fitness.values)
K = math.sqrt(N)
strength_fits = [0] * N
fits = [0] * N
dominating_inds = [list() for i in xrange(N)]
for i in xrange(N):
for j in xrange(i + 1, N):
if individuals[i].fitness.isDominated(individuals[j].fitness):
strength_fits[j] += 1
dominating_inds[i].append(j)
elif individuals[j].fitness.isDominated(individuals[i].fitness):
strength_fits[i] += 1
dominating_inds[j].append(i)
for i in xrange(N):
for j in dominating_inds[i]:
fits[i] += strength_fits[j]
# Choose all non-dominated individuals
chosen_indices = [i for i in xrange(N) if fits[i] < 1]
if len(chosen_indices) < k: # The archive is too small
for i in xrange(N):
distances = [0.0] * N
for j in xrange(i + 1, N):
dist = 0.0
for l in xrange(L):
val = individuals[i].fitness.values[l] - \
individuals[j].fitness.values[l]
dist += val * val
distances[j] = dist
kth_dist = _randomizedSelect(distances, 0, N - 1, K)
density = 1.0 / (kth_dist + 2.0)
fits[i] += density
next_indices = [(fits[i], i) for i in xrange(N) \
if not i in chosen_indices]
next_indices.sort()
#print next_indices
chosen_indices += [i for _, i in next_indices[:k - len(chosen_indices)]]
elif len(chosen_indices) > k: # The archive is too large
N = len(chosen_indices)
distances = [[0.0] * N for i in xrange(N)]
sorted_indices = [[0] * N for i in xrange(N)]
for i in xrange(N):
for j in xrange(i + 1, N):
dist = 0.0
for l in xrange(L):
val = individuals[chosen_indices[i]].fitness.values[l] - \
individuals[chosen_indices[j]].fitness.values[l]
dist += val * val
distances[i][j] = dist
distances[j][i] = dist
distances[i][i] = -1
# Insert sort is faster than quick sort for short arrays
for i in xrange(N):
for j in xrange(1, N):
l = j
while l > 0 and distances[i][j] < distances[i][sorted_indices[i][l - 1]]:
sorted_indices[i][l] = sorted_indices[i][l - 1]
l -= 1
sorted_indices[i][l] = j
size = N
to_remove = []
while size > k:
# Search for minimal distance
min_pos = 0
for i in xrange(1, N):
for j in xrange(1, size):
dist_i_sorted_j = distances[i][sorted_indices[i][j]]
dist_min_sorted_j = distances[min_pos][sorted_indices[min_pos][j]]
if dist_i_sorted_j < dist_min_sorted_j:
min_pos = i
break
elif dist_i_sorted_j > dist_min_sorted_j:
break
# Remove minimal distance from sorted_indices
for i in xrange(N):
distances[i][min_pos] = float("inf")
distances[min_pos][i] = float("inf")
for j in xrange(1, size - 1):
if sorted_indices[i][j] == min_pos:
sorted_indices[i][j] = sorted_indices[i][j + 1]
sorted_indices[i][j + 1] = min_pos
# Remove corresponding individual from chosen_indices
to_remove.append(min_pos)
size -= 1
for index in reversed(sorted(to_remove)):
del chosen_indices[index]
return [individuals[i] for i in chosen_indices]
def _randomizedSelect(array, begin, end, i):
"""Allows to select the ith smallest element from array without sorting it.
Runtime is expected to be O(n).
"""
if begin == end:
return array[begin]
q = _randomizedPartition(array, begin, end)
k = q - begin + 1
if i < k:
return _randomizedSelect(array, begin, q, i)
else:
return _randomizedSelect(array, q + 1, end, i - k)
def _randomizedPartition(array, begin, end):
i = random.randint(begin, end)
array[begin], array[i] = array[i], array[begin]
return _partition(array, begin, end)
def _partition(array, begin, end):
x = array[begin]
i = begin - 1
j = end + 1
while True:
j -= 1
while array[j] > x:
j -= 1
i += 1
while array[i] < x:
i += 1
if i < j:
array[i], array[j] = array[j], array[i]
else:
return j
######################################
# Replacement Strategies (ES) #
######################################
######################################
# Migrations #
######################################
def migRing(populations, k, selection, replacement=None, migarray=None):
"""Perform a ring migration between the *populations*. The migration first
select *k* emigrants from each population using the specified *selection*
operator and then replace *k* individuals from the associated population
in the *migarray* by the emigrants. If no *replacement* operator is
specified, the immigrants will replace the emigrants of the population,
otherwise, the immigrants will replace the individuals selected by the
*replacement* operator. The migration array, if provided, shall contain
each population's index once and only once. If no migration array is
provided, it defaults to a serial ring migration (1 -- 2 -- ... -- n --
1). Selection and replacement function are called using the signature
``selection(populations[i], k)`` and ``replacement(populations[i], k)``.
It is important to note that the replacement strategy must select *k*
**different** individuals. For example, using a traditional tournament for
replacement strategy will thus give undesirable effects, two individuals
will most likely try to enter the same slot.
"""
if migarray is None:
migarray = range(1, len(populations)) + [0]
immigrants = [[] for i in xrange(len(migarray))]
emigrants = [[] for i in xrange(len(migarray))]
for from_deme in xrange(len(migarray)):
emigrants[from_deme].extend(selection(populations[from_deme], k))
if replacement is None:
# If no replacement strategy is selected, replace those who migrate
immigrants[from_deme] = emigrants[from_deme]
else:
# Else select those who will be replaced
immigrants[from_deme].extend(replacement(populations[from_deme], k))
mig_buf = emigrants[0]
for from_deme, to_deme in enumerate(migarray[1:]):
from_deme += 1 # Enumerate starts at 0
for i, immigrant in enumerate(immigrants[to_deme]):
indx = populations[to_deme].index(immigrant)
populations[to_deme][indx] = emigrants[from_deme][i]
to_deme = migarray[0]
for i, immigrant in enumerate(immigrants[to_deme]):
indx = populations[to_deme].index(immigrant)
populations[to_deme][indx] = mig_buf[i]
######################################
# Decoration tool #
######################################
# This function is a simpler version of the decorator module (version 3.2.0)
# from Michele Simionato available at http://pypi.python.org/pypi/decorator.
# Copyright (c) 2005, Michele Simionato
# All rights reserved.
# Modified by Francois-Michel De Rainville, 2010
def decorate(decorator):
"""Decorate a function preserving its signature. There is two way of
using this function, first as a decorator passing the decorator to
use as argument, for example ::
@decorate(a_decorator)
def myFunc(arg1, arg2, arg3="default"):
do_some_work()
return "some_result"
Or as a decorator ::
@decorate
def myDecorator(func):
def wrapFunc(*args, **kargs):
decoration_work()
return func(*args, **kargs)
return wrapFunc
@myDecorator
def myFunc(arg1, arg2, arg3="default"):
do_some_work()
return "some_result"
Using the :mod:`inspect` module, we can retrieve the signature of the
decorated function, what is not possible when not using this method. ::
print inspect.getargspec(myFunc)
It shall return something like ::
(["arg1", "arg2", "arg3"], None, None, ("default",))
This function is a simpler version of the decorator module (version 3.2.0)
from Michele Simionato available at http://pypi.python.org/pypi/decorator.
"""
def wrapDecorate(func):
# From __init__
assert func.__name__
if inspect.isfunction(func):
argspec = inspect.getargspec(func)
defaults = argspec[-1]
signature = inspect.formatargspec(formatvalue=lambda val: "",
*argspec)[1:-1]
elif inspect.isclass(func):
argspec = inspect.getargspec(func.__init__)
defaults = argspec[-1]
signature = inspect.formatargspec(formatvalue=lambda val: "",
*argspec)[1:-1]
if not signature:
raise TypeError("You are decorating a non function: %s" % func)
# From create
src = ("def %(name)s(%(signature)s):\n"
" return _call_(%(signature)s)\n") % dict(name=func.__name__,
signature=signature)
# From make
evaldict = dict(_call_=decorator(func))
reserved_names = set([func.__name__] + \
[arg.strip(' *') for arg in signature.split(',')])
for name in evaldict.iterkeys():
if name in reserved_names:
raise NameError("%s is overridden in\n%s" % (name, src))
try:
# This line does all the dirty work of reassigning the signature
code = compile(src, "", "single")
exec code in evaldict
except:
raise RuntimeError("Error in generated code:\n%s" % src)
new_func = evaldict[func.__name__]
# From update
new_func.__source__ = src
new_func.__name__ = func.__name__
new_func.__doc__ = func.__doc__
new_func.__dict__ = func.__dict__.copy()
new_func.func_defaults = defaults
new_func.__module__ = func.__module__
return new_func
return wrapDecorate
if __name__ == "__main__":
import doctest
import random
random.seed(64)
doctest.run_docstring_examples(initRepeat, globals())
random.seed(64)
doctest.run_docstring_examples(initIterate, globals())
doctest.run_docstring_examples(initCycle, globals())
doctest.run_docstring_examples(Statistics.register, globals())
doctest.run_docstring_examples(Statistics.update, globals())
deap-0.7.1/doc/ 0000755 0000765 0000024 00000000000 11650301263 013424 5 ustar felix staff 0000000 0000000 deap-0.7.1/doc/_images/ 0000755 0000765 0000024 00000000000 11650301263 015030 5 ustar felix staff 0000000 0000000 deap-0.7.1/doc/_images/deap.png 0000644 0000765 0000024 00000104235 11641072614 016461 0 ustar felix staff 0000000 0000000 ‰PNG
IHDR t º ÒCß pHYs .# .#x¥?v tEXtSoftware Adobe ImageReadyqÉe<