SimGrid-3.18/0000755000175000017500000000000013217757343013323 5ustar mquinsonmquinsonSimGrid-3.18/NEWS0000644000175000017500000004464013217757340014027 0ustar mquinsonmquinson _ _____ _ ___ __ _____ _ __ ___(_) ___ _ __ |___ / / |( _ ) \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | |/ _ \ \ V / __/ | \__ \ | (_) | | | | ___) || | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|\___/ December 24 2017 The "Ho Ho Ho! SimGrid 4 beta is coming to town" release. SimGrid 4 *may* be there by the next solstice. * Convert almost all interesting MSG examples to S4U. * New model: energy consumption due to the network. * Major cleanups in the disk and storage subsystems. * (+ further deprecate XBT, bug fixes and doc improvement) _ _____ _ _____ __ _____ _ __ ___(_) ___ _ __ |___ / / |___ | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | / / \ V / __/ | \__ \ | (_) | | | | ___) || | / / \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|/_/ October 8 2017 The Drained Leaks release: (almost) no known leaks despite the tests. * Many many internal cleanups (almost 700 commits since 3.16). * The coverage of our tests is above 80%. * All memleaks but one plugged; A dozen of bugs fixed. * XBT: Further replace XBT with std::* constructs. _ _____ _ __ __ _____ _ __ ___(_) ___ _ __ |___ / / |/ /_ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | '_ \ \ V / __/ | \__ \ | (_) | | | | ___) || | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|\___/ June 22 2017 The Blooming Spring Release: developments are budding. * S4U: Progress; Activity refcounting is now automatic. * XML: can now be named as they should. * SMPI: Further performance improvements; RMA support. * Cloud: Multi-core VMs (do not overcommit them yet) * (+ bug fixes, cleanups and documentation improvements) _ _____ _ ____ __ _____ _ __ ___(_) ___ _ __ |___ / / | ___| \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | |___ \ \ V / __/ | \__ \ | (_) | | | | ___) || |___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|____/ Mar 22 2017 The Spring Release: continuous integration servers become green. * S4U: progress, integrating more parts of SimDag; New examples. * SMPI: Support MPI 2.2; Convert internals to C++ (TBC). * Java: Massive memleaks and performance issues fixed. * (+ bug fixes, cleanups and documentation improvements) _ _____ _ _ _ _ ____ ___ __ _____ _ __ ___(_) ___ _ __ |___ / / | || | / | ___|/ _ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | || |_ | |___ \ (_) | \ V / __/ | \__ \ | (_) | | | | ___) || |__ _|| |___) \__, | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_| |_|(_)_|____/ /_/ Dec 28 2016 The Christmas Pi Release (better approximation). * Fix various glitches in the previous release. _ _____ _ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / / | || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | || |_ \ V / __/ | \__ \ | (_) | | | | ___) || |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_)_| |_| Dec 24 2016 The Christmas Pi Release. * Documentation reorganized and improved * S4U interface further rising, toward SimGrid 4 - Routing code rewritten for readability - Virtual Machines almost turned into a plugin - MSG, SimDag, MPI interfaces mostly unchanged * The model-checker now works on FreeBSD too. _ _____ _ _____ __ _____ _ __ ___(_) ___ _ __ |___ / / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | |_ \ \ V / __/ | \__ \ | (_) | | | | ___) || |___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|____/ Apr 27 2016 The Half Release, a.k.a. the Zealous Easter Trim. * Half of the lines of code are gone. - v3.12: 286k lines; v3.13: 142k lines (+ comments) - Experimental untested unused "features" removed - We rewrote several parts in C++ instead of C * Introducing v4 of the XML platform format - Many long-due cleanups (details in the Changelog) * MSG examples fully reorganized (in C and Java) * The S4U interface is rising, toward SimGrid 4 - All host manipulations now done in S4U - SimDag was mostly rewritten on top of S4U - MSG & SimDag interfaces mostly unchanged _ _____ _ ____ __ _____ _ __ ___(_) ___ _ __ |___ / / |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | __) | \ V / __/ | \__ \ | (_) | | | | ___) || |/ __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|_____| Oct 12 2015 The Facelift Release. Major changes: * Many interface glitches addressed, everywhere. - Require g++-4.7, Java 7 and boost 1.48 - Builds on Linux, OS X, Windows and FreeBSD - See ChangeLog for renamed functions and options. * Energy plugin: major cleanups/rewrites. http://simgrid.org/tutorials/simgrid-energy-101.pdf * Model-Checker progresses toward production use: - More robust (now runs in a separate process). - More efficient (with snapshots' memory compaction). - More usable (execution paths replayed out of MC) http://simgrid.org/tutorials/simgrid-mc-101.pdf * SMPI improvements: - Better standard coverage - SMP-aware collectives - InfiniBand model - Early support for RMA - Replay: Dynamic selector - MVAPICH2 selector http://simgrid.org/tutorials/simgrid-smpi-101.pdf Upcoming v3.13 will introduce new interfaces, toward SimGrid 4. _ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / / / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | | \ V / __/ | \__ \ | (_) | | | | ___) || | | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|_| May 31 2014 The Class Release. Major changes: * Surf is now in C++ (and documented!) * Virtual Machine model for Cloud Simulation * Surf callbacks: plug your code directly in Surf! * Simcalls are script-generated to clean the mess _ _____ _ ___ __ _____ _ __ ___(_) ___ _ __ |___ / / |/ _ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | | | | \ V / __/ | \__ \ | (_) | | | | ___) || | |_| | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|\___/ Nov 17 2013 The Clean Diaper Release, a.k.a. SimGrid is leak-free. Major changes: * Preliminary DVFS support to track the energy consumption * Java is back in the main package (and is very stable). * The storage sub-modules is now believed to be usable. * SMPI is now very stable (we pass most MPICH tests). * Lots of memory leaks were corrected in this release. * Verification and model checking further improved: liveness+SMPI works; reduction through state equality detection Plus the usual load of bug fixes and small improvements _ _____ ___ __ _____ _ __ ___(_) ___ _ __ |___ // _ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ (_) | \ V / __/ | \__ \ | (_) | | | | ___) \__, | \_/ \___|_| |___/_|\___/|_| |_| |____(_)/_/ Feb 5 2013 The "Grasgory" release. Major changes: * Gras was completely removed from this version. * Documentation reorganization to ease browsing it. * New default value for the TCP_gamma parameter: 4MiB _ _____ ___ _ __ _____ _ __ ___(_) ___ _ __ |___ / ( _ ) / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / _ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | (_) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_| Oct 27 2012 The "we are told that some people want to also *install* the simgrid framework" release. * Add missing manpage to the archive. _ _____ ___ __ _____ _ __ ___(_) ___ _ __ |___ / ( _ ) \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / _ \ \ V / __/ | \__ \ | (_) | | | | ___) | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___/ Oct 25 2012 The Psssshiiiit release: SimGrid jumps into the Cloud. Major changes: * Experimental interface to manipulate VMs, EC2-style. * Fixes around process restart and stochastic workloads * platf: New C interface to create fixed or random platforms * SimDag: Many fixes and improvements of typed tasks * SMPI now covers more of the MPI interface. More datatypes, more functions, more robust. * Model-checking: mmalloc is more robust to user errors. _ _____ _____ _ __ _____ _ __ ___(_) ___ _ __ |___ /|___ / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / /| | \ V / __/ | \__ \ | (_) | | | | ___) | / /_| | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_/(_)_| June 7 2012 The "cleaning the cleanup" release. Major changes: * Portability fixups for Mac OSX and Windows * Some other bug fixing and various polishing. _ ____ _____ __ _____ _ __ ___(_) ___ _ __ |___ /|___ | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / / \ V / __/ | \__ \ | (_) | | | | ___) | / / \_/ \___|_| |___/_|\___/|_| |_| |____(_)_/ May 15 2012 The "spring cleanups (before the next Big Project kicks in)" release. Major changes: * Major cleanups all around (doc, user options, MSG, Lua, internals) Small backward compatibility glitches may have been introduced * Parallel execution of user code should be stable and efficient now * SMPI is now considered stable * Add temporals to Model-Checking (highly experimental right now) _ _____ __ ____ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ __) | \ V / __/ | \__ \ | (_) | | | | ___) | (_) | / __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_____| Oct 5 2011 The "Not coding new stuff allows to polish old things" release. * Portability to Mac and Windows improved. * Possible misconfigurations (contexts, libPCRE) made impossible by removing the option or providing sane default value. * Experimental support to NS3 simulator as a back-end. _ _____ __ _ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | (_) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_| Jun 27 2011 The "Oops, we broke Macs too" release. Bug fix to an issue preventing SimGrid from working on Mac OSX. _ _____ __ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ \ V / __/ | \__ \ | (_) | | | | ___) | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___/ Jun 21 2011 The "OMG! They Killed Kenny!" version. Major changes: * Java and Ruby bindings were removed from the main archive - They are now distributed separately * GRAS is not considered as stable anymore, but experimental. Sorry. * First support for parallel simulation: - Can run the user code of any simulation in parallel. - Basic support for multi-cores hosts in platform files * Improved P2P support: - Peer tag in platform files for DSL connected nodes - Vivaldi routing scheme for lightweigted yet accurate models * Improved SMPI: Faster Fortran, automatic privatization of C globals * Tracing: trace contains the full platform hierarchy exactly as declared using the ASes of the platform file Plus numerous other goodies (check the ChangeLog for details) _ _____ ____ __ _____ _ __ ___(_) ___ _ __ |___ / | ___| \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |___ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____/ 01 Dec 2010 The "Winter in Frejus" release. Also known as "ANR/ADT funding helps" Major changes are: * New feature: Model check any simgrid simulation * SMPI is now very usable. * Visualization: - now covers the whole framework - major usability improvements * SURF: - scalable platform management through hierarchical description - new efficient way to bypass the XML parser * MSG: at last asynchronous functions are available * SIMDAG: many usability improvements (dotloader, ...) * GRAS: finally catch up with latest internal evolutions * Build chain: - Windows port: should be usable now but still considered experimental - Autotools have now been completely removed _ _____ _ _ _ __ _____ _ __ ___(_) ___ _ __ |___ /| || | / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| || |_ | | \ V / __/ | \__ \ | (_) | | | | ___) |__ _|| | \_/ \___|_| |___/_|\___/|_| |_| |____(_) |_|(_)_| 04 May 2010 The "Polishing easter eggs is probably a good idea" release. This is a bug fixes release only. _ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ /| || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| || |_ \ V / __/ | \__ \ | (_) | | | | ___) |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_) |_| 28 Apr 2010 The "Easter in Cargese" release. Also known as (major changes): * the "se habla Java, Ruby 話せます, fala-se Lua (and deaf-friendly)" ~> bindings were greatly improved ~> new tracing infrastructure for better visualization introduced * the "Welcome to configury modernity" release. ~> we switched from autotools to cmake, and improved our cdash _ _____ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / |___ /| || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \| || |_ \ V / __/ | \__ \ | (_) | | | | ___) | ___) |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_) |_| 24 Dec 2009 The "Desktop Grid needs love too" release (also called "Xmas release"). Most important changes: * Big speedup through lazy evaluation of the linear models * Supernovae mode: Compile everything in one unit to improve inlining * Simix network module for internal cleanups * Load DAX of applications into SimDag * Lot of small cleanups and other bug fixes _ _____ _____ _____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ |_ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)____/ 20 Aug 2009 The "Need for Speed" release. Big speedup through some function inlining. _ _____ _____ ____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ __) | \ V / __/ | \__ \ | (_) | | | | ___) | ___) | / __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)_____| 19 Aug 2009 The "Simplicity does not preceed complexity, but follows it" release. Most important changes: * surf and simix modules reworked for simplification. It lays the ground for future extensions and improvements of SimGrid * SMPI was further improved, although not ready for production use yet. On the way, we gained a bit more than 5% on the classical master/slave example. More to come on this in future releases. _ _____ _____ _ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | ___) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)_| 27 Jun 2009 This dot release is mainly a maintenance one. Most important changes: * We fixed a large amount of bugs all around * We sanitized the way configuration is handled internally. Try passing --cfg-help to binaries compiled against this version. * SMPI is in better shape: lot of bugs fixing & usability improvements It may be worth trying it (even if all bugs are not gone yet) This version may have a bit more of memleaks than 3.3. This will be fixed in a latter release. ____ _ ____ _ _ / ___|(_)_ __ ___ / ___|_ __(_) __| | \___ \| | '_ ` _ \| | _| '__| |/ _` | ___) | | | | | | | |_| | | | | (_| | |____/|_|_| |_| |_|\____|_| |_|\__,_| _ _____ _____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____/ _ _ _ _ _ _ _ | |___ __ _ __| |_ ___ __| | | || | ' \| / -_) _` (_-< ' \/ -_) _` | \_,_|_||_|_\___\__,_/__/_||_\___\__,_| Mar 16 2007 homepage: http://simgrid.gforge.inria.fr/ download: http://gforge.inria.fr/frs/?group_id=12 At least, after 2 years of hard work, we managed to release the 3.3 version of the SimGrid framework. There is so many changes that the changelog only lists the most important ones, leaving alone the small improvements, bug fixing and new gadgets. In short: * Java bindings * New simulation models, and improvement of the GTNetS wrapper * Large memory savings (mainly in parser) * Faster (twice faster is not uncommon, and from 20 hours to 2 minutes on very large scenarios) * Much better scalability (tested up to 250,000 processes) * Complete regression testing framework so that you can trust the tool * Lot of neat new modules in the XBT toolbox This version was thoroughly tested on linux 32bits and 64bits (debian), as well as Mac OSX (leopard). Unfortunately, our windows-guy left, and we cannot release the windows version at the same time than the other archs. Any help would be really welcomed here. Some of the 96 included test suites are known to fail, but everything should work anyway (don't panic): * The amok module does not work in real deployment ATM, but I don't see this as release critical since I'm not aware of anyone needing this right now * Some tests about the ability of GRAS to receive messages from exotic platforms fail because I've lost access to these platforms (such as AIX) * the example/gras/pmm sometimes fails (about 1/10 of cases). I'm quite puzzled, but I suspect a bug in the test, not in the lib. * the tesh auto-tests "fail" on Mac OSX. This is because "rm -rf" is sometimes too verbose (when the OS creates hidden files, I suspect), but tesh definitely work as the rest on this arch. We hope to manage to do more timely releases in the future, even if that may turn out difficult since big stuff is coming (I don't say much here for the suspense ;) We hope you'll enjoy this new version, and please report any feedback on the list. Martin Quinson (for Da SimGrid Team) SimGrid-3.18/CMakeLists.txt0000644000175000017500000013156513217757322016073 0ustar mquinsonmquinsoncmake_minimum_required(VERSION 2.8.8) message(STATUS "Cmake version ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}.${CMAKE_PATCH_VERSION}") set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_HOME_DIRECTORY}/tools/cmake/Modules) project(SimGrid C CXX) #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# # Check for the compiler # #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# ### Need to set rc ccompiler before enable language if(WIN32) SET(CMAKE_RC_COMPILER "windres") endif() ## ## Check the C/C++ standard that we need ## See also tools/cmake/GCCFlags.cmake that sets our paranoid warning flags INCLUDE(CheckCCompilerFlag) CHECK_C_COMPILER_FLAG(-fstack-cleaner HAVE_C_STACK_CLEANER) ## Request full debugging flags set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g3") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g3") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -g") if (CMAKE_COMPILER_IS_GNUCC) if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.7") message(FATAL_ERROR "SimGrid needs at least g++ version 4.7 to compile but you have ${CMAKE_CXX_COMPILER_VERSION}." "You need a sufficient support of c++11 to compile SimGrid.") endif() endif() ## We need a decent support of the c++11 standard include(CheckCXXCompilerFlag) CHECK_CXX_COMPILER_FLAG("-std=gnu++11" COMPILER_SUPPORTS_CXX11) if(COMPILER_SUPPORTS_CXX11) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11") else() message(FATAL_ERROR "The compiler ${CMAKE_CXX_COMPILER} (v${CMAKE_CXX_COMPILER_VERSION}) has no C++11 support. " "Please install a decent C++ compiler (remove CMakeCache.txt once it's installed).") endif() ### And we need C11 standard, too include(CheckCCompilerFlag) CHECK_C_COMPILER_FLAG("-std=gnu11" COMPILER_SUPPORTS_C11) if(COMPILER_SUPPORTS_C11) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=gnu11") else() message(FATAL_ERROR "The compiler ${CMAKE_C_COMPILER} (v${CMAKE_C_COMPILER_VERSION}) has no C11 support. " "Please use a decent C compiler " "(note that c++11 support of ${CMAKE_CXX_COMPILER} seems ok).") endif() if(APPLE AND (CMAKE_C_COMPILER_VERSION VERSION_LESS "4.6")) ### gcc 4.[1-5] cannot compile ucontext on OSX message(STATUS "Ucontext can't be used with this version of gcc (must be greater than 4.5)") set(HAVE_UCONTEXT_H 0) endif() ### Check threading support set(CMAKE_THREAD_PREFER_PTHREAD TRUE) find_package(Threads) ### Setup Options include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Option.cmake) ### SMPI vs. Fortran if ((NOT DEFINED enable_smpi) OR enable_smpi) # First unset the compiler in case we're re-running cmake over a previous # configuration where it was saved as smpiff unset(CMAKE_Fortran_COMPILER) SET(SMPI_FORTRAN 0) if(enable_fortran) enable_language(Fortran OPTIONAL) endif() if(CMAKE_Fortran_COMPILER) # Fortran compiler detected: save it, then replace by smpiff set(SMPI_Fortran_COMPILER "${CMAKE_Fortran_COMPILER}" CACHE FILEPATH "The real Fortran compiler") set(CMAKE_Fortran_COMPILER smpiff) # Set flags/libs to be used in smpiff if(CMAKE_Fortran_COMPILER_ID MATCHES "GNU") set(SMPI_Fortran_FLAGS "\"-fpic\" \"-ff2c\" \"-fno-second-underscore\"") set(SMPI_Fortran_LIBS "\"-lgfortran\"") elseif(CMAKE_Fortran_COMPILER_ID MATCHES "Intel") set(SMPI_Fortran_FLAGS "\"-fPIC\" \"-nofor-main\"") set(SMPI_Fortran_LIBS "\"-lifcore\"") elseif(CMAKE_Fortran_COMPILER_ID MATCHES "PGI") # flang set(SMPI_Fortran_FLAGS "\"-fPIC\"") set(SMPI_Fortran_LIBS "") endif() set(SMPI_FORTRAN 1) endif(CMAKE_Fortran_COMPILER) endif() #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# # Build the version number # #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# set(SIMGRID_VERSION_MAJOR "3") set(SIMGRID_VERSION_MINOR "18") set(SIMGRID_VERSION_PATCH "0") # set(SIMGRID_VERSION_EXTRA "-DEVEL") # Extra words to add to version string (e.g. -rc1) set(SIMGRID_VERSION_DATE "2017") # Year for copyright information if(${SIMGRID_VERSION_PATCH} EQUAL "0") set(release_version "${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}") else() set(release_version "${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}.${SIMGRID_VERSION_PATCH}") endif() set(SIMGRID_VERSION_STRING "SimGrid version ${release_version}${SIMGRID_VERSION_EXTRA}") set(libsimgrid_version "${release_version}") set(libsimgrid-java_version "${release_version}") ### SET THE LIBRARY EXTENSION if(APPLE) set(LIB_EXE "dylib") elseif(WIN32) set(LIB_EXE "a") set(BIN_EXE ".exe") else() set(LIB_EXE "so") endif() execute_process(COMMAND ${CMAKE_LINKER} -version OUTPUT_VARIABLE LINKER_VERSION ERROR_VARIABLE LINKER_VERSION) string(REGEX MATCH "[0-9].[0-9]*" LINKER_VERSION "${LINKER_VERSION}") ### Set the library providing dlopen if("${CMAKE_SYSTEM}" MATCHES "Linux") find_library(DL_LIBRARY dl) endif("${CMAKE_SYSTEM}" MATCHES "Linux") ### Find programs and paths FIND_PROGRAM(GCOV_PATH gcov) include(FindPerl) if(NOT PERL_FOUND) message(FATAL_ERROR "Please install Perl to compile SimGrid.") endif() # tesh.py needs python 3 (or the module python-subprocess32 on python2.8+) set(PythonInterp_FIND_VERSION 3) set(PythonInterp_FIND_VERSION_COUNT 1) set(PythonInterp_FIND_VERSION_MAJOR 3) include(FindPythonInterp) if(NOT PYTHONINTERP_FOUND) message(FATAL_ERROR "Please install Python (version 3 or higher).") endif() SET(LIBRARY_OUTPUT_PATH ${CMAKE_BINARY_DIR}/lib) ### Compute the include paths set(INCLUDES ${CMAKE_BINARY_DIR} ${CMAKE_BINARY_DIR}/include ${CMAKE_HOME_DIRECTORY} ${CMAKE_HOME_DIRECTORY}/include ${CMAKE_HOME_DIRECTORY}/src/include ) if(enable_smpi) set (INCLUDES ${INCLUDES} ${CMAKE_HOME_DIRECTORY}/src/smpi/include) endif() if(NOT CMAKE_CROSSCOMPILING AND EXISTS /usr/include/) set(INCLUDES ${INCLUDES} /usr/include/) endif() if(WIN32) set(CMAKE_INCLUDE_WIN "${CMAKE_C_COMPILER}") set(CMAKE_LIB_WIN "${CMAKE_C_COMPILER}") string(REGEX REPLACE "/bin/gcc.*" "/include" CMAKE_INCLUDE_WIN "${CMAKE_INCLUDE_WIN}") string(REGEX REPLACE "/bin/gcc.*" "/lib" CMAKE_LIB_WIN "${CMAKE_LIB_WIN}") set(INCLUDES ${INCLUDES} ${CMAKE_INCLUDE_WIN}) unset(CMAKE_INCLUDE_WIN) endif() include_directories(${INCLUDES}) # library dependency cannot start with a space (CMP0004), so initialize it with something that is never desactivated. set(SIMGRID_DEP "-lm") ### Determine the assembly flavor that we need today set(HAVE_RAW_CONTEXTS 0) include(CMakeDetermineSystem) IF(CMAKE_SYSTEM_PROCESSOR MATCHES ".86|AMD64|amd64") IF(CMAKE_SIZEOF_VOID_P EQUAL 4) # 32 bits message(STATUS "System processor: i686 (${CMAKE_SYSTEM_PROCESSOR}, 32 bits)") set(SIMGRID_PROCESSOR_i686 1) set(SIMGRID_PROCESSOR_x86_64 0) ELSE() message(STATUS "System processor: x86_64 (${CMAKE_SYSTEM_PROCESSOR}, 64 bits)") set(SIMGRID_PROCESSOR_i686 0) set(SIMGRID_PROCESSOR_x86_64 1) ENDIF() if (WIN32) message(STATUS "Disable fast raw contexts on Windows.") elseif(enable_address_sanitizer) message(STATUS "Disable fast raw contexts with ASan") else() set(HAVE_RAW_CONTEXTS 1) endif() ELSE() set(SIMGRID_PROCESSOR_i686 0) set(SIMGRID_PROCESSOR_x86_64 0) ENDIF() include(CheckFunctionExists) include(CheckTypeSize) include(CheckIncludeFile) include(CheckIncludeFiles) include(CheckLibraryExists) include(CheckSymbolExists) set(HAVE_GRAPHVIZ 0) include(FindGraphviz) set(SIMGRID_HAVE_LUA 0) if(enable_lua) include(FindLuaSimgrid) endif() set(SIMGRID_HAVE_NS3 0) if(enable_ns3) include(FindNS3) if (SIMGRID_HAVE_NS3) set(SIMGRID_HAVE_NS3 1) else() message(FATAL_ERROR "Cannot find NS3. Please install it (apt-get install ns3 libns3-dev) or disable that cmake option") endif() endif() if(WIN32) set(Boost_USE_STATIC_LIBS 1) endif() set(HAVE_PAPI 0) if(enable_smpi_papi) include(FindPAPI) if (NOT HAVE_PAPI) message(FATAL_ERROR "Cannot find PAPI. Please install it (apt-get install papi-tools libpapi-dev) or disable PAPI bindings.") endif() endif() # Not finding this is perfectly OK find_package(Boost COMPONENTS unit_test_framework) if (Boost_UNIT_TEST_FRAMEWORK_FOUND) message(STATUS "Enabling the Boost-based unit tests.") else() message(STATUS "Disabling the Boost-based unit tests -- please install libboost-test-dev.") endif() find_package(Boost 1.48) if(Boost_FOUND) include_directories(${Boost_INCLUDE_DIRS}) else() if(APPLE) message(FATAL_ERROR "Boost libraries not found. Try to install them with 'sudo fink install boost1.53.nopython' (check the exact name with 'fink list boost') or 'sudo brew install boost'") else() message(FATAL_ERROR "Boost libraries not found. Install libboost-dev (>= 1.48.0).") endif() endif() find_package(Boost COMPONENTS context) set(Boost_FOUND 1) # This component is optional if(Boost_CONTEXT_FOUND) message(STATUS "Found Boost.Context") set(HAVE_BOOST_CONTEXTS 1) else() message (" boost : found.") message (" boost-context: missing. Install libboost-context-dev for this optional feature.") set(HAVE_BOOST_CONTEXTS 0) endif() # Checks for header libraries functions. CHECK_LIBRARY_EXISTS(rt clock_gettime "" HAVE_POSIX_GETTIME) if(NOT APPLE) # OS X El Capitan deprecates this function CHECK_LIBRARY_EXISTS(pthread sem_init "" HAVE_SEM_INIT_LIB) endif() CHECK_LIBRARY_EXISTS(pthread sem_open "" HAVE_SEM_OPEN_LIB) set(HAVE_PTHREAD_SETAFFINITY 0) CHECK_LIBRARY_EXISTS(pthread pthread_setaffinity_np "" HAVE_PTHREAD_SETAFFINITY) if(CMAKE_SYSTEM_NAME MATCHES "Darwin") set(CMAKE_REQUIRED_DEFINITIONS "-D_XOPEN_SOURCE=700 -D_DARWIN_C_SOURCE") elseif(MINGW) # Use the GNU version of unusual modifiers like PRIx64 add_definitions(-D__USE_MINGW_ANSI_STDIO=1) set(CMAKE_REQUIRED_DEFINITIONS "-D__USE_MINGW_ANSI_STDIO=1") else() set(CMAKE_REQUIRED_DEFINITIONS "-D_GNU_SOURCE") endif() CHECK_INCLUDE_FILE("valgrind/valgrind.h" HAVE_VALGRIND_H) CHECK_INCLUDE_FILE("unistd.h" HAVE_UNISTD_H) CHECK_INCLUDE_FILE("execinfo.h" HAVE_EXECINFO_H) CHECK_INCLUDE_FILE("signal.h" HAVE_SIGNAL_H) CHECK_INCLUDE_FILE("sys/param.h" HAVE_SYS_PARAM_H) CHECK_INCLUDE_FILE("sys/sysctl.h" HAVE_SYS_SYSCTL_H) CHECK_INCLUDE_FILE("ucontext.h" HAVE_UCONTEXT_H) CHECK_INCLUDE_FILE("linux/futex.h" HAVE_FUTEX_H) CHECK_FUNCTION_EXISTS(backtrace HAVE_BACKTRACE) CHECK_FUNCTION_EXISTS(dlfunc HAVE_DLFUNC) CHECK_FUNCTION_EXISTS(gettimeofday HAVE_GETTIMEOFDAY) CHECK_FUNCTION_EXISTS(nanosleep HAVE_NANOSLEEP) CHECK_FUNCTION_EXISTS(getdtablesize HAVE_GETDTABLESIZE) CHECK_FUNCTION_EXISTS(sysconf HAVE_SYSCONF) CHECK_FUNCTION_EXISTS(popen HAVE_POPEN) CHECK_FUNCTION_EXISTS(process_vm_readv HAVE_PROCESS_VM_READV) CHECK_FUNCTION_EXISTS(mmap HAVE_MMAP) CHECK_FUNCTION_EXISTS(mremap HAVE_MREMAP) CHECK_SYMBOL_EXISTS(vasprintf stdio.h HAVE_VASPRINTF) if(MINGW) # The detection of vasprintf fails on MinGW, assumingly because it's # defined as an inline function in stdio.h instead of a regular # function. So force the result to be 1 despite of the test. set(HAVE_VASPRINTF 1) endif() #Check if __thread is defined execute_process( COMMAND "${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_thread_storage.c -o testprog" WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_thread_storage_run) file(REMOVE testprog) if(HAVE_thread_storage_run) set(HAVE_THREAD_LOCAL_STORAGE 1) else() set(HAVE_THREAD_LOCAL_STORAGE 0) endif() CHECK_INCLUDE_FILE("sys/sendfile.h" HAVE_SENDFILE_H) CHECK_FUNCTION_EXISTS(sendfile HAVE_SENDFILE) if(HAVE_SENDFILE_H AND HAVE_SENDFILE) set(HAVE_SENDFILE 1) else() set(HAVE_SENDFILE 0) endif() if(enable_model-checking AND NOT "${CMAKE_SYSTEM}" MATCHES "Linux|FreeBSD") message(WARNING "Support for model-checking has not been enabled on ${CMAKE_SYSTEM}: disabling it") set(enable_model-checking FALSE) endif() if(HAVE_MMAP AND HAVE_THREAD_LOCAL_STORAGE) SET(HAVE_MMALLOC 1) else() SET(HAVE_MMALLOC 0) if(enable_model-checking) message(STATUS "Warning: support for model-checking has been disabled because you are missing either mmap or __thread.") endif() SET(enable_model-checking 0) endif() if(enable_jedule) set(SIMGRID_HAVE_JEDULE 1) else() set(SIMGRID_HAVE_JEDULE 0) endif() if(enable_mallocators) SET(SIMGRID_HAVE_MALLOCATOR 1) else() SET(SIMGRID_HAVE_MALLOCATOR 0) endif() include(FindLibunwind) if(HAVE_LIBUNWIND) if(NOT APPLE) SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind") else() # Apple forbids to link directly against its libunwind implementation # So let's comply and link against the System framework SET(SIMGRID_DEP "${SIMGRID_DEP} -lSystem") endif() if("${CMAKE_SYSTEM}" MATCHES "Linux|FreeBSD") set(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-ptrace") # This supposes that the host machine is either an AMD or a X86. # This is deeply wrong, and should be fixed by manually loading -lunwind-PLAT (FIXME) if(SIMGRID_PROCESSOR_x86_64) SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-x86_64") else() SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-x86") endif() endif() else() if(enable_model-checking) message(FATAL_ERROR "Please install libunwind-dev libdw-dev libelf-dev libevent-dev if you want to compile the SimGrid model checker.") endif() endif() if(enable_model-checking) find_package(Libdw REQUIRED) find_package(Libelf REQUIRED) find_package(Libevent REQUIRED) include_directories(${LIBDW_INCLUDE_DIR} ${LIBELF_INCLUDE_DIR} ${LIBEVENT_INCLUDE_DIR}) set(SIMGRID_DEP "${SIMGRID_DEP} ${LIBEVENT_LIBRARIES} ${LIBELF_LIBRARIES} ${LIBDW_LIBRARIES}") set(SIMGRID_HAVE_MC 1) if("${CMAKE_SYSTEM}" MATCHES "FreeBSD" AND enable_java) message(WARNING "FreeBSD + Model-Checking + Java = too much for now. Disabling java") set(enable_java FALSE) endif() else() SET(SIMGRID_HAVE_MC 0) set(HAVE_MMALLOC 0) endif() if (enable_model-checking AND enable_ns3) message(FATAL_ERROR "Cannot activate both model-checking and NS3 bindings: NS3 pull too much dependencies for the MC to work") endif() if(enable_smpi) SET(HAVE_SMPI 1) if("${CMAKE_SYSTEM}" MATCHES "Darwin|FreeBSD|Linux") SET(USE_LIBUTIL 0) SET(HAVE_PRIVATIZATION 1) else() message (STATUS "Warning: no support for SMPI automatic privatization on this platform") SET(HAVE_PRIVATIZATION 0) endif() else() SET(HAVE_SMPI 0) endif() #-------------------------------------------------------------------------------------------------- ### Check for GNU dynamic linker CHECK_INCLUDE_FILE("dlfcn.h" HAVE_DLFCN_H) if (HAVE_DLFCN_H) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_gnu_dynlinker.c ${DL_LIBRARY} -o test_gnu_ld WORKING_DIRECTORY ${CMAKE_BINARY_DIR} OUTPUT_VARIABLE HAVE_GNU_LD_compil ) if(HAVE_GNU_LD_compil) set(HAVE_GNU_LD 0) message(STATUS "Warning: test program toward GNU ld failed to compile:") message(STATUS "${HAVE_GNU_LD_comp_output}") else() execute_process(COMMAND ./test_gnu_ld WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_GNU_LD_run OUTPUT_VARIABLE var_exec ) if(NOT HAVE_GNU_LD_run) set(HAVE_GNU_LD 1) message(STATUS "We are using GNU dynamic linker") else() set(HAVE_GNU_LD 0) message(STATUS "Warning: error while checking for GNU ld:") message(STATUS "Test output: '${var_exec}'") message(STATUS "Exit status: ${HAVE_GNU_LD_run}") endif() file(REMOVE test_gnu_ld) endif() endif() #-------------------------------------------------------------------------------------------------- ### Initialize of CONTEXT THREADS set(HAVE_THREAD_CONTEXTS 0) if(CMAKE_USE_PTHREADS_INIT) ### Test that we have a way to create semaphores if(HAVE_SEM_OPEN_LIB) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_sem_open.c -lpthread -o sem_open WORKING_DIRECTORY ${CMAKE_BINARY_DIR} OUTPUT_VARIABLE HAVE_SEM_OPEN_compil ) # Test sem_open by compiling: if(HAVE_SEM_OPEN_compil) set(HAVE_SEM_OPEN 0) message(STATUS "Warning: sem_open not compilable") message(STATUS "HAVE_SEM_OPEN_comp_output: ${HAVE_SEM_OPEN_comp_output}") else() set(HAVE_SEM_OPEN 1) message(STATUS "sem_open is compilable") endif() # If we're not crosscompiling, we check by executing the program: if (HAVE_SEM_OPEN AND NOT CMAKE_CROSSCOMPILING) execute_process(COMMAND ./sem_open WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_OPEN_run OUTPUT_VARIABLE var_compil ) if (NOT HAVE_SEM_OPEN_run) set(HAVE_SEM_OPEN 1) message(STATUS "sem_open is executable") else() set(HAVE_SEM_OPEN 0) if(EXISTS "${CMAKE_BINARY_DIR}/sem_open") message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_open exists!") else() message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_open not exists!") endif() message(STATUS "Warning: sem_open not executable") message(STATUS "Compilation output: '${var_compil}'") message(STATUS "Exit result of sem_open: ${HAVE_SEM_OPEN_run}") endif() endif() file(REMOVE sem_open) else() set(HAVE_SEM_OPEN 0) endif() if(HAVE_SEM_INIT_LIB) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_sem_init.c -lpthread -o sem_init WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_INIT_run OUTPUT_VARIABLE HAVE_SEM_INIT_compil) # Test sem_init by compiling: if(HAVE_SEM_INIT_compil) set(HAVE_SEM_INIT 0) message(STATUS "Warning: sem_init not compilable") message(STATUS "HAVE_SEM_INIT_comp_output: ${HAVE_SEM_OPEN_comp_output}") else() set(HAVE_SEM_INIT 1) message(STATUS "sem_init is compilable") endif() # If we're not crosscompiling, we check by executing the program: if (HAVE_SEM_INIT AND NOT CMAKE_CROSSCOMPILING) execute_process(COMMAND ./sem_init WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_INIT_run OUTPUT_VARIABLE var_compil ) if (NOT HAVE_SEM_INIT_run) set(HAVE_SEM_INIT 1) message(STATUS "sem_init is executable") else() set(HAVE_SEM_INIT 0) if(EXISTS "${CMAKE_BINARY_DIR}/sem_init") message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_init exists!") else() message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_init not exists!") endif() message(STATUS "Warning: sem_init not executable") message(STATUS "Compilation output: '${var_compil}'") message(STATUS "Exit result of sem_init: ${HAVE_SEM_INIT_run}") endif() endif() file(REMOVE sem_init) else() set(HAVE_SEM_INIT 0) endif() if(NOT HAVE_SEM_OPEN AND NOT HAVE_SEM_INIT) message(FATAL_ERROR "Semaphores are not usable (neither sem_open nor sem_init is both compilable and executable), but they are mandatory to threads (you may need to mount /dev).") endif() set(HAVE_THREAD_CONTEXTS 1) message(STATUS "Support for thread context factory ok.") endif() set(HAVE_UCONTEXT_CONTEXTS 0) if(NOT HAVE_UCONTEXT_H) message(STATUS "No ucontext factory: not found.") elseif(APPLE) message(STATUS "No ucontext factory: Apple don't want us to use them.") set(HAVE_UCONTEXT_H 0) elseif(enable_address_sanitizer) message(STATUS "No ucontext factory: ASan does not support it (see http://code.google.com/p/address-sanitizer/issues/detail?id=189)") else() try_compile(compile_makecontext ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_makecontext.c OUTPUT_VARIABLE compile_makecontext_output) #If can have both context if(compile_makecontext) set(HAVE_UCONTEXT_CONTEXTS 1) message(STATUS "Support for ucontext factory ok.") else() message(STATUS "Error: exists, but makecontext is not compilable. Compilation output:\n ${compile_makecontext_output}") message(STATUS "No ucontext factory: makecontext() is not compilable.") endif() # Stack setup (size and address) try_run(RUN_makecontext_VAR COMPILE_makecontext_VAR ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_stacksetup.c RUN_OUTPUT_VARIABLE stack_setup) LIST(LENGTH stack_setup stack_setup_len) if("${stack_setup_len}" STREQUAL "2") LIST(GET stack_setup 0 makecontext_addr) LIST(GET stack_setup 1 makecontext_size) set(sg_makecontext_stack_addr "#define sg_makecontext_stack_addr(skaddr) (${makecontext_addr})") set(sg_makecontext_stack_size "#define sg_makecontext_stack_size(sksize) (${makecontext_size})") else() message(FATAL_ERROR "Could not figure out the stack setup. Compil: ${RUN_makecontext_VAR}. Exec: ${COMPILE_makecontext_VAR}. Output: ${stack_setup}") endif() endif() # Stack growth direction (upward or downward). Used for the following contexts: SysV, raw, Boost try_run(RUN_stackgrowth_VAR COMPILE_stackgrowth_VAR ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_stackgrowth.c RUN_OUTPUT_VARIABLE stack COPY_FILE test_stackgrowth) if("${stack}" STREQUAL "down") set(PTH_STACKGROWTH "-1") elseif("${stack}" STREQUAL "up") set(PTH_STACKGROWTH "1") else() if("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "x86_64") set(PTH_STACKGROWTH "-1") elseif("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "i686") set(PTH_STACKGROWTH "-1") else() message(FATAL_ERROR "Could not figure out the stack direction. Test prog returned: ${stack}; CMAKE_SYSTEM_PROCESSOR: ${CMAKE_SYSTEM_PROCESSOR}.") endif() endif() # If the test ran well, remove the test binary file(REMOVE test_stackgrowth) #-------------------------------------------------------------------------------------------------- ### check for addr2line find_path(ADDR2LINE NAMES addr2line PATHS NO_DEFAULT_PATHS) if(ADDR2LINE) set(ADDR2LINE "${ADDR2LINE}/addr2line") endif() ############### ## GIT version check ## if(EXISTS ${CMAKE_HOME_DIRECTORY}/.git/) execute_process( COMMAND git remote COMMAND head -n 1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE remote OUTPUT_STRIP_TRAILING_WHITESPACE) #message(STATUS "Git remote: ${remote}") execute_process(COMMAND git config --get remote.${remote}.url WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE url OUTPUT_STRIP_TRAILING_WHITESPACE) #message(STATUS "Git url: ${url}") if(url) execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log --pretty=oneline --abbrev-commit -1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE GIT_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE) message(STATUS "Git version: ${GIT_VERSION}") execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log -n 1 --pretty=format:%ai . WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE GIT_DATE OUTPUT_STRIP_TRAILING_WHITESPACE) message(STATUS "Git date: ${GIT_DATE}") string(REGEX REPLACE " .*" "" GIT_VERSION "${GIT_VERSION}") execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log --pretty=format:%H -1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE SIMGRID_GITHASH OUTPUT_STRIP_TRAILING_WHITESPACE) endif() elseif(EXISTS ${CMAKE_HOME_DIRECTORY}/.gitversion) FILE(STRINGS ${CMAKE_HOME_DIRECTORY}/.gitversion GIT_VERSION) else() set(GIT_VERSION "none, release version") endif() ### Setup gcc & clang flags if (NOT MSVC) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/GCCFlags.cmake) endif() ### Generate the required headers and scripts ############################################# # Avoid triggering a (full) rebuild by touching the files if they did not really change configure_file("${CMAKE_HOME_DIRECTORY}/tools/cmake/src/internal_config.h.in" "${CMAKE_BINARY_DIR}/src/internal_config.h.generated" @ONLY IMMEDIATE) configure_file("${CMAKE_HOME_DIRECTORY}/include/simgrid_config.h.in" "${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated" @ONLY IMMEDIATE) execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/src/internal_config.h.generated ${CMAKE_BINARY_DIR}/src/internal_config.h) execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated ${CMAKE_BINARY_DIR}/include/simgrid_config.h) file(REMOVE ${CMAKE_BINARY_DIR}/src/internal_config.h.generated) file(REMOVE ${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated) # We need two versions of the SMPI scripts because they contain the path to the library # so, it depends of whether SimGrid is installed, or run from the sources (during the build) file(READ ${CMAKE_HOME_DIRECTORY}/src/smpi/smpitools.sh SMPITOOLS_SH) # Definitions shared amongst all SMPI scripts, inlined in each of them ### SMPI script used when simgrid is installed set(exec_prefix ${CMAKE_INSTALL_PREFIX}) set(includeflag "-I${CMAKE_INSTALL_PREFIX}/include -I${CMAKE_INSTALL_PREFIX}/include/smpi") set(includedir "${CMAKE_INSTALL_PREFIX}/include") set(libdir ${exec_prefix}/lib) set(CMAKE_SMPI_COMMAND "export LD_LIBRARY_PATH=\"${CMAKE_INSTALL_PREFIX}/lib") if(NS3_LIBRARY_PATH) set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:${NS3_LIBRARY_PATH}") endif() set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:\${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\"") set(SMPIMAIN smpimain) configure_file(${CMAKE_HOME_DIRECTORY}/include/smpi/mpif.h.in ${CMAKE_BINARY_DIR}/include/smpi/mpif.h @ONLY) foreach(script cc cxx ff f90 run) configure_file(${CMAKE_HOME_DIRECTORY}/src/smpi/smpi${script}.in ${CMAKE_BINARY_DIR}/bin/smpi${script} @ONLY) endforeach() ### SMPI scripts used when compiling simgrid set(exec_prefix "${CMAKE_BINARY_DIR}/smpi_script/") set(includeflag "-I${CMAKE_HOME_DIRECTORY}/include -I${CMAKE_HOME_DIRECTORY}/include/smpi") set(includeflag "${includeflag} -I${CMAKE_BINARY_DIR}/include -I${CMAKE_BINARY_DIR}/include/smpi") set(includedir "${CMAKE_HOME_DIRECTORY}/include") set(libdir "${CMAKE_BINARY_DIR}/lib") set(CMAKE_SMPI_COMMAND "export LD_LIBRARY_PATH=\"${CMAKE_BINARY_DIR}/lib") if(NS3_LIBRARY_PATH) set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:${NS3_LIBRARY_PATH}") endif() set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:\${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\"") set(SMPIMAIN ${CMAKE_BINARY_DIR}/bin/smpimain) foreach(script cc cxx ff f90 run) configure_file(${CMAKE_HOME_DIRECTORY}/src/smpi/smpi${script}.in ${CMAKE_BINARY_DIR}/smpi_script/bin/smpi${script} @ONLY) endforeach() if(NOT WIN32) foreach(script cc cxx ff f90 run) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpi${script}) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/smpi_script/bin/smpi${script}) endforeach() endif() set(generated_headers_to_install ${CMAKE_CURRENT_BINARY_DIR}/include/smpi/mpif.h ${CMAKE_CURRENT_BINARY_DIR}/include/simgrid_config.h ) set(generated_headers ${CMAKE_CURRENT_BINARY_DIR}/src/internal_config.h ) set(generated_files_to_clean ${generated_headers} ${generated_headers_to_install} ${CMAKE_BINARY_DIR}/bin/smpicc ${CMAKE_BINARY_DIR}/bin/smpicxx ${CMAKE_BINARY_DIR}/bin/smpiff ${CMAKE_BINARY_DIR}/bin/smpif90 ${CMAKE_BINARY_DIR}/bin/smpirun ${CMAKE_BINARY_DIR}/bin/colorize ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml ${CMAKE_BINARY_DIR}/examples/smpi/tracing/smpi_traced.trace ) if(NOT "${CMAKE_BINARY_DIR}" STREQUAL "${CMAKE_HOME_DIRECTORY}") configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions0.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions1.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_allReduce.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allReduce.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_barrier.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_barrier.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_bcast.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_bcast.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_with_isend.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_with_isend.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_alltoall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoall.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_alltoallv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoallv.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_waitall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_waitall.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_reducescatter.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_reducescatter.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_gather.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_gather.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_allgatherv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allgatherv.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/hostfile ${CMAKE_BINARY_DIR}/teshsuite/smpi/hostfile COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/description_file ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/description_file COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/README ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/README COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/smpi_replay.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/smpi_replay.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt COPYONLY) set(generated_files_to_clean ${generated_files_to_clean} ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allReduce.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_barrier.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_bcast.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_with_isend.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoallv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_waitall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_gather.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allgatherv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_reducescatter.txt ${CMAKE_BINARY_DIR}/teshsuite/smpi/hostfile ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/description_file ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/README ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/smpi_replay.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt ) endif() SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${generated_files_to_clean}") ### Define source packages for Libs include(${CMAKE_HOME_DIRECTORY}/tools/cmake/DefinePackages.cmake) ### Build some Maintainer files include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MaintainerMode.cmake) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/UnitTesting.cmake) ### Make Libs if(NOT WIN32) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MakeLib.cmake) else() include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MakeLibWin.cmake) endif() if(enable_java) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Java.cmake) endif() ### Make tests if(enable_memcheck_xml) set(enable_memcheck true) endif() INCLUDE(CTest) ENABLE_TESTING() include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Tests.cmake) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/CTestConfig.cmake) ### Define subdirectories foreach(cmakefile ${CMAKEFILES_TXT}) string(REPLACE "/CMakeLists.txt" "" repository ${cmakefile}) add_subdirectory("${CMAKE_HOME_DIRECTORY}/${repository}") endforeach() ### Setup the distrib include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Distrib.cmake) ### Build the docs if asked to include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Documentation.cmake) ### Print the result of configuration message("") message("##########################################") message("#### Content of src/internal_config.h ####") message("##########################################") file(STRINGS ${CMAKE_CURRENT_BINARY_DIR}/src/internal_config.h config_output) LIST(REMOVE_AT config_output 0 1 2 3 4 5 6 7 8 9 10) # Pass the file header foreach(line ${config_output}) message(" ${line}") endforeach() message("##########################################") message("#### Content of simgrid_config.h ####") message("##########################################") file(STRINGS ${CMAKE_CURRENT_BINARY_DIR}/include/simgrid_config.h config_output) LIST(REMOVE_AT config_output 0 1 2 3 4 5 6 7 8 9 -1) # Pass the file header foreach(line ${config_output}) message(" ${line}") endforeach() message("##########################################") message("#### End of configuration headers ####") message("##########################################") message("\nConfiguration of package `simgrid':") message(" Home directory ..............: ${CMAKE_HOME_DIRECTORY}") message(" Build Name ..................: ${BUILDNAME}") message(" Cmake Generator .............: ${CMAKE_GENERATOR}") message(" Site ........................: ${SITE}") message(" Install prefix ..............: ${CMAKE_INSTALL_PREFIX}") if(release) message(" Release .....................: simgrid-${release_version}${SIMGRID_VERSION_EXTRA} (release build)") else() message(" Release .....................: simgrid-${release_version}${SIMGRID_VERSION_EXTRA} (development build)") endif() message("") message(" Compiler: C .................: ${CMAKE_C_COMPILER} (id: ${CMAKE_C_COMPILER_ID})") message(" version .............: ${CMAKE_C_COMPILER_VERSION}") message(" is gnu ..............: ${CMAKE_COMPILER_IS_GNUCC}") message(" Compiler: C++ ...............: ${CMAKE_CXX_COMPILER} (id: ${CMAKE_CXX_COMPILER_ID})") message(" version .............: ${CMAKE_CXX_COMPILER_VERSION}") if(${Java_FOUND}) message(" Compiler: Javac .............: ${Java_JAVAC_EXECUTABLE}") message(" version .............: ${Java_VERSION_STRING}") endif() if(CMAKE_Fortran_COMPILER) message(" Compiler: Fortran ...........: ${SMPI_Fortran_COMPILER} (id: ${CMAKE_Fortran_COMPILER_ID})") message(" version .............: ${CMAKE_Fortran_COMPILER_VERSION}") endif() message(" Linker: .....................: ${CMAKE_LINKER}") message(" version .............: ${LINKER_VERSION}") message(" Make program: ...............: ${CMAKE_MAKE_PROGRAM}") message("") message(" CFlags ......................: ${CMAKE_C_FLAGS}") message(" CXXFlags ....................: ${CMAKE_CXX_FLAGS}") message(" LDFlags .....................: ${CMAKE_C_LINK_FLAGS}") message(" with LTO ....................: ${enable_lto}") message("") if (SIMGRID_HAVE_NS3) message(" Compile NS-3 ................: yes (path: ${NS3_PATH})") else() message(" Compile NS-3 ................: NO (hint: ${NS3_HINT})") endif() if (${Java_FOUND}) message(" Compile Java ................: yes") message(" Native lib in jar .........: ${enable_lib_in_jar}") else() message(" Compile Java ................: NO") endif() message(" Compile Lua .................: ${SIMGRID_HAVE_LUA}") message(" Compile Smpi ................: ${HAVE_SMPI}") message(" Smpi fortran ..............: ${SMPI_FORTRAN}") message(" MPICH3 testsuite ..........: ${enable_smpi_MPICH3_testsuite}") message(" Privatization .............: ${HAVE_PRIVATIZATION}") message(" PAPI support...............: ${HAVE_PAPI}") message(" Compile Boost.Context support: ${HAVE_BOOST_CONTEXTS}") message("") message(" Maintainer mode .............: ${enable_maintainer_mode}") message(" Documentation................: ${enable_documentation}") message(" Model checking ..............: ${SIMGRID_HAVE_MC}") message(" Jedule mode ................: ${SIMGRID_HAVE_JEDULE}") message(" Graphviz mode ...............: ${HAVE_GRAPHVIZ}") message(" Mallocators .................: ${enable_mallocators}") message("") message(" Simgrid dependencies ........: ${SIMGRID_DEP}") message("") execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_BINARY_DIR}/Testing/Notes/) file(WRITE ${PROJECT_BINARY_DIR}/Testing/Notes/Build "GIT version : ${GIT_VERSION}\n") file(APPEND ${PROJECT_BINARY_DIR}/Testing/Notes/Build "Release : simgrid-${release_version}\n") INCLUDE(Dart) SimGrid-3.18/examples/0000755000175000017500000000000013217757324015140 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/0000755000175000017500000000000013217757326016063 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/energy/0000755000175000017500000000000013217757326017354 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/energy/consumption/0000755000175000017500000000000013217757334021731 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/energy/consumption/Main.java0000644000175000017500000000200313217757326023454 0ustar mquinsonmquinson/* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.consumption; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.energyInit(); Msg.init(args); if (args.length < 1) { Msg.info("Usage : Energy platform_file"); Msg.info("Usage : Energy ../platforms/energy_platform.xml"); System.exit(1); } /* Construct the platform */ Msg.createEnvironment(args[0]); /* Instantiate a process */ new EnergyConsumer("MyHost1","energyConsumer").start(); /* Execute the simulation */ Msg.run(); Msg.info("Total simulation time: "+ Msg.getClock()); } } SimGrid-3.18/examples/java/energy/consumption/energy-consumption.tesh0000644000175000017500000000223013217757334026460 0ustar mquinsonmquinson#! tesh ! timeout 15 $ java -classpath ${classpath:=.} energy/consumption/Main ${srcdir:=.}/../platforms/energy_platform.xml > [0.000000] [java/INFO] Using regular java threads. > [MyHost1:energyConsumer:(1) 0.000000] [java/INFO] Energetic profile: 100.0:120.0:200.0, 93.0:110.0:170.0, 90.0:105.0:150.0 > [MyHost1:energyConsumer:(1) 0.000000] [java/INFO] Initial peak speed= 1.0E8 flop/s; Energy dissipated = 0.0 J > [MyHost1:energyConsumer:(1) 10.000000] [java/INFO] Currently consumed energy after sleeping 10 sec: 1000.0 > [MyHost1:energyConsumer:(1) 20.000000] [java/INFO] Currently consumed energy after executing 1E9 flops: 2200.0 > [20.000000] [surf_energy/INFO] Total energy consumption: 6200.000000 Joules (used hosts: 2200.000000 Joules; unused/idle hosts: 4000.000000) > [20.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... > [20.000000] [java/INFO] Total simulation time: 20.0 > [20.000000] [surf_energy/INFO] Energy consumption of host MyHost1: 2200.000000 Joules > [20.000000] [surf_energy/INFO] Energy consumption of host MyHost2: 2000.000000 Joules > [20.000000] [surf_energy/INFO] Energy consumption of host MyHost3: 2000.000000 Joules SimGrid-3.18/examples/java/energy/consumption/EnergyConsumer.java0000644000175000017500000000225713217757326025550 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.consumption; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class EnergyConsumer extends Process { public EnergyConsumer(String hostname, String name) throws HostNotFoundException { super(hostname,name); } public void main(String[] args) throws MsgException { Msg.info("Energetic profile: " + getHost().getProperty("watt_per_state")); Msg.info("Initial peak speed= " + getHost().getSpeed() + " flop/s; Energy dissipated = " + getHost().getConsumedEnergy() + " J"); this.waitFor(10); Msg.info("Currently consumed energy after sleeping 10 sec: "+getHost().getConsumedEnergy()); new Task(null, 1E9, 0).execute(); Msg.info("Currently consumed energy after executing 1E9 flops: "+getHost().getConsumedEnergy()); } } SimGrid-3.18/examples/java/energy/vm/0000755000175000017500000000000013217757334017775 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/energy/vm/Main.java0000644000175000017500000000161113217757326021524 0ustar mquinsonmquinson/* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.vm; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws HostNotFoundException { Msg.energyInit(); Msg.init(args); if (args.length < 1) { Msg.info("Usage: Main ../platforms/energy_platform_file.xml"); System.exit(1); } /* construct the platform */ Msg.createEnvironment(args[0]); /* Create and start a runner for the experiment */ new EnergyVMRunner(Host.all()[0],"energy VM runner",null).start(); Msg.run(); } } SimGrid-3.18/examples/java/energy/vm/energy-vm.tesh0000644000175000017500000000327313217757334022600 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} energy/vm/Main ${srcdir:=.}/../platforms/energy_platform.xml > [0.000000] [java/INFO] Using regular java threads. > [MyHost1:energy VM runner:(1) 0.000000] [java/INFO] Creating and starting two VMs > [MyHost1:energy VM runner:(1) 0.000000] [java/INFO] Create two tasks on Host1: one inside a VM, the other directly on the host > [MyHost1:energy VM runner:(1) 0.000000] [java/INFO] Create two tasks on Host2: both directly on the host > [MyHost1:energy VM runner:(1) 0.000000] [java/INFO] Create two tasks on Host3: both inside a VM > [MyHost1:energy VM runner:(1) 0.000000] [java/INFO] Wait 5 seconds. The tasks are still running (they run for 3 seconds, but 2 tasks are co-located, so they run for 6 seconds) > [MyHost1:energy VM runner:(1) 5.000000] [java/INFO] Wait another 5 seconds. The tasks stop at some point in between > [MyHost2:p22:(5) 6.000000] [java/INFO] This worker is done. > [MyHost3:p312:(7) 6.000000] [java/INFO] This worker is done. > [MyHost3:p31:(6) 6.000000] [java/INFO] This worker is done. > [vmHost1:p12:(3) 6.000000] [java/INFO] This worker is done. > [vmHost1:p11:(2) 6.000000] [java/INFO] This worker is done. > [vmHost3:p21:(4) 6.000000] [java/INFO] This worker is done. > [10.000000] [surf_energy/INFO] Total energy consumption: 4320.000000 Joules (used hosts: 4320.000000 Joules; unused/idle hosts: 0.000000) > [10.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... > [10.000000] [surf_energy/INFO] Energy consumption of host MyHost1: 1120.000000 Joules > [10.000000] [surf_energy/INFO] Energy consumption of host MyHost2: 1600.000000 Joules > [10.000000] [surf_energy/INFO] Energy consumption of host MyHost3: 1600.000000 Joules SimGrid-3.18/examples/java/energy/vm/EnergyVMRunner.java0000644000175000017500000000507013217757326023531 0ustar mquinsonmquinson/* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.vm; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.Task; import org.simgrid.msg.TaskCancelledException; import org.simgrid.msg.VM; /* This class is a process in charge of running the test. It creates and starts the VMs, and fork processes within VMs */ public class EnergyVMRunner extends Process { public class DummyProcess extends Process { public DummyProcess (Host host, String name) { super(host, name); } @Override public void main(String[] strings) { Task task = new Task(this.getHost().getName()+"-task", 300E6 , 0); try { task.execute(); } catch (HostFailureException | TaskCancelledException e) { Msg.error(e.getMessage()); e.printStackTrace(); } Msg.info("This worker is done."); } } EnergyVMRunner(Host host, String name, String[] args) throws HostNotFoundException { super(host, name, args); } @Override public void main(String[] strings) throws HostNotFoundException, HostFailureException { /* get hosts */ Host host1 = Host.getByName("MyHost1"); Host host2 = Host.getByName("MyHost2"); Host host3 = Host.getByName("MyHost3"); Msg.info("Creating and starting two VMs"); VM vmHost1 = new VM(host1, "vmHost1", 2048, 10, 50); vmHost1.start(); VM vmHost2 = new VM(host2, "vmHost3", 2048, 10, 50); vmHost2.start(); Msg.info("Create two tasks on Host1: one inside a VM, the other directly on the host"); new DummyProcess (vmHost1, "p11").start(); new DummyProcess (vmHost1, "p12").start(); Msg.info("Create two tasks on Host2: both directly on the host"); new DummyProcess (vmHost2, "p21").start(); new DummyProcess (host2, "p22").start(); Msg.info("Create two tasks on Host3: both inside a VM"); new DummyProcess (host3, "p31").start(); new DummyProcess (host3, "p312").start(); Msg.info("Wait 5 seconds. The tasks are still running (they run for 3 seconds, but 2 tasks are co-located, " + "so they run for 6 seconds)"); waitFor(5); Msg.info("Wait another 5 seconds. The tasks stop at some point in between"); waitFor(5); vmHost1.destroy(); vmHost2.destroy(); } } SimGrid-3.18/examples/java/energy/pstate/0000755000175000017500000000000013217757334020653 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/energy/pstate/PstateRunner.java0000644000175000017500000000476213217757326024162 0ustar mquinsonmquinson/* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.pstate; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.Task; import org.simgrid.msg.TaskCancelledException; /* This class is a process in charge of running the test. It creates and starts the VMs, and fork processes within VMs */ public class PstateRunner extends Process { public class DVFS extends Process { public DVFS (Host host, String name) { super(host, name); } @Override public void main(String[] strings) throws HostNotFoundException, HostFailureException, TaskCancelledException { double workload = 100E6; int newPstate = 2; Host host = getHost(); int nb = host.getPstatesCount(); Msg.info("Count of Processor states="+ nb); double currentPeak = host.getCurrentPowerPeak(); Msg.info("Current power peak=" + currentPeak); // Run a task Task task1 = new Task("t1", workload, 0); task1.execute(); double taskTime = Msg.getClock(); Msg.info("Task1 simulation time: "+ taskTime); // Change power peak if ((newPstate >= nb) || (newPstate < 0)){ Msg.info("Cannot set pstate "+newPstate+"%d, host supports only "+nb+" pstates."); return; } double peakAt = host.getPowerPeakAt(newPstate); Msg.info("Changing power peak value to "+peakAt+" (at index "+newPstate+")"); host.setPstate(newPstate); currentPeak = host.getCurrentPowerPeak(); Msg.info("Current power peak="+ currentPeak); // Run a second task new Task("t1", workload, 0).execute(); taskTime = Msg.getClock() - taskTime; Msg.info("Task2 simulation time: "+ taskTime); // Verify the default pstate is set to 0 host = Host.getByName("MyHost2"); int nb2 = host.getPstatesCount(); Msg.info("Count of Processor states="+ nb2); double currentPeak2 = host.getCurrentPowerPeak(); Msg.info("Current power peak=" + currentPeak2); } } PstateRunner(Host host, String name, String[] args) { super(host, name, args); } @Override public void main(String[] strings) throws HostNotFoundException, HostFailureException { new DVFS (Host.getByName("MyHost1"), "dvfs_test").start(); new DVFS (Host.getByName("MyHost2"), "dvfs_test").start(); } } SimGrid-3.18/examples/java/energy/pstate/Main.java0000644000175000017500000000147513217757326022412 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.pstate; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.energyInit(); Msg.init(args); if (args.length < 1) { Msg.info("Usage: Main ../platforms/energy_platform_file.xml"); System.exit(1); } /* construct the platform */ Msg.createEnvironment(args[0]); /* Create and start a runner for the experiment */ new PstateRunner(Host.all()[0],"pstate runner",null).start(); Msg.run(); } } SimGrid-3.18/examples/java/energy/pstate/energy-pstate.tesh0000644000175000017500000000347413217757334024337 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} energy/pstate/Main ${srcdir:=.}/../platforms/energy_platform.xml > [0.000000] [java/INFO] Using regular java threads. > [MyHost1:dvfs_test:(2) 0.000000] [java/INFO] Count of Processor states=3 > [MyHost1:dvfs_test:(2) 0.000000] [java/INFO] Current power peak=1.0E8 > [MyHost2:dvfs_test:(3) 0.000000] [java/INFO] Count of Processor states=3 > [MyHost2:dvfs_test:(3) 0.000000] [java/INFO] Current power peak=1.0E8 > [MyHost1:dvfs_test:(2) 1.000000] [java/INFO] Task1 simulation time: 1.0 > [MyHost2:dvfs_test:(3) 1.000000] [java/INFO] Task1 simulation time: 1.0 > [MyHost1:dvfs_test:(2) 1.000000] [java/INFO] Changing power peak value to 2.0E7 (at index 2) > [MyHost2:dvfs_test:(3) 1.000000] [java/INFO] Changing power peak value to 2.0E7 (at index 2) > [MyHost1:dvfs_test:(2) 1.000000] [java/INFO] Current power peak=2.0E7 > [MyHost2:dvfs_test:(3) 1.000000] [java/INFO] Current power peak=2.0E7 > [MyHost1:dvfs_test:(2) 6.000000] [java/INFO] Task2 simulation time: 5.0 > [MyHost1:dvfs_test:(2) 6.000000] [java/INFO] Count of Processor states=3 > [MyHost1:dvfs_test:(2) 6.000000] [java/INFO] Current power peak=2.0E7 > [MyHost2:dvfs_test:(3) 6.000000] [java/INFO] Task2 simulation time: 5.0 > [MyHost2:dvfs_test:(3) 6.000000] [java/INFO] Count of Processor states=3 > [MyHost2:dvfs_test:(3) 6.000000] [java/INFO] Current power peak=2.0E7 > [6.000000] [surf_energy/INFO] Total energy consumption: 2195.000000 Joules (used hosts: 1595.000000 Joules; unused/idle hosts: 600.000000) > [6.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... > [6.000000] [surf_energy/INFO] Energy consumption of host MyHost1: 645.000000 Joules > [6.000000] [surf_energy/INFO] Energy consumption of host MyHost2: 950.000000 Joules > [6.000000] [surf_energy/INFO] Energy consumption of host MyHost3: 600.000000 Joules SimGrid-3.18/examples/java/.classpath0000644000175000017500000000061213217757322020041 0ustar mquinsonmquinson SimGrid-3.18/examples/java/CMakeLists.txt0000644000175000017500000001173313217757322020624 0ustar mquinsonmquinsonset(app-bittorrent_files Main Common Connection MessageTask Peer Tracker TrackerTask) set(app-centralizedmutex_files Main Coordinator GrantTask Node ReleaseTask RequestTask) set(app-masterworker_files Main Master Worker) set(app-pingpong_files Main PingPongTask Receiver Sender) set(app-tokenring_files Main RelayRunner) set(async-waitall_files Main Receiver Sender) set(async-yield_files Main Yielder) set(async-dsend_files Main Receiver Sender) set(cloud-masterworker_files Main Master Worker) set(cloud-migration_files Main Daemon Test TestHostOnOff XVM) set(dht-chord_files Main ChordTask Common FindSuccessorAnswerTask FindSuccessorTask GetPredecessorAnswerTask GetPredecessorTask Node NotifyTask) set(dht-kademlia_files Main Answer Bucket Common Contact FindNodeAnswerTask FindNodeTask KademliaTask Node PingAnswerTask PingTask RoutingTable) set(trace-pingpong_files Main PingPongTask Receiver Sender) set(energy-consumption_files Main EnergyConsumer) set(energy-pstate_files Main PstateRunner) set(energy-vm_files Main EnergyVMRunner) set(io-file_files Main Node) set(io-storage_files Main Client) set(process-kill_files Main Killer Victim) set(process-migration_files Main Emigrant Policeman) set(process-startkilltime_files Main Sleeper) set(process-suspend_files Main DreamMaster LazyGuy) set(task-priority_files Main Test) foreach (example app-bittorrent app-centralizedmutex app-masterworker app-pingpong app-tokenring async-yield async-waitall async-dsend cloud-migration cloud-masterworker dht-chord dht-kademlia energy-consumption energy-pstate energy-vm io-file io-storage process-kill process-migration process-startkilltime process-suspend task-priority trace-pingpong) string (REPLACE "-" "/" example_dir ${example}) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/${example_dir}) foreach (filename ${${example}_files} ) set( ${example}_sources "${${example}_sources}" "${srcdir}/${filename}.java") endforeach() if(enable_java) add_custom_command( COMMENT "Building java-${example}..." OUTPUT ${example_dir}/java-${example}_compiled DEPENDS ${example_sources} simgrid-java_jar ${SIMGRID_JAR} COMMAND ${CMAKE_COMMAND} -E make_directory ${example_dir} COMMAND ${JAVA_COMPILE} -classpath ${SIMGRID_JAR} -d ${CMAKE_CURRENT_BINARY_DIR} ${${example}_sources} COMMAND ${CMAKE_COMMAND} -E remove ${example_dir}/java-${example}_compiled COMMAND ${CMAKE_COMMAND} -E touch ${example_dir}/java-${example}_compiled ) add_custom_target(java-${example} ALL DEPENDS ${example_dir}/java-${example}_compiled) endif() set(examples_src ${examples_src} ${${example}_sources}) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${example_dir}/${example}.tesh) endforeach() set(examples_src ${examples_src} PARENT_SCOPE) set(tesh_files ${tesh_files} PARENT_SCOPE) set(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/bittorrent/generate.py PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/masterworker/README ${CMAKE_CURRENT_SOURCE_DIR}/cloud/migration/README PARENT_SCOPE) set(xml_files ${xml_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/bittorrent/bittorrent.xml ${CMAKE_CURRENT_SOURCE_DIR}/app/centralizedmutex/centralizedmutex.xml ${CMAKE_CURRENT_SOURCE_DIR}/app/masterworker/masterworker.xml ${CMAKE_CURRENT_SOURCE_DIR}/dht/chord/chord.xml ${CMAKE_CURRENT_SOURCE_DIR}/dht/kademlia/kademlia.xml ${CMAKE_CURRENT_SOURCE_DIR}/process/startkilltime/startkilltime.xml ${CMAKE_CURRENT_SOURCE_DIR}/task/priority/priority.xml PARENT_SCOPE) if(enable_java) foreach (example app-bittorrent app-centralizedmutex app-masterworker app-pingpong app-tokenring async-yield async-waitall async-dsend cloud-migration cloud-masterworker dht-chord dht-kademlia energy-consumption energy-pstate energy-vm io-file io-storage process-kill process-migration process-startkilltime process-suspend task-priority trace-pingpong) string (REPLACE "-" "/" example_dir ${example}) ADD_TESH(java-${example} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/java --setenv LD_LIBRARY_PATH=${CMAKE_BINARY_DIR}/lib --setenv classpath=${TESH_CLASSPATH} --cd ${CMAKE_BINARY_DIR}/examples/java/${example_dir} ${CMAKE_HOME_DIRECTORY}/examples/java/${example_dir}/${example}.tesh) endforeach() endif() SimGrid-3.18/examples/java/.project0000644000175000017500000000370113217757322017527 0ustar mquinsonmquinson SimGrid Examples org.eclipse.jdt.core.javabuilder org.eclipse.jdt.core.javanature 1471870571873 26 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-CMakeFiles 1471870571874 22 org.eclipse.ui.ide.multiFilter 1.0-projectRelativePath-matches-false-false-**/*.cmake 1471870571876 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*~ 1471870571880 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*_compiled 1471870571882 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*.cmake 1471870571887 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-CMakeLists.txt 1471870571889 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-Makefile SimGrid-3.18/examples/java/cloud/0000755000175000017500000000000013217757326017171 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/cloud/masterworker/0000755000175000017500000000000013217757334021715 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/cloud/masterworker/Main.java0000644000175000017500000000265513217757326023455 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import java.io.File; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; class Main { public static final double TASK_COMP_SIZE = 10; public static final double TASK_COMM_SIZE = 10; public static final int NHOSTS = 6; public static final int NSTEPS = 50; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); Msg.liveMigrationInit(); String platfFile = "../../examples/platforms/small_platform.xml"; if (args.length >= 1) platfFile = args[0]; File f = new File(platfFile); if (!f.exists()) { Msg.error("File " + platfFile + " does not exist in " + System.getProperty("user.dir")); Msg.error("Usage : Main ../platforms/platform.xml"); } Msg.createEnvironment(platfFile); Host[] hosts = Host.all(); if (hosts.length < NHOSTS+1) { Msg.info("I need at least "+ (NHOSTS+1) +" hosts in the platform file, but " + args[0] + " contains only " + hosts.length + " hosts"); System.exit(42); } new Master(hosts[0],"Master",hosts).start(); /* Execute the simulation */ Msg.run(); } } SimGrid-3.18/examples/java/cloud/masterworker/Worker.java0000644000175000017500000000177513217757326024044 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; public class Worker extends Process { public Worker(Host host, String name) { super(host, name); } public void main(String[] args) throws MsgException { Msg.verb(this.getName() +" is listening on "+ getName()); while(true) { Task task = null; try { task = Task.receive(getName()); } catch (MsgException e) { Msg.info("Received failed. I'm done. See you!"); exit(); } Msg.verb("Received '" + task.getName() + "'. Processing it."); task.execute(); Msg.verb("Done executing task '" + task.getName() +"'"); } } } SimGrid-3.18/examples/java/cloud/masterworker/cloud-masterworker.tesh0000644000175000017500000003454213217757334026443 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} cloud/masterworker/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Master@Boivin) Launched 6 VMs > [ 0.000000] (1:Master@Boivin) Send some work to everyone > [ 2.186532] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [1971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 1 done. > [1971.662691] (1:Master@Boivin) Launched 6 VMs > [1971.662691] (1:Master@Boivin) Send some work to everyone > [1973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [2971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 2 done. > [2971.662691] (1:Master@Boivin) Launched 6 VMs > [2971.662691] (1:Master@Boivin) Send some work to everyone > [2973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [3971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 3 done. > [3971.662691] (1:Master@Boivin) Launched 6 VMs > [3971.662691] (1:Master@Boivin) Send some work to everyone > [3973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [4971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 4 done. > [4971.662691] (1:Master@Boivin) Launched 6 VMs > [4971.662691] (1:Master@Boivin) Send some work to everyone > [4973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [5971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 5 done. > [5971.662691] (1:Master@Boivin) Launched 6 VMs > [5971.662691] (1:Master@Boivin) Send some work to everyone > [5973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [6971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 6 done. > [6971.662691] (1:Master@Boivin) Launched 6 VMs > [6971.662691] (1:Master@Boivin) Send some work to everyone > [6973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [7971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 7 done. > [7971.662691] (1:Master@Boivin) Launched 6 VMs > [7971.662691] (1:Master@Boivin) Send some work to everyone > [7973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [8971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 8 done. > [8971.662691] (1:Master@Boivin) Launched 6 VMs > [8971.662691] (1:Master@Boivin) Send some work to everyone > [8973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [9971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 9 done. > [9971.662691] (1:Master@Boivin) Launched 6 VMs > [9971.662691] (1:Master@Boivin) Send some work to everyone > [9973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [10971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 10 done. > [10971.662691] (1:Master@Boivin) Launched 6 VMs > [10971.662691] (1:Master@Boivin) Send some work to everyone > [10973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [11971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 11 done. > [11971.662691] (1:Master@Boivin) Launched 6 VMs > [11971.662691] (1:Master@Boivin) Send some work to everyone > [11973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [12971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 12 done. > [12971.662691] (1:Master@Boivin) Launched 6 VMs > [12971.662691] (1:Master@Boivin) Send some work to everyone > [12973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [13971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 13 done. > [13971.662691] (1:Master@Boivin) Launched 6 VMs > [13971.662691] (1:Master@Boivin) Send some work to everyone > [13973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [14971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 14 done. > [14971.662691] (1:Master@Boivin) Launched 6 VMs > [14971.662691] (1:Master@Boivin) Send some work to everyone > [14973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [15971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 15 done. > [15971.662691] (1:Master@Boivin) Launched 6 VMs > [15971.662691] (1:Master@Boivin) Send some work to everyone > [15973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [16971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 16 done. > [16971.662691] (1:Master@Boivin) Launched 6 VMs > [16971.662691] (1:Master@Boivin) Send some work to everyone > [16973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [17971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 17 done. > [17971.662691] (1:Master@Boivin) Launched 6 VMs > [17971.662691] (1:Master@Boivin) Send some work to everyone > [17973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [18971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 18 done. > [18971.662691] (1:Master@Boivin) Launched 6 VMs > [18971.662691] (1:Master@Boivin) Send some work to everyone > [18973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [19971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 19 done. > [19971.662691] (1:Master@Boivin) Launched 6 VMs > [19971.662691] (1:Master@Boivin) Send some work to everyone > [19973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [20971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 20 done. > [20971.662691] (1:Master@Boivin) Launched 6 VMs > [20971.662691] (1:Master@Boivin) Send some work to everyone > [20973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [21971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 21 done. > [21971.662691] (1:Master@Boivin) Launched 6 VMs > [21971.662691] (1:Master@Boivin) Send some work to everyone > [21973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [22971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 22 done. > [22971.662691] (1:Master@Boivin) Launched 6 VMs > [22971.662691] (1:Master@Boivin) Send some work to everyone > [22973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [23971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 23 done. > [23971.662691] (1:Master@Boivin) Launched 6 VMs > [23971.662691] (1:Master@Boivin) Send some work to everyone > [23973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [24971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 24 done. > [24971.662691] (1:Master@Boivin) Launched 6 VMs > [24971.662691] (1:Master@Boivin) Send some work to everyone > [24973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [25971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 25 done. > [25971.662691] (1:Master@Boivin) Launched 6 VMs > [25971.662691] (1:Master@Boivin) Send some work to everyone > [25973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [26971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 26 done. > [26971.662691] (1:Master@Boivin) Launched 6 VMs > [26971.662691] (1:Master@Boivin) Send some work to everyone > [26973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [27971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 27 done. > [27971.662691] (1:Master@Boivin) Launched 6 VMs > [27971.662691] (1:Master@Boivin) Send some work to everyone > [27973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [28971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 28 done. > [28971.662691] (1:Master@Boivin) Launched 6 VMs > [28971.662691] (1:Master@Boivin) Send some work to everyone > [28973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [29971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 29 done. > [29971.662691] (1:Master@Boivin) Launched 6 VMs > [29971.662691] (1:Master@Boivin) Send some work to everyone > [29973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [30971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 30 done. > [30971.662691] (1:Master@Boivin) Launched 6 VMs > [30971.662691] (1:Master@Boivin) Send some work to everyone > [30973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [31971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 31 done. > [31971.662691] (1:Master@Boivin) Launched 6 VMs > [31971.662691] (1:Master@Boivin) Send some work to everyone > [31973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [32971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 32 done. > [32971.662691] (1:Master@Boivin) Launched 6 VMs > [32971.662691] (1:Master@Boivin) Send some work to everyone > [32973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [33971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 33 done. > [33971.662691] (1:Master@Boivin) Launched 6 VMs > [33971.662691] (1:Master@Boivin) Send some work to everyone > [33973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [34971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 34 done. > [34971.662691] (1:Master@Boivin) Launched 6 VMs > [34971.662691] (1:Master@Boivin) Send some work to everyone > [34973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [35971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 35 done. > [35971.662691] (1:Master@Boivin) Launched 6 VMs > [35971.662691] (1:Master@Boivin) Send some work to everyone > [35973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [36971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 36 done. > [36971.662691] (1:Master@Boivin) Launched 6 VMs > [36971.662691] (1:Master@Boivin) Send some work to everyone > [36973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [37971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 37 done. > [37971.662691] (1:Master@Boivin) Launched 6 VMs > [37971.662691] (1:Master@Boivin) Send some work to everyone > [37973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [38971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 38 done. > [38971.662691] (1:Master@Boivin) Launched 6 VMs > [38971.662691] (1:Master@Boivin) Send some work to everyone > [38973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [39971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 39 done. > [39971.662691] (1:Master@Boivin) Launched 6 VMs > [39971.662691] (1:Master@Boivin) Send some work to everyone > [39973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [40971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 40 done. > [40971.662691] (1:Master@Boivin) Launched 6 VMs > [40971.662691] (1:Master@Boivin) Send some work to everyone > [40973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [41971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 41 done. > [41971.662691] (1:Master@Boivin) Launched 6 VMs > [41971.662691] (1:Master@Boivin) Send some work to everyone > [41973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [42971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 42 done. > [42971.662691] (1:Master@Boivin) Launched 6 VMs > [42971.662691] (1:Master@Boivin) Send some work to everyone > [42973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [43971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 43 done. > [43971.662691] (1:Master@Boivin) Launched 6 VMs > [43971.662691] (1:Master@Boivin) Send some work to everyone > [43973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [44971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 44 done. > [44971.662691] (1:Master@Boivin) Launched 6 VMs > [44971.662691] (1:Master@Boivin) Send some work to everyone > [44973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [45971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 45 done. > [45971.662691] (1:Master@Boivin) Launched 6 VMs > [45971.662691] (1:Master@Boivin) Send some work to everyone > [45973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [46971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 46 done. > [46971.662691] (1:Master@Boivin) Launched 6 VMs > [46971.662691] (1:Master@Boivin) Send some work to everyone > [46973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [47971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 47 done. > [47971.662691] (1:Master@Boivin) Launched 6 VMs > [47971.662691] (1:Master@Boivin) Send some work to everyone > [47973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [48971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 48 done. > [48971.662691] (1:Master@Boivin) Launched 6 VMs > [48971.662691] (1:Master@Boivin) Send some work to everyone > [48973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [49971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 49 done. > [49971.662691] (1:Master@Boivin) Launched 6 VMs > [49971.662691] (1:Master@Boivin) Send some work to everyone > [49973.849223] (1:Master@Boivin) Suspend all VMs, wait a while, resume them, migrate them and shut them down. > [50971.662691] (1:Master@Boivin) XXXXXXXXXXXXXXX Step 50 done. > [50971.662691] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/cloud/masterworker/Master.java0000644000175000017500000000440013217757326024012 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; import org.simgrid.msg.VM; public class Master extends Process { private Host[] hosts; public Master(Host host, String name, Host[] hosts) { super(host,name,null); this.hosts = hosts; } public void main(String[] args) throws MsgException { int workersCount = Main.NHOSTS; for (int step = 1; step <= Main.NSTEPS ; step++) { // Create one VM per host and bind a process inside each one. for (int i = 0; i < workersCount; i++) { Msg.verb("create VM0-s"+step+"-"+i); VM vm = new VM(hosts[i+1],"VM0-s"+step+"-"+i); vm.start(); Worker worker= new Worker(vm,"WK:"+step+":"+ i); Msg.verb("Put Worker "+worker.getName()+ " on "+vm.getName()); worker.start(); } VM[] vms = VM.all(); Msg.info("Launched " + vms.length + " VMs"); Msg.info("Send some work to everyone"); workBatch(workersCount,"WK:"+step+":"); Msg.info("Suspend all VMs, wait a while, resume them, migrate them and shut them down."); for (VM vm : vms) { Msg.verb("Suspend "+vm.getName()); vm.suspend(); } Msg.verb("Wait a while, and resume all VMs."); waitFor(2); for (VM vm : vms) vm.resume(); Msg.verb("Sleep long enough for everyone to be done with previous batch of work"); waitFor(1000*step - Msg.getClock()); Msg.verb("Migrate everyone to "+hosts[3].getName()); for (VM vm : vms) { Msg.verb("Migrate "+vm.getName()+" to "+hosts[3].getName()); vm.migrate(hosts[3]); } Msg.verb("Let's kill everyone."); for (VM vm : vms) vm.destroy(); Msg.info("XXXXXXXXXXXXXXX Step "+step+" done."); } } public void workBatch(int workersCount, String nameRoot) throws MsgException { for (int i = 0; i < workersCount; i++) { Task task = new Task("Task "+nameRoot + i, Main.TASK_COMP_SIZE, Main.TASK_COMM_SIZE); Msg.verb("Sending to "+ nameRoot + i); task.send(nameRoot + i); } } } SimGrid-3.18/examples/java/cloud/migration/0000755000175000017500000000000013217757340021156 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/cloud/migration/Main.java0000644000175000017500000000200013217757325022700 0ustar mquinsonmquinson/* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { private static boolean endOfTest = false; private Main() { throw new IllegalAccessError("Utility class"); } public static void setEndOfTest(){ endOfTest=true; } public static boolean isEndOfTest(){ return endOfTest; } public static void main(String[] args) throws MsgException { Msg.init(args); Msg.liveMigrationInit(); if (args.length < 1) { Msg.info("Usage : Main platform_file.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); new cloud.migration.Test("PM0","Test").start(); Msg.run(); } } SimGrid-3.18/examples/java/cloud/migration/XVM.java0000644000175000017500000000377313217757326022511 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.VM; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; public class XVM extends VM { private int dpIntensity; private int ramsize; private int currentLoad; private Daemon daemon; public XVM(Host host, String name, int ramsize, int migNetBW, int dpIntensity){ super(host, name, ramsize, (int)(migNetBW*0.9), dpIntensity); this.currentLoad = 0; this.dpIntensity = dpIntensity ; this.ramsize= ramsize; this.daemon = new Daemon(this); } public void setLoad(int load){ if (load >0) { this.setBound(this.getSpeed()*load/100); daemon.resume(); } else{ daemon.suspend(); } currentLoad = load ; } @Override public void start() { super.start(); daemon.start(); this.setLoad(0); } public Daemon getDaemon(){ return this.daemon; } public int getLoad(){ Msg.info("Remaining comp:" + this.daemon.getRemaining()); return this.currentLoad; } @Override public void migrate(Host host) throws HostFailureException { Msg.info("Start migration of VM " + this.getName() + " to " + host.getName()); Msg.info(" currentLoad:" + this.currentLoad + "/ramSize:" + this.ramsize + "/dpIntensity:" + this.dpIntensity + "/remaining:" + String.format(java.util.Locale.US, "%.2E",this.daemon.getRemaining())); try{ super.migrate(host); } catch (Exception e){ Msg.info("Something wrong during the live migration of VM "+this.getName()); throw new HostFailureException(); } this.setLoad(this.currentLoad); //Fixed the fact that setBound is not propagated to the new node. Msg.info("End of migration of VM " + this.getName() + " to node " + host.getName()); } } SimGrid-3.18/examples/java/cloud/migration/cloud-migration.tesh0000644000175000017500000000544713217757334025155 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} cloud/migration/Main ${srcdir:=.}/../platforms/three_multicore_hosts.xml > [0.000000] [java/INFO] Using regular java threads. > [PM0:Test:(1) 0.000000] [java/INFO] This example evaluates the migration time of a VM in presence of collocated VMs on the source and the dest nodes > [PM0:Test:(1) 0.000000] [java/INFO] The migrated VM has a memory intensity rate of 70% of the network BW and a cpu load of 90% " (see cloudcom 2013 paper "Adding a Live Migration Model Into SimGrid" for further information) > [PM0:Test:(1) 0.000000] [java/INFO] Load of collocated VMs fluctuate between 0 and 90% in order to create a starvation issue and see whether it impacts or not the migration time > [PM0:Test:(1) 0.000000] [java/INFO] Round trip of VM1 (load 90%) > [PM0:Test:(1) 0.000000] [java/INFO] - Launch migration from PM0 to PM1 > [PM0:Test:(1) 0.000000] [java/INFO] Start migration of VM vm0 to PM1 > [PM0:Test:(1) 0.000000] [java/INFO] currentLoad:90/ramSize:2048/dpIntensity:70/remaining:8.10E+11 > [PM0:Test:(1) 45.731913] [java/INFO] End of migration of VM vm0 to node PM1 > [PM0:Test:(1) 45.731913] [java/INFO] - End of Migration from PM0 to PM1 (duration:45.73191265731957) > [PM0:Test:(1) 45.731913] [java/INFO] - Launch migration from PM1 to PM0 > [PM0:Test:(1) 45.731913] [java/INFO] Start migration of VM vm0 to PM0 > [PM0:Test:(1) 45.731913] [java/INFO] currentLoad:90/ramSize:2048/dpIntensity:70/remaining:5.01E+11 > [PM0:Test:(1) 97.502571] [java/INFO] End of migration of VM vm0 to node PM0 > [PM0:Test:(1) 97.502571] [java/INFO] - End of Migration from PM1 to PM0 (duration:51.770657959072125) > [PM0:Test:(1) 97.502571] [java/INFO] > [PM0:Test:(1) 97.502571] [java/INFO] > [PM0:Test:(1) 97.502571] [java/INFO] Round trip of VM1 (load 80%) > [PM0:Test:(1) 97.502571] [java/INFO] - Launch migration from PM0 to PM1 > [PM0:Test:(1) 97.502571] [java/INFO] Start migration of VM vm0 to PM1 > [PM0:Test:(1) 97.502571] [java/INFO] currentLoad:80/ramSize:2048/dpIntensity:70/remaining:1.24E+11 > [PM0:Test:(1) 140.711178] [java/INFO] End of migration of VM vm0 to node PM1 > [PM0:Test:(1) 140.711178] [java/INFO] - End of Migration from PM0 to PM1 (duration:43.20860711422699) > [PM0:Test:(1) 140.711178] [java/INFO] - Launch migration from PM1 to PM0 > [PM0:Test:(1) 140.711178] [java/INFO] Start migration of VM vm0 to PM0 > [PM0:Test:(1) 140.711178] [java/INFO] currentLoad:80/ramSize:2048/dpIntensity:70/remaining:6.54E+11 > [PM0:Test:(1) 183.918679] [java/INFO] End of migration of VM vm0 to node PM0 > [PM0:Test:(1) 183.918679] [java/INFO] - End of Migration from PM1 to PM0 (duration:43.207501264227034) > [PM0:Test:(1) 183.918679] [java/INFO] Forcefully destroy VMs > [183.918679] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/cloud/migration/Daemon.java0000644000175000017500000000221413217757325023226 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.*; import org.simgrid.msg.Process; public class Daemon extends Process { private Task currentTask; public Daemon(VM vm) { super((Host)vm,"Daemon"); currentTask = new Task(this.getHost().getName()+"-daemon-0", this.getHost().getSpeed()*100, 0); } public void main(String[] args) throws MsgException { int i = 1; while(!Main.isEndOfTest()) { // TODO the binding is not yet available try { currentTask.execute(); } catch (HostFailureException e) { e.printStackTrace(); } catch (TaskCancelledException e) { Msg.info("task cancelled"); suspend(); // Suspend the process } currentTask = new Task(this.getHost().getName()+"-daemon-"+(i++), this.getHost().getSpeed()*100, 0); } } public double getRemaining(){ return this.currentTask.getFlopsAmount(); } } SimGrid-3.18/examples/java/cloud/migration/README0000644000175000017500000000061713217757340022042 0ustar mquinsonmquinsonThere are two tests: - The first one aims at validating the migration in presence of consolidated VMs - The second one aims at validating the robustness of the migration process (the SRC node or the DST node are turned off during the migration process) To switch between the first and the second tests, you should instrument Main.java (sorry for that :( ) Adsein - Thu Oct 9 18:25:39 CEST 2014 SimGrid-3.18/examples/java/cloud/migration/Test.java0000644000175000017500000000643113217757325022747 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import java.util.ArrayList; import java.util.List; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.VM; public class Test extends Process{ Test(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void doMigration(VM vm, Host src, Host dst) throws HostFailureException{ Msg.info(" - Launch migration from "+ src.getName() +" to " + dst.getName()); double startTime = Msg.getClock(); vm.migrate(dst); double endTime = Msg.getClock(); Msg.info(" - End of Migration from "+ src.getName() +" to " + dst.getName()+ " (duration:" + (endTime-startTime)+")"); } public void main(String[] strings) throws MsgException { Host host0 = Host.getByName("PM0"); Host host1 = Host.getByName("PM1"); List vms = new ArrayList<>(); /* Create VM1 */ int dpRate = 70; int load1 = 90; int load2 = 80; Msg.info("This example evaluates the migration time of a VM in presence of collocated VMs on the source and " + "the dest nodes"); Msg.info("The migrated VM has a memory intensity rate of 70% of the network BW and a cpu load of 90% \" " +"(see cloudcom 2013 paper \"Adding a Live Migration Model Into SimGrid\" for further information) "); Msg.info("Load of collocated VMs fluctuate between 0 and 90% in order to create a starvation issue and see " + "whether it impacts or not the migration time"); XVM vm1 = new XVM(host0, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(vm1); vm1.start(); /* Collocated VMs */ int[] vmSrcLoad = { 80, 0, 90, 40, 30, 90, }; XVM tmp; for (int i=1 ; i<= vmSrcLoad.length ; i++){ tmp = new XVM(host0, "vm"+i, 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(tmp); tmp.start(); tmp.setLoad(vmSrcLoad[i-1]); } int[] vmDstLoad = { 0, 40, 90, 100, 0, 80, }; for (int i=1 ; i <= vmDstLoad.length ; i++){ tmp = new XVM(host1, "vm"+(i+vmSrcLoad.length), 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(tmp); tmp.start(); tmp.setLoad(vmDstLoad[i-1]); } Msg.info("Round trip of VM1 (load "+load1+"%)"); vm1.setLoad(load1); doMigration(vm1, host0, host1); doMigration(vm1, host1, host0); Msg.info(""); Msg.info(""); Msg.info("Round trip of VM1 (load "+load2+"%)"); vm1.setLoad(load2); doMigration(vm1, host0, host1); doMigration(vm1, host1, host0); Main.setEndOfTest(); Msg.info("Forcefully destroy VMs"); for (VM vm: vms) vm.destroy(); } } SimGrid-3.18/examples/java/cloud/migration/TestHostOnOff.java0000644000175000017500000001072513217757326024537 0ustar mquinsonmquinson/* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.VM; // This test aims at validating that the migration process is robust in face of host turning off either on the SRC // node or on the DST node. public class TestHostOnOff extends Process{ protected Host host0 = null; protected Host host1 = null; protected Host host2 = null; TestHostOnOff(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] strings) throws MsgException { /* get hosts 1 and 2*/ try { host0 = Host.getByName("PM0"); host1 = Host.getByName("PM1"); host2 = Host.getByName("PM2"); }catch (HostNotFoundException e) { Msg.error("You are trying to use a non existing node:" + e.getMessage()); } // Robustness on the SRC node for (int i =100 ; i < 55000 ; i+=100) testVMMigrate(host1, i); // Robustness on the DST node for (int i =0 ; i < 55000 ; i++) testVMMigrate(host2, i); /* End of Tests */ Msg.info("Nor more tests, Bye Bye !"); Main.setEndOfTest(); } public void testVMMigrate (Host hostToKill, long killAt) throws MsgException { Msg.info("**** **** **** ***** ***** Test Migrate with host shutdown ***** ***** **** **** ****"); Msg.info("Turn on one host, assign a VM on this host, launch a process inside the VM, migrate the VM and " + "turn off either the SRC or DST"); host1.off(); host2.off(); host1.on(); host2.on(); // Create VM0 int dpRate = 70; XVM vm0 = null; vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vm0.start(); vm0.setLoad(90); String[] args = new String[3]; args[0] = "vm0"; args[1] = "PM1"; args[2] = "PM2"; new Process(host1, "Migrate-" + killAt, args) { public void main(String[] args) { Host destHost = null; Host sourceHost = null; try { sourceHost = Host.getByName(args[1]); destHost = Host.getByName(args[2]); } catch (HostNotFoundException e) { Msg.error("You are trying to migrate from/to a non existing node: " + e.getMessage()); e.printStackTrace(); } if (destHost != null && sourceHost.isOn() && destHost.isOn()) { try { Msg.info("Migrate vm "+args[0]+" to node "+destHost.getName()); VM.getVMByName(args[0]).migrate(destHost); } catch (HostFailureException e) { Msg.error("Something occurs during the migration that cannot validate the operation: " + e.getMessage()); e.printStackTrace(); } } } }.start(); // Wait killAt ms before killing thehost Process.sleep(killAt); Msg.info("The migration process should be stopped and we should catch an exception"); hostToKill.off(); Process.sleep(50000); Msg.info("Destroy VMs"); vm0.shutdown(); Process.sleep(20000); } public void testVMShutdownDestroy () throws HostFailureException { Msg.info("**** **** **** ***** ***** Test shutdown a VM ***** ***** **** **** ****"); Msg.info("Turn on host1, assign a VM on host1, launch a process inside the VM, and turn off the vm, " + "and check whether you can reallocate the same VM"); // Create VM0 int dpRate = 70; XVM vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); Msg.info("Start VM0"); vm0.start(); vm0.setLoad(90); Process.sleep(5000); Msg.info("Shutdown VM0"); vm0.shutdown(); Process.sleep(5000); Msg.info("Restart VM0"); vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vm0.start(); vm0.setLoad(90); Msg.info("You suceed to recreate and restart a VM without generating any exception ! Great the Test is ok"); Process.sleep(5000); vm0.shutdown(); } } SimGrid-3.18/examples/java/task/0000755000175000017500000000000013217757326017025 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/task/priority/0000755000175000017500000000000013217757343020705 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/task/priority/Main.java0000644000175000017500000000173113217757326022437 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package task.priority; import org.simgrid.msg.Msg; /* Demonstrates the use of Task.setPriority to change the computation priority of a task */ public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Priority platform_file deployment_file"); Msg.info("example : Priority ../platforms/small_platform.xml priority.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/task/priority/task-priority.tesh0000644000175000017500000000122613217757334024414 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} task/priority/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/task/priority/priority.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:task.priority.Test@Fafard) Hello! Running a task of size 7.6296E7 with priority 1.0 > [ 0.000000] (2:task.priority.Test@Fafard) Hello! Running a task of size 7.6296E7 with priority 2.0 > [ 1.500000] (2:task.priority.Test@Fafard) Goodbye now! > [ 2.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 2.000000] (1:task.priority.Test@Fafard) Goodbye now! SimGrid-3.18/examples/java/task/priority/priority.xml0000644000175000017500000000060713217757343023313 0ustar mquinsonmquinson SimGrid-3.18/examples/java/task/priority/Test.java0000644000175000017500000000175413217757326022477 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package task.priority; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Test extends Process { public Test(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { double computationAmount = Double.parseDouble(args[0]); double priority = Double.parseDouble(args[1]); Msg.info("Hello! Running a task of size " + computationAmount + " with priority " + priority); Task task = new Task("Task", computationAmount, 0); task.setPriority(priority); task.execute(); Msg.info("Goodbye now!"); } } SimGrid-3.18/examples/java/dht/0000755000175000017500000000000013217757326016642 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/dht/chord/0000755000175000017500000000000013217757343017740 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/dht/chord/ChordTask.java0000644000175000017500000000141613217757326022470 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import dht.chord.Common; import org.simgrid.msg.Task; public class ChordTask extends Task { private String issuerHostName; private String answerTo; public ChordTask() { this(null,null); } public ChordTask(String issuerHostName, String answerTo) { super(null, Common.COMP_SIZE, Common.COMM_SIZE); this.issuerHostName = issuerHostName; this.answerTo = answerTo; } public String getIssuerHostName(){ return this.issuerHostName; } public String getAnswerTo(){ return this.answerTo; } } SimGrid-3.18/examples/java/dht/chord/GetPredecessorAnswerTask.java0000644000175000017500000000113513217757326025525 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class GetPredecessorAnswerTask extends ChordTask { private int answerId; public GetPredecessorAnswerTask(String issuerHostname, String answerTo, int answerId) { super(issuerHostname,answerTo); this.answerId = answerId; } public int getAnswerId(){ return this.answerId; } } SimGrid-3.18/examples/java/dht/chord/Main.java0000644000175000017500000000153413217757326021473 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Chord platform_file deployment_file"); Msg.info("example : Chord ../platforms/platform.xml chord.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/dht/chord/FindSuccessorAnswerTask.java0000644000175000017500000000113313217757326025357 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class FindSuccessorAnswerTask extends ChordTask { private int answerId; public FindSuccessorAnswerTask(String issuerHostname, String answerTo, int answerId) { super(issuerHostname,answerTo); this.answerId = answerId; } public int getAnswerId(){ return this.answerId; } } SimGrid-3.18/examples/java/dht/chord/NotifyTask.java0000644000175000017500000000111013217757326022670 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class NotifyTask extends ChordTask { private int requestId; public NotifyTask(String issuerHostname, String answerTo, int requestId) { super(issuerHostname, answerTo); this.requestId = requestId; } public int getRequestId(){ return this.requestId; } } SimGrid-3.18/examples/java/dht/chord/dht-chord.tesh0000644000175000017500000000241113217757334022477 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} dht/chord/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dht/chord/chord.xml > [0.000000] [java/INFO] Using regular java threads. > [1046.732943] [java/INFO] MSG_main finished; Cleaning up the simulation... > [node-1.acme.org:dht.chord.Node:(2) 0.000000] [java/INFO] Joining the ring with id 366680 knowing node 42 > [node-2.acme.org:dht.chord.Node:(3) 0.000000] [java/INFO] Joining the ring with id 533744 knowing node 366680 > [node-3.acme.org:dht.chord.Node:(4) 0.000000] [java/INFO] Joining the ring with id 1319738 knowing node 42 > [node-4.acme.org:dht.chord.Node:(5) 0.000000] [java/INFO] Joining the ring with id 16509405 knowing node 366680 > [node-5.acme.org:dht.chord.Node:(6) 0.000000] [java/INFO] Joining the ring with id 10874876 knowing node 533744 > [node-6.acme.org:dht.chord.Node:(7) 0.000000] [java/INFO] Joining the ring with id 16728096 knowing node 1319738 > [node-7.acme.org:dht.chord.Node:(8) 0.000000] [java/INFO] Joining the ring with id 10004760 knowing node 16509405 > [node-8.acme.org:dht.chord.Node:(9) 0.000000] [java/INFO] Joining the ring with id 6518808 knowing node 42 > [node-9.acme.org:dht.chord.Node:(10) 0.000000] [java/INFO] Joining the ring with id 2015253 knowing node 1319738 SimGrid-3.18/examples/java/dht/chord/FindSuccessorTask.java0000644000175000017500000000112713217757326024202 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class FindSuccessorTask extends ChordTask { private int requestId; public FindSuccessorTask(String issuerHostname, String answerTo, int requestId) { super(issuerHostname, answerTo); this.requestId = requestId; } public int getRequestId(){ return this.requestId; } } SimGrid-3.18/examples/java/dht/chord/Node.java0000644000175000017500000003167313217757326021503 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.TimeoutException; public class Node extends Process { protected int id; protected String mailbox; protected int predId; protected String predMailbox; protected int nextFingerToFix; protected Comm commReceive; ///Last time I changed a finger or my predecessor protected double lastChangeDate; private int[] fingers; public Node(Host host, String name, String[] args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { if (args.length != 2 && args.length != 4) { Msg.info("You need to provide 2 or 4 arguments."); return; } double initTime = Msg.getClock(); int i; boolean joinSuccess; double deadline; double nextStabilizeDate = initTime + Common.PERIODIC_STABILIZE_DELAY; double nextFixFingersDate = initTime + Common.PERIODIC_FIX_FINGERS_DELAY; double nextCheckPredecessorDate = initTime + Common.PERIODIC_CHECK_PREDECESSOR_DELAY; double nextLookupDate = initTime + Common.PERIODIC_LOOKUP_DELAY; mailbox = args[0]; id = Integer.parseInt(args[0]); fingers = new int[Common.NB_BITS]; for (i = 0; i < Common.NB_BITS; i++) { fingers[i] = -1; setFinger(i,this.id); } //First node if (args.length == 2) { deadline = Integer.parseInt(args[1]); create(); joinSuccess = true; } else { int knownId = Integer.parseInt(args[1]); deadline = Integer.parseInt(args[3]); Msg.debug("Hey! Let's join the system with the id " + id + "."); joinSuccess = join(knownId); } if (!joinSuccess) { Msg.info("I couldn't join the ring"); return; } double currentClock = Msg.getClock(); while (currentClock < (initTime + deadline) && currentClock < Common.MAX_SIMULATION_TIME) { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } try { if (!commReceive.test()) { if (currentClock >= nextStabilizeDate) { stabilize(); nextStabilizeDate = Msg.getClock() + Common.PERIODIC_STABILIZE_DELAY; } else if (currentClock >= nextFixFingersDate) { fixFingers(); nextFixFingersDate = Msg.getClock() + Common.PERIODIC_FIX_FINGERS_DELAY; } else if (currentClock >= nextCheckPredecessorDate) { this.checkPredecessor(); nextCheckPredecessorDate = Msg.getClock() + Common.PERIODIC_CHECK_PREDECESSOR_DELAY; } else if (currentClock >= nextLookupDate) { this.randomLookup(); nextLookupDate = Msg.getClock() + Common.PERIODIC_LOOKUP_DELAY; } else { waitFor(5); } currentClock = Msg.getClock(); } else { handleTask(commReceive.getTask()); currentClock = Msg.getClock(); commReceive = null; } } catch (Exception e) { currentClock = Msg.getClock(); commReceive = null; } } leave(); if (commReceive != null) { commReceive = null; } } private void handleTask(Task task) { if (task instanceof FindSuccessorTask) { FindSuccessorTask fTask = (FindSuccessorTask)task; Msg.debug("Receiving a 'Find Successor' request from " + fTask.getIssuerHostName() + " for id " + fTask.getRequestId()); // is my successor the successor? if (isInInterval(fTask.getRequestId(), this.id + 1, fingers[0])) { Msg.debug("Send the request to " + fTask.getAnswerTo() + " with answer " + fingers[0]); FindSuccessorAnswerTask answer = new FindSuccessorAnswerTask(getHost().getName(), mailbox, fingers[0]); answer.dsend(fTask.getAnswerTo()); } else { // otherwise, forward the request to the closest preceding finger in my table int closest = closestPrecedingNode(fTask.getRequestId()); Msg.debug("Forward the request to " + closest); fTask.dsend(Integer.toString(closest)); } } else if (task instanceof GetPredecessorTask) { GetPredecessorTask gTask = (GetPredecessorTask)(task); Msg.debug("Receiving a 'Get Predecessor' request from " + gTask.getIssuerHostName()); GetPredecessorAnswerTask answer = new GetPredecessorAnswerTask(getHost().getName(), mailbox, predId); answer.dsend(gTask.getAnswerTo()); } else if (task instanceof NotifyTask) { NotifyTask nTask = (NotifyTask)task; notify(nTask.getRequestId()); } else { Msg.debug("Ignoring unexpected task of type:" + task); } } private void leave() { Msg.debug("Well Guys! I Think it's time for me to quit ;)"); // TODO: Notify my successor and predecessor. } /** @brief Initializes the current node as the first one of the system */ private void create() { Msg.debug("Create a new Chord ring..."); setPredecessor(-1); } // Makes the current node join the ring, knowing the id of a node already in the ring private boolean join(int knownId) { Msg.info("Joining the ring with id " + this.id + " knowing node " + knownId); setPredecessor(-1); int successorId = remoteFindSuccessor(knownId, this.id); if (successorId == -1) { Msg.info("Cannot join the ring."); } else { setFinger(0, successorId); } return successorId != -1; } private void setPredecessor(int predecessorId) { if (predecessorId != predId) { predId = predecessorId; if (predecessorId != -1) { predMailbox = Integer.toString(predId); } lastChangeDate = Msg.getClock(); } } /** * @brief Asks another node its predecessor. * @param askTo the node to ask to * @return the id of its predecessor node, or -1 if the request failed(or if the node does not know its predecessor) */ private int remoteGetPredecessor(int askTo) { int predecessorId = -1; boolean stop = false; Msg.debug("Sending a 'Get Predecessor' request to " + askTo); String mailboxTo = Integer.toString(askTo); GetPredecessorTask sendTask = new GetPredecessorTask(getHost().getName(), this.mailbox); try { sendTask.send(mailboxTo, Common.TIMEOUT); try { do { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } commReceive.waitCompletion(Common.TIMEOUT); Task taskReceived = commReceive.getTask(); if (taskReceived instanceof GetPredecessorAnswerTask) { predecessorId = ((GetPredecessorAnswerTask) taskReceived).getAnswerId(); stop = true; } else { handleTask(taskReceived); } commReceive = null; } while (!stop); } catch (MsgException e) { commReceive = null; } } catch (MsgException e) { Msg.debug("Failed to send the Get Predecessor request"); } return predecessorId; } /** * @brief Makes the current node find the successor node of an id. * @param node the current node * @param id the id to find * @return the id of the successor node, or -1 if the request failed */ private int findSuccessor(int id) { if (isInInterval(id, this.id + 1, fingers[0])) { return fingers[0]; } int closest = this.closestPrecedingNode(id); return remoteFindSuccessor(closest, id); } // Asks another node the successor node of an id. private int remoteFindSuccessor(int askTo, int id) { int successor = -1; boolean stop = false; String askToMailbox = Integer.toString(askTo); Task sendTask = new FindSuccessorTask(getHost().getName(), this.mailbox, id); Msg.debug("Sending a 'Find Successor' request to " + askToMailbox + " for id " + id); try { sendTask.send(askToMailbox, Common.TIMEOUT); do { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } try { commReceive.waitCompletion(Common.TIMEOUT); Task task = commReceive.getTask(); if (task instanceof FindSuccessorAnswerTask) { //TODO: Check if this this our answer. FindSuccessorAnswerTask fTask = (FindSuccessorAnswerTask) task; stop = true; successor = fTask.getAnswerId(); } else { handleTask(task); } commReceive = null; } catch (TimeoutException e) { stop = true; commReceive = null; } } while (!stop); } catch (TimeoutException e) { Msg.debug("Failed to send the 'Find Successor' request"); } catch (MsgException e) { Msg.debug("Failed to receive Find Successor"); } return successor; } // This function is called periodically. It checks the immediate successor of the current node. private void stabilize() { Msg.debug("Stabilizing node"); int candidateId; int successorId = fingers[0]; if (successorId != this.id){ candidateId = remoteGetPredecessor(successorId); } else { candidateId = predId; } //This node is a candidate to become my new successor if (candidateId != -1 && isInInterval(candidateId, this.id + 1, successorId - 1)) { setFinger(0, candidateId); } if (successorId != this.id) { remoteNotify(successorId, this.id); } } /** * @brief Notifies the current node that its predecessor may have changed. * @param candidate_id the possible new predecessor */ private void notify(int predecessorCandidateId) { if (predId == -1 || isInInterval(predecessorCandidateId, predId + 1, this.id - 1 )) { setPredecessor(predecessorCandidateId); } } /** * @brief Notifies a remote node that its predecessor may have changed. * @param notify_id id of the node to notify * @param candidate_id the possible new predecessor */ private void remoteNotify(int notifyId, int predecessorCandidateId) { Msg.debug("Sending a 'Notify' request to " + notifyId); Task sentTask = new NotifyTask(getHost().getName(), this.mailbox, predecessorCandidateId); sentTask.dsend(Integer.toString(notifyId)); } // This function is called periodically. // It refreshes the finger table of the current node. private void fixFingers() { Msg.debug("Fixing fingers"); int i = this.nextFingerToFix; int successorId = this.findSuccessor(this.id + (int)Math.pow(2,i)); //FIXME: SLOW if (successorId != -1) { if (successorId != fingers[i]) { setFinger(i, successorId); } nextFingerToFix = (i + 1) % Common.NB_BITS; } } // This function is called periodically. // It checks whether the predecessor has failed private void checkPredecessor() { //TODO } // Performs a find successor request to a random id. private void randomLookup() { int dest = 1337; findSuccessor(dest); } /** * @brief Returns the closest preceding finger of an id with respect to the finger table of the current node. * @param id the id to find * @return the closest preceding finger of that id */ private int closestPrecedingNode(int id) { for (int i = Common.NB_BITS - 1; i >= 0; i--) { if (isInInterval(fingers[i], this.id + 1, id - 1)) { return fingers[i]; } } return this.id; } /** * @brief Returns whether an id belongs to the interval [start, end]. * * The parameters are noramlized to make sure they are between 0 and nb_keys - 1). * 1 belongs to [62, 3] * 1 does not belong to [3, 62] * 63 belongs to [62, 3] * 63 does not belong to [3, 62] * 24 belongs to [21, 29] * 24 does not belong to [29, 21] * * @param id id to check * @param start lower bound * @param end upper bound * @return a non-zero value if id in in [start, end] */ private static boolean isInInterval(int id, int start, int end) { int normId = normalize(id); int normStart = normalize(start); int normEnd = normalize(end); // make sure end >= start and id >= start if (normEnd < normStart) { normEnd += Common.NB_KEYS; } if (normId < normStart) { normId += Common.NB_KEYS; } return (normId <= normEnd); } /** * @brief Turns an id into an equivalent id in [0, nb_keys). * @param id an id * @return the corresponding normalized id */ private static int normalize(int id) { return id & (Common.NB_KEYS - 1); } /** * @brief Sets a finger of the current node. * @param finger_index index of the finger to set (0 to nb_bits - 1) * @param id the id to set for this finger */ private void setFinger(int fingerIndex, int id) { if (id != fingers[fingerIndex]) { fingers[fingerIndex] = id; lastChangeDate = Msg.getClock(); } } } SimGrid-3.18/examples/java/dht/chord/GetPredecessorTask.java0000644000175000017500000000072213217757326024346 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class GetPredecessorTask extends ChordTask { public GetPredecessorTask(String issuerHostName, String answerTo) { super(issuerHostName, answerTo); } } SimGrid-3.18/examples/java/dht/chord/chord.xml0000644000175000017500000000375213217757343021570 0ustar mquinsonmquinson SimGrid-3.18/examples/java/dht/chord/Common.java0000644000175000017500000000161013217757326022032 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class Common { public static final int COMM_SIZE = 10; public static final int COMP_SIZE = 0; public static final int NB_BITS = 24; public static final int NB_KEYS = 16777216; public static final int TIMEOUT = 50; public static final int MAX_SIMULATION_TIME = 1000; public static final int PERIODIC_STABILIZE_DELAY = 20; public static final int PERIODIC_FIX_FINGERS_DELAY = 120; public static final int PERIODIC_CHECK_PREDECESSOR_DELAY = 120; public static final int PERIODIC_LOOKUP_DELAY = 10; private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.18/examples/java/dht/kademlia/0000755000175000017500000000000013217757343020410 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/dht/kademlia/RoutingTable.java0000644000175000017500000000701513217757326023656 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.Collections; import java.util.Vector; import org.simgrid.msg.Msg; public class RoutingTable { /* Bucket list */ private Vector buckets; /* Id of the routing table owner */ private int id; public RoutingTable(int id) { this.id = id; buckets = new Vector<>(); for (int i = 0; i < Common.IDENTIFIER_SIZE + 1; i++) { buckets.add(new Bucket(i)); } } /** * @brief Returns an identifier which is in a specific bucket of a routing table * @param id id of the routing table owner * @param prefix id of the bucket where we want that identifier to be */ public int getIdInPrefix(int id, int prefix) { if (prefix == 0) { return 0; } int identifier = 1; identifier = identifier << (prefix - 1); identifier = identifier ^ id; return identifier; } /* Returns the corresponding node prefix for a given id */ public int getNodePrefix(int id) { for (int j = 0; j < 32; j++) { if ((id >> (32 - 1 - j) & 0x1) != 0) { return 32 - j; } } return 0; } /* Finds the corresponding bucket in a routing table for a given identifier */ public Bucket findBucket(int id) { int xorNumber = id ^ this.id; int prefix = this.getNodePrefix(xorNumber); return buckets.get(prefix); } /* Updates the routing table with a new value. */ public void update(int id) { Bucket bucket = this.findBucket(id); if (bucket.contains(id)) { Msg.debug("Updating " + Integer.toString(id) + " in my routing table"); //If the element is already in the bucket, we update it. bucket.pushToFront(id); } else { Msg.debug("Adding " + id + " to my routing table"); bucket.add(id); if (bucket.size() > Common.BUCKET_SIZE) { // TODO Msg.debug("Should ping the least seen guy and remove him if he is offline."); } } } /* Returns the closest notes we know to a given id */ public Answer findClosest(int destinationId) { Answer answer = new Answer(destinationId); Bucket bucket = this.findBucket(destinationId); bucket.addToAnswer(answer,destinationId); for (int i = 1; answer.size() < Common.BUCKET_SIZE && ((bucket.getId() - i) >= 0 || (bucket.getId() + i) <= Common.IDENTIFIER_SIZE); i++) { //Check the previous buckets if (bucket.getId() - i >= 0) { Bucket bucketP = this.buckets.get(bucket.getId() - i); bucketP.addToAnswer(answer,destinationId); } //Check the next buckets if (bucket.getId() + i <= Common.IDENTIFIER_SIZE) { Bucket bucketN = this.buckets.get(bucket.getId() + i); bucketN.addToAnswer(answer, destinationId); } } //We sort the list Collections.sort(answer.getNodes()); //We trim the list while (answer.size() > Common.BUCKET_SIZE) { answer.remove(answer.size() - 1); //TODO: Not the best thing. } return answer; } @Override public String toString() { StringBuilder string = new StringBuilder("RoutingTable [ id=" + id + " "); for (int i = 0; i < buckets.size(); i++) { if (buckets.get(i).size() > 0) { string.append(buckets.get(i) + " "); } } return string.toString(); } } SimGrid-3.18/examples/java/dht/kademlia/FindNodeTask.java0000644000175000017500000000142313217757326023565 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; /** * @brief Find node tasks sent by a node to another "Find Node" task sent by a node to another. Ask him for its closest * nodes from a destination. */ public class FindNodeTask extends KademliaTask { /* Id of the node we are trying to find: the destination */ private int destination; public FindNodeTask(int senderId, int destination) { super(senderId); this.destination = destination; } public int getDestination() { return destination; } } SimGrid-3.18/examples/java/dht/kademlia/KademliaTask.java0000644000175000017500000000116213217757326023606 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import dht.kademlia.Common; import org.simgrid.msg.Task; public class KademliaTask extends Task { protected int senderId; public KademliaTask(int senderId) { super("kademliatask",Common.COMP_SIZE,Common.COMM_SIZE); this.senderId = senderId; } public int getSenderId() { return senderId; } } SimGrid-3.18/examples/java/dht/kademlia/Main.java0000644000175000017500000000147213217757326022144 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Kademlia platform_file deployment_file"); Msg.info("example : Kademlia ../platforms/platform.xml kademlia.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/dht/kademlia/PingTask.java0000644000175000017500000000064013217757326022774 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class PingTask extends KademliaTask { public PingTask(int senderId) { super(senderId); } } SimGrid-3.18/examples/java/dht/kademlia/Answer.java0000644000175000017500000000334413217757326022517 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.ArrayList; import java.util.Collections; /* Answer to a "FIND_NODE" query. Contains the nodes closest to an id given */ public class Answer { private int destinationId; /* Closest nodes in the answer. */ private ArrayList nodes; public Answer(int destinationId) { this.destinationId = destinationId; nodes = new ArrayList<>(); } protected int getDestinationId() { return destinationId; } protected ArrayList getNodes() { return nodes; } protected int size() { return nodes.size(); } public void remove(int index) { nodes.remove(index); } public void add(Contact contact) { nodes.add(contact); } /* Merge the contents of this answer with another answer */ public int merge(Answer answer) { int nbAdded = 0; for (Contact c: answer.getNodes()) { if (!nodes.contains(c)) { nbAdded++; nodes.add(c); } } Collections.sort(nodes); //Trim the list while (answer.size() > Common.BUCKET_SIZE) { answer.remove(answer.size() - 1); } return nbAdded; } /* Returns if the destination has been found */ public boolean destinationFound() { if (nodes.isEmpty()) { return false; } Contact tail = nodes.get(0); return tail.getDistance() == 0; } @Override public String toString() { return "Answer [destinationId=" + destinationId + ", nodes=" + nodes + "]"; } } SimGrid-3.18/examples/java/dht/kademlia/FindNodeAnswerTask.java0000644000175000017500000000134413217757326024747 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import dht.kademlia.Answer; public class FindNodeAnswerTask extends KademliaTask { protected int destinationId; protected Answer answer; public FindNodeAnswerTask(int senderId, int destinationId, Answer answer) { super(senderId); this.destinationId = destinationId; this.answer = answer; } public int getDestinationId() { return destinationId; } public Answer getAnswer() { return answer; } } SimGrid-3.18/examples/java/dht/kademlia/PingAnswerTask.java0000644000175000017500000000065413217757326024161 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class PingAnswerTask extends KademliaTask { public PingAnswerTask(int senderId) { super(senderId); } } SimGrid-3.18/examples/java/dht/kademlia/kademlia.xml0000644000175000017500000000173013217757343022702 0ustar mquinsonmquinson SimGrid-3.18/examples/java/dht/kademlia/Bucket.java0000644000175000017500000000247113217757326022475 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.ArrayList; public class Bucket { private ArrayList nodes; private int id; public Bucket(int id) { this.nodes = new ArrayList<>(); this.id = id; } public int getId() { return this.id; } public int size() { return nodes.size(); } public boolean contains(int id) { return nodes.contains(id); } /* Add a node to the front of the bucket */ public void add(int id) { nodes.add(0,id); } /* Push a node to the front of a bucket */ public void pushToFront(int id) { int i = nodes.indexOf(id); nodes.remove(i); nodes.add(0, id); } public int getNode(int id) { return nodes.get(id); } /* Add the content of the bucket into a answer object. */ public void addToAnswer(Answer answer, int destination) { for (int nodeId : this.nodes) { answer.getNodes().add(new Contact(nodeId,nodeId ^ destination)); } } @Override public String toString() { return "Bucket [id= " + id + " nodes=" + nodes + "]"; } } SimGrid-3.18/examples/java/dht/kademlia/Node.java0000644000175000017500000002447513217757326022155 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Node extends Process { protected int id; protected RoutingTable table; protected int deadline; protected int findNodeSuccedded = 0; protected int findNodeFailed = 0; protected Comm comm; public Node(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { //Check the number of arguments. if (args.length != 2 && args.length != 3) { Msg.info("Wrong argument count."); return; } this.id = Integer.parseInt(args[0]); this.table = new RoutingTable(this.id); if (args.length == 3) { this.deadline = Integer.parseInt(args[2]); Msg.info("Hi, I'm going to join the network with the id " + id + "!"); if (joinNetwork(Integer.parseInt(args[1]))) { this.mainLoop(); } else { Msg.info("I couldn't join the network :("); } } else { this.deadline = Integer.parseInt(args[1]); Msg.info("Hi, I'm going to create the network with the id " + id + "!"); table.update(this.id); this.mainLoop(); } Msg.debug("I'm leaving the network"); Msg.debug("Here is my routing table:" + table); } public void mainLoop() { double nextLookupTime = Msg.getClock() + Common.RANDOM_LOOKUP_INTERVAL; while (Msg.getClock() < this.deadline) { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (!comm.test()) { if (Msg.getClock() >= nextLookupTime) { randomLookup(); nextLookupTime += Common.RANDOM_LOOKUP_INTERVAL; } else { waitFor(1); } } else { Task task = comm.getTask(); handleTask(task); comm = null; } } catch (Exception e) { Msg.debug("Caught exception: " + e); } } Msg.info(findNodeSuccedded + "/" + (findNodeSuccedded + findNodeFailed) + " FIND_NODE have succedded."); } /** * @brief Try to make the node join the network * @param idKnown Id of someone we know in the system */ public boolean joinNetwork(int idKnown) { boolean answerGot = false; double timeBegin = Msg.getClock(); Msg.debug("Joining the network knowing " + idKnown); //Add ourselves and the node we know to our routing table table.update(this.id); table.update(idKnown); //Send a "FIND_NODE" to the node we know. sendFindNode(idKnown,this.id); //Wait for the answer. int trials = 0; do { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (comm != null) { if (!comm.test()) { waitFor(1); } else { Task task = comm.getTask(); if (task instanceof FindNodeAnswerTask) { //Retrieve the node list and ping them FindNodeAnswerTask answerTask = (FindNodeAnswerTask)task; Answer answer = answerTask.getAnswer(); answerGot = true; if (answer.getDestinationId() == this.id) { //Ping everyone in the list for (Contact c : answer.getNodes()) { table.update(c.getId()); } } } else { handleTask(task); } comm = null; } } } catch (Exception ex) { trials++; Msg.info("FIND_NODE failed"); } } while (!answerGot && trials < Common.MAX_JOIN_TRIALS); /* Second step: Send a FIND_NODE in a node in each bucket */ int bucketId = table.findBucket(idKnown).getId(); for (int i = 0; ((bucketId - i) > 0 || (bucketId + i) <= Common.IDENTIFIER_SIZE) && i < Common.JOIN_BUCKETS_QUERIES; i++) { if (bucketId - i > 0) { int idInBucket = table.getIdInPrefix(this.id,bucketId - i); this.findNode(idInBucket,false); } if (bucketId + i <= Common.IDENTIFIER_SIZE) { int idInBucket = table.getIdInPrefix(this.id,bucketId + i); findNode(idInBucket,false); } } Msg.debug("Time spent:" + (Msg.getClock() - timeBegin)); return answerGot; } /* Send a request to find a node in the node's routing table. */ public boolean findNode(int destination, boolean counts) { int queries; int answers; int nodesAdded; boolean destinationFound; int steps = 0; double timeBeginReceive; double timeout; double globalTimeout = Msg.getClock() + Common.FIND_NODE_GLOBAL_TIMEOUT; //Build a list of the closest nodes we already know. Answer nodeList = table.findClosest(destination); Msg.verb("Doing a FIND_NODE on " + destination); do { timeBeginReceive = Msg.getClock(); answers = 0; queries = this.sendFindNodeToBest(nodeList); nodesAdded = 0; timeout = Msg.getClock() + Common.FIND_NODE_TIMEOUT; steps++; do { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (!comm.test()) { waitFor(1); } else { Task task = comm.getTask(); if (task instanceof FindNodeAnswerTask) { FindNodeAnswerTask answerTask = (FindNodeAnswerTask)task; //Check if we received what we are looking for. if (answerTask.getDestinationId() == destination) { table.update(answerTask.getSenderId()); //Add the answer to our routing table for (Contact c: answerTask.getAnswer().getNodes()) { table.update(c.getId()); } answers++; nodesAdded = nodeList.merge(answerTask.getAnswer()); } else { /* If it's not our answer, we answer to the node that has queried us anyway */ handleTask(task); //Update the timeout if it's not our answer. timeout += Msg.getClock() - timeBeginReceive; timeBeginReceive = Msg.getClock(); } } else { handleTask(task); timeout += Msg.getClock() - timeBeginReceive; timeBeginReceive = Msg.getClock(); } comm = null; } } catch (Exception e) { comm = null; } } while (answers < queries && Msg.getClock() < timeout); destinationFound = nodeList.destinationFound(); } while (!destinationFound && (nodesAdded > 0 || answers == 0) && Msg.getClock() < globalTimeout && steps < Common.MAX_STEPS); if (destinationFound) { if (counts) { findNodeSuccedded++; } Msg.debug("Find node on " + destination + " succedded"); } else { Msg.debug("Find node on " + destination + " failed"); Msg.debug("Queried " + queries + " nodes to find " + destination); Msg.debug(nodeList.toString()); if (counts) { findNodeFailed++; } } return destinationFound; } /** * @brief Sends a "PING" request to a node * @param destination Ping destination id. */ public void ping(int destination) { boolean destinationFound = false; double timeout = Msg.getClock() + Common.PING_TIMEOUT; PingTask pingTask = new PingTask(this.id); /* Sending the ping task */ pingTask.dsend(Integer.toString(destination)); do { try { Task task = Task.receive(Integer.toString(this.id),Common.PING_TIMEOUT); if (task instanceof PingAnswerTask) { PingAnswerTask answerTask = (PingAnswerTask)task; if (answerTask.getSenderId() == destination) { this.table.update(destination); destinationFound = true; } else { handleTask(task); } } else { handleTask(task); } waitFor(1); } catch (Exception ex) { Msg.debug("Caught exception: " + ex); } } while (Msg.getClock() < timeout && !destinationFound); } /** * @brief Sends a "FIND_NODE" request (task) to a node we know. * @param id Id of the node we are querying * @param destination id of the node we are trying to find. */ public void sendFindNode(int id, int destination) { Msg.debug("Sending a FIND_NODE to " + Integer.toString(id) + " to find " + destination ); FindNodeTask task = new FindNodeTask(this.id,destination); task.dsend(Integer.toString(id)); } /** Sends a "FIND_NODE" request to the best "alpha" nodes in a node list */ public int sendFindNodeToBest(Answer nodeList) { int destination = nodeList.getDestinationId(); int i; for (i = 0; i < Common.ALPHA && i < nodeList.size(); i++) { Contact node = nodeList.getNodes().get(i); if (node.getId() != this.id) { this.sendFindNode(node.getId(),destination); } } return i; } public void randomLookup() { findNode(0,true); } /** * @brief Handles an incomming task * @param task The task we need to handle */ public void handleTask(Task task) { if (task instanceof KademliaTask) { table.update(((KademliaTask) task).getSenderId()); if (task instanceof FindNodeTask) { handleFindNode((FindNodeTask)task); } else if (task instanceof PingTask) { handlePing((PingTask)task); } } } public void handleFindNode(FindNodeTask task) { Msg.debug("Received a FIND_NODE from " + task.getSenderId()); Answer answer = table.findClosest(task.getDestination()); FindNodeAnswerTask taskToSend = new FindNodeAnswerTask(this.id,task.getDestination(),answer); taskToSend.dsend(Integer.toString(task.getSenderId())); } public void handlePing(PingTask task) { Msg.debug("Received a PING from " + task.getSenderId()); PingAnswerTask taskToSend = new PingAnswerTask(this.id); taskToSend.dsend(Integer.toString(task.getSenderId())); } } SimGrid-3.18/examples/java/dht/kademlia/Common.java0000644000175000017500000000235013217757326022504 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class Common { /* Common constants used all over the simulation */ public static final int COMM_SIZE = 1; public static final int COMP_SIZE = 0; public static final int RANDOM_LOOKUP_INTERVAL = 100; public static final int ALPHA = 3; public static final int IDENTIFIER_SIZE = 32; /* Maximum size of the buckets */ public static final int BUCKET_SIZE = 20; /* Maximum number of trials for the "JOIN" request */ public static final int MAX_JOIN_TRIALS = 4; /* Timeout for a "FIND_NODE" request to a node */ public static final int FIND_NODE_TIMEOUT = 10; /* Global timeout for a FIND_NODE request */ public static final int FIND_NODE_GLOBAL_TIMEOUT = 50; /* Timeout for a "PING" request */ public static final int PING_TIMEOUT = 35; public static final int MAX_STEPS = 10; public static final int JOIN_BUCKETS_QUERIES = 1; private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.18/examples/java/dht/kademlia/Contact.java0000644000175000017500000000223313217757326022647 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class Contact implements Comparable { private int id; private int distance; public Contact(int id, int distance) { this.id = id; this.distance = distance; } public int getId() { return id; } public int getDistance() { return distance; } @Override public boolean equals(Object x) { return (x == null) ? false : x.equals(id) ; } @Override public int hashCode() { int hash = 1; hash = hash * 17 + id; hash = hash * 31 + distance; return hash; } @Override public int compareTo(Object o) { Contact c = (Contact)o; if (distance < c.distance) { return -1; } else if (distance == c.distance) { return 0; } else { return 1; } } @Override public String toString() { return "Contact [id=" + id + ", distance=" + distance + "]"; } }SimGrid-3.18/examples/java/dht/kademlia/dht-kademlia.tesh0000644000175000017500000000213313217757334023620 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} dht/kademlia/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dht/kademlia/kademlia.xml > [0.000000] [java/INFO] Using regular java threads. > [900.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... > [node-0.acme.org:dht.kademlia.Node:(1) 0.000000] [java/INFO] Hi, I'm going to create the network with the id 0! > [node-0.acme.org:dht.kademlia.Node:(1) 900.000000] [java/INFO] 8/8 FIND_NODE have succedded. > [node-1.acme.org:dht.kademlia.Node:(2) 0.000000] [java/INFO] Hi, I'm going to join the network with the id 1! > [node-1.acme.org:dht.kademlia.Node:(2) 900.000000] [java/INFO] 8/8 FIND_NODE have succedded. > [node-2.acme.org:dht.kademlia.Node:(3) 0.000000] [java/INFO] Hi, I'm going to join the network with the id 2! > [node-2.acme.org:dht.kademlia.Node:(3) 900.000000] [java/INFO] 8/8 FIND_NODE have succedded. > [node-3.acme.org:dht.kademlia.Node:(4) 0.000000] [java/INFO] Hi, I'm going to join the network with the id 4! > [node-3.acme.org:dht.kademlia.Node:(4) 900.000000] [java/INFO] 8/8 FIND_NODE have succedded. SimGrid-3.18/examples/java/trace/0000755000175000017500000000000013217757326017161 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/trace/pingpong/0000755000175000017500000000000013217757334021001 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/trace/pingpong/Main.java0000644000175000017500000000276713217757326022545 0ustar mquinsonmquinson/* Copyright (c) 2006-2007, 2012-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.trace.Trace; public class Main { public static final String PM_STATE = "PM_STATE"; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Main platform_file"); Msg.info("example : Main ../platforms/platform.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); new Sender("Jacquelin", "Sender", new String[] {"Boivin", "Tremblay"}).start(); new Receiver ("Boivin", "Receiver", null).start(); new Receiver ("Tremblay", "Receiver", null).start(); /* Initialize some state for the hosts */ Trace.hostStateDeclare (PM_STATE); Trace.hostStateDeclareValue (PM_STATE, "waitingPing", "0 0 1"); Trace.hostStateDeclareValue (PM_STATE, "sendingPong", "0 1 0"); Trace.hostStateDeclareValue (PM_STATE, "sendingPing", "0 1 1"); Trace.hostStateDeclareValue (PM_STATE, "waitingPong", "1 0 0"); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/trace/pingpong/Sender.java0000644000175000017500000000447213217757326023074 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; import org.simgrid.trace.Trace; public class Sender extends Process { private static final double COMM_SIZE_LAT = 1; private static final double COMM_SIZE_BW = 100000000; private static final String PM_STATE = Main.PM_STATE; public Sender(String hostname, String name, String[] args) throws HostNotFoundException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello !"); Trace.hostPushState (getHost().getName(), PM_STATE, "sendingPing"); int hostCount = args.length; Msg.info("host count: " + hostCount); String[] mailboxes = new String[hostCount]; double time; double computeDuration = 0; PingPongTask ping; for(int pos = 0; pos < args.length ; pos++) { mailboxes[pos] = Host.getByName(args[pos]).getName(); } for (int pos = 0; pos < hostCount; pos++) { time = Msg.getClock(); Msg.info("sender time: " + time); ping = new PingPongTask("no name",computeDuration,COMM_SIZE_LAT); ping.setTime(time); ping.send(mailboxes[pos]); Trace.hostPushState (getHost().getName(), PM_STATE, "waitingPong"); Task.receive(getHost().getName()); double timeGot = Msg.getClock(); double timeSent = ping.getTime(); double communicationTime; Msg.info("Got at time "+ timeGot); Msg.info("Was sent at time "+timeSent); time = timeSent; communicationTime=timeGot - time; Msg.info("Communication time : " + communicationTime); Msg.info(" --- bw "+ COMM_SIZE_BW/communicationTime + " ----"); /* Pop the last state (going back to sending ping) */ Trace.hostPopState (getHost().getName(), PM_STATE); } /* Pop the sendingPong state */ Trace.hostPopState (getHost().getName(), PM_STATE); Msg.info("goodbye!"); } } SimGrid-3.18/examples/java/trace/pingpong/trace-pingpong.tesh0000644000175000017500000000475213217757334024613 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} trace/pingpong/Main ${srcdir:=.}/../platforms/small_platform.xml --cfg=tracing:yes --cfg=tracing/filename:simulation.trace --cfg=tracing/platform:yes > [0.000000] [java/INFO] Using regular java threads. > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/filename' to 'simulation.trace' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/platform' to 'yes' > [3.817809] [java/INFO] MSG_main finished; Cleaning up the simulation... > [Boivin:Receiver:(2) 0.000000] [java/INFO] hello! > [Boivin:Receiver:(2) 0.000000] [java/INFO] try to get a task > [Boivin:Receiver:(2) 1.048882] [java/INFO] Got at time 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [java/INFO] Was sent at time 0.0 > [Boivin:Receiver:(2) 1.048882] [java/INFO] Communication time : 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [java/INFO] --- bw 9.533962169004269E7 ---- > [Boivin:Receiver:(2) 2.097764] [java/INFO] goodbye! > [Jacquelin:Sender:(1) 0.000000] [java/INFO] hello ! > [Jacquelin:Sender:(1) 0.000000] [java/INFO] host count: 2 > [Jacquelin:Sender:(1) 0.000000] [java/INFO] sender time: 0.0 > [Jacquelin:Sender:(1) 2.097764] [java/INFO] Got at time 2.0977637256650463 > [Jacquelin:Sender:(1) 2.097764] [java/INFO] Was sent at time 0.0 > [Jacquelin:Sender:(1) 2.097764] [java/INFO] Communication time : 2.0977637256650463 > [Jacquelin:Sender:(1) 2.097764] [java/INFO] --- bw 4.7669810845021345E7 ---- > [Jacquelin:Sender:(1) 2.097764] [java/INFO] sender time: 2.0977637256650463 > [Jacquelin:Sender:(1) 3.817809] [java/INFO] Got at time 3.8178087458100927 > [Jacquelin:Sender:(1) 3.817809] [java/INFO] Was sent at time 2.0977637256650463 > [Jacquelin:Sender:(1) 3.817809] [java/INFO] Communication time : 1.7200450201450463 > [Jacquelin:Sender:(1) 3.817809] [java/INFO] --- bw 5.813801315012516E7 ---- > [Jacquelin:Sender:(1) 3.817809] [java/INFO] goodbye! > [Tremblay:Receiver:(3) 0.000000] [java/INFO] hello! > [Tremblay:Receiver:(3) 0.000000] [java/INFO] try to get a task > [Tremblay:Receiver:(3) 2.957786] [java/INFO] Got at time 2.9577862357375695 > [Tremblay:Receiver:(3) 2.957786] [java/INFO] Was sent at time 2.0977637256650463 > [Tremblay:Receiver:(3) 2.957786] [java/INFO] Communication time : 0.8600225100725232 > [Tremblay:Receiver:(3) 2.957786] [java/INFO] --- bw 1.1627602630025032E8 ---- > [Tremblay:Receiver:(3) 3.817809] [java/INFO] goodbye! $ rm -rf simulation.trace SimGrid-3.18/examples/java/trace/pingpong/PingPongTask.java0000644000175000017500000000132513217757326024212 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Task; public class PingPongTask extends Task { private double timeVal; public PingPongTask() { this.timeVal = 0; } public PingPongTask(String name, double computeDuration, double messageSize) { super(name,computeDuration,messageSize); } public void setTime(double timeVal){ this.timeVal = timeVal; } public double getTime() { return this.timeVal; } } SimGrid-3.18/examples/java/trace/pingpong/Receiver.java0000644000175000017500000000365713217757326023424 0ustar mquinsonmquinson/* Copyright (c) 2006-2007, 2012-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; import org.simgrid.trace.Trace; public class Receiver extends Process { private static final double COMM_SIZE_LAT = 1; private static final double COMM_SIZE_BW = 100000000; private static final String PM_STATE = Main.PM_STATE; public Receiver(String hostname, String name, String[]args) throws HostNotFoundException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello!"); Trace.hostPushState (getHost().getName(), PM_STATE, "waitingPing"); /* Wait for the ping */ Msg.info("try to get a task"); PingPongTask ping = (PingPongTask)Task.receive(getHost().getName()); double timeGot = Msg.getClock(); double timeSent = ping.getTime(); Msg.info("Got at time "+ timeGot); Msg.info("Was sent at time "+timeSent); double time=timeSent; double communicationTime=timeGot - time; Msg.info("Communication time : " + communicationTime); Msg.info(" --- bw "+ COMM_SIZE_BW/communicationTime + " ----"); /* Send the pong */ Trace.hostPushState (getHost().getName(), PM_STATE, "sendingPong"); double computeDuration = 0; PingPongTask pong = new PingPongTask("no name",computeDuration,COMM_SIZE_LAT); pong.setTime(time); pong.send(ping.getSource().getName()); /* Pop the two states */ Trace.hostPopState (getHost().getName(), PM_STATE); Trace.hostPopState (getHost().getName(), PM_STATE); Msg.info("goodbye!"); } } SimGrid-3.18/examples/java/app/0000755000175000017500000000000013217757325016642 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/app/tokenring/0000755000175000017500000000000013217757334020642 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/app/tokenring/Main.java0000644000175000017500000000177113217757325022377 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.tokenring; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; class Main { private Main() { /* This is just to prevent anyone from creating an instance of this singleton */ throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); String platform = "../platforms/small_platform.xml"; if(args.length >= 1) platform = args[0]; Msg.createEnvironment(platform); Host[] hosts = Host.all(); for (int rank = 0; rank < hosts.length; rank++) { Process proc = new RelayRunner(hosts[rank], Integer.toString(rank), null); proc.start(); } Msg.info("Number of hosts '"+hosts.length+"'"); Msg.run(); Msg.info("Simulation time "+Msg.getClock()); } } SimGrid-3.18/examples/java/app/tokenring/app-tokenring.tesh0000644000175000017500000002453313217757334024314 0ustar mquinsonmquinson#! ./tesh $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/routing_cluster.xml '--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '6' > [ 0.000000] (1:0@host1) Host '0' send 'Token' to Host '1' > [ 0.017354] (2:1@host2) Host '1' received 'Token' > [ 0.017354] (2:1@host2) Host '1' send 'Token' to Host '2' > [ 0.035121] (3:2@host3) Host '2' received 'Token' > [ 0.035121] (3:2@host3) Host '2' send 'Token' to Host '3' > [ 0.065898] (4:3@host4) Host '3' received 'Token' > [ 0.065898] (4:3@host4) Host '3' send 'Token' to Host '4' > [ 0.083252] (5:4@host5) Host '4' received 'Token' > [ 0.083252] (5:4@host5) Host '4' send 'Token' to Host '5' > [ 0.101019] (6:5@host6) Host '5' received 'Token' > [ 0.101019] (6:5@host6) Host '5' send 'Token' to Host '0' > [ 0.131796] (1:0@host1) Host '0' received 'Token' > [ 0.131796] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 0.131796] (0:maestro@) Simulation time 0.13179602061855672 $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/two_peers.xml '--log=root.fmt:[%12.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '2' > [ 0.000000] (1:0@100030591) Host '0' send 'Token' to Host '1' > [ 0.624423] (2:1@100036570) Host '1' received 'Token' > [ 0.624423] (2:1@100036570) Host '1' send 'Token' to Host '0' > [ 1.248846] (1:0@100030591) Host '0' received 'Token' > [ 1.248846] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 1.248846] (0:maestro@) Simulation time 1.2488464578972847 $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/meta_cluster.xml '--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '60' > [ 0.000000] (1:0@host-1.cluster1) Host '0' send 'Token' to Host '1' > [ 0.030364] (2:1@host-1.cluster2) Host '1' received 'Token' > [ 0.030364] (2:1@host-1.cluster2) Host '1' send 'Token' to Host '2' > [ 0.060729] (3:2@host-10.cluster1) Host '2' received 'Token' > [ 0.060729] (3:2@host-10.cluster1) Host '2' send 'Token' to Host '3' > [ 0.091093] (4:3@host-10.cluster2) Host '3' received 'Token' > [ 0.091093] (4:3@host-10.cluster2) Host '3' send 'Token' to Host '4' > [ 0.121458] (5:4@host-11.cluster1) Host '4' received 'Token' > [ 0.121458] (5:4@host-11.cluster1) Host '4' send 'Token' to Host '5' > [ 0.151822] (6:5@host-11.cluster2) Host '5' received 'Token' > [ 0.151822] (6:5@host-11.cluster2) Host '5' send 'Token' to Host '6' > [ 0.182187] (7:6@host-12.cluster1) Host '6' received 'Token' > [ 0.182187] (7:6@host-12.cluster1) Host '6' send 'Token' to Host '7' > [ 0.212551] (8:7@host-12.cluster2) Host '7' received 'Token' > [ 0.212551] (8:7@host-12.cluster2) Host '7' send 'Token' to Host '8' > [ 0.242915] (9:8@host-13.cluster1) Host '8' received 'Token' > [ 0.242915] (9:8@host-13.cluster1) Host '8' send 'Token' to Host '9' > [ 0.273280] (10:9@host-13.cluster2) Host '9' received 'Token' > [ 0.273280] (10:9@host-13.cluster2) Host '9' send 'Token' to Host '10' > [ 0.303644] (11:10@host-14.cluster1) Host '10' received 'Token' > [ 0.303644] (11:10@host-14.cluster1) Host '10' send 'Token' to Host '11' > [ 0.334009] (12:11@host-14.cluster2) Host '11' received 'Token' > [ 0.334009] (12:11@host-14.cluster2) Host '11' send 'Token' to Host '12' > [ 0.364373] (13:12@host-15.cluster1) Host '12' received 'Token' > [ 0.364373] (13:12@host-15.cluster1) Host '12' send 'Token' to Host '13' > [ 0.394737] (14:13@host-15.cluster2) Host '13' received 'Token' > [ 0.394737] (14:13@host-15.cluster2) Host '13' send 'Token' to Host '14' > [ 0.425102] (15:14@host-16.cluster1) Host '14' received 'Token' > [ 0.425102] (15:14@host-16.cluster1) Host '14' send 'Token' to Host '15' > [ 0.455466] (16:15@host-16.cluster2) Host '15' received 'Token' > [ 0.455466] (16:15@host-16.cluster2) Host '15' send 'Token' to Host '16' > [ 0.485831] (17:16@host-17.cluster1) Host '16' received 'Token' > [ 0.485831] (17:16@host-17.cluster1) Host '16' send 'Token' to Host '17' > [ 0.516195] (18:17@host-17.cluster2) Host '17' received 'Token' > [ 0.516195] (18:17@host-17.cluster2) Host '17' send 'Token' to Host '18' > [ 0.546560] (19:18@host-18.cluster1) Host '18' received 'Token' > [ 0.546560] (19:18@host-18.cluster1) Host '18' send 'Token' to Host '19' > [ 0.576924] (20:19@host-18.cluster2) Host '19' received 'Token' > [ 0.576924] (20:19@host-18.cluster2) Host '19' send 'Token' to Host '20' > [ 0.607288] (21:20@host-19.cluster1) Host '20' received 'Token' > [ 0.607288] (21:20@host-19.cluster1) Host '20' send 'Token' to Host '21' > [ 0.637653] (22:21@host-19.cluster2) Host '21' received 'Token' > [ 0.637653] (22:21@host-19.cluster2) Host '21' send 'Token' to Host '22' > [ 0.668017] (23:22@host-2.cluster1) Host '22' received 'Token' > [ 0.668017] (23:22@host-2.cluster1) Host '22' send 'Token' to Host '23' > [ 0.698382] (24:23@host-2.cluster2) Host '23' received 'Token' > [ 0.698382] (24:23@host-2.cluster2) Host '23' send 'Token' to Host '24' > [ 0.728746] (25:24@host-20.cluster1) Host '24' received 'Token' > [ 0.728746] (25:24@host-20.cluster1) Host '24' send 'Token' to Host '25' > [ 0.759111] (26:25@host-20.cluster2) Host '25' received 'Token' > [ 0.759111] (26:25@host-20.cluster2) Host '25' send 'Token' to Host '26' > [ 0.789475] (27:26@host-21.cluster1) Host '26' received 'Token' > [ 0.789475] (27:26@host-21.cluster1) Host '26' send 'Token' to Host '27' > [ 0.819839] (28:27@host-21.cluster2) Host '27' received 'Token' > [ 0.819839] (28:27@host-21.cluster2) Host '27' send 'Token' to Host '28' > [ 0.850204] (29:28@host-22.cluster1) Host '28' received 'Token' > [ 0.850204] (29:28@host-22.cluster1) Host '28' send 'Token' to Host '29' > [ 0.880568] (30:29@host-22.cluster2) Host '29' received 'Token' > [ 0.880568] (30:29@host-22.cluster2) Host '29' send 'Token' to Host '30' > [ 0.910933] (31:30@host-23.cluster1) Host '30' received 'Token' > [ 0.910933] (31:30@host-23.cluster1) Host '30' send 'Token' to Host '31' > [ 0.941297] (32:31@host-23.cluster2) Host '31' received 'Token' > [ 0.941297] (32:31@host-23.cluster2) Host '31' send 'Token' to Host '32' > [ 0.971662] (33:32@host-24.cluster1) Host '32' received 'Token' > [ 0.971662] (33:32@host-24.cluster1) Host '32' send 'Token' to Host '33' > [ 1.002026] (34:33@host-24.cluster2) Host '33' received 'Token' > [ 1.002026] (34:33@host-24.cluster2) Host '33' send 'Token' to Host '34' > [ 1.032390] (35:34@host-25.cluster1) Host '34' received 'Token' > [ 1.032390] (35:34@host-25.cluster1) Host '34' send 'Token' to Host '35' > [ 1.062755] (36:35@host-25.cluster2) Host '35' received 'Token' > [ 1.062755] (36:35@host-25.cluster2) Host '35' send 'Token' to Host '36' > [ 1.093119] (37:36@host-26.cluster1) Host '36' received 'Token' > [ 1.093119] (37:36@host-26.cluster1) Host '36' send 'Token' to Host '37' > [ 1.123484] (38:37@host-26.cluster2) Host '37' received 'Token' > [ 1.123484] (38:37@host-26.cluster2) Host '37' send 'Token' to Host '38' > [ 1.153848] (39:38@host-27.cluster1) Host '38' received 'Token' > [ 1.153848] (39:38@host-27.cluster1) Host '38' send 'Token' to Host '39' > [ 1.184212] (40:39@host-27.cluster2) Host '39' received 'Token' > [ 1.184212] (40:39@host-27.cluster2) Host '39' send 'Token' to Host '40' > [ 1.214577] (41:40@host-28.cluster1) Host '40' received 'Token' > [ 1.214577] (41:40@host-28.cluster1) Host '40' send 'Token' to Host '41' > [ 1.244941] (42:41@host-28.cluster2) Host '41' received 'Token' > [ 1.244941] (42:41@host-28.cluster2) Host '41' send 'Token' to Host '42' > [ 1.275306] (43:42@host-29.cluster1) Host '42' received 'Token' > [ 1.275306] (43:42@host-29.cluster1) Host '42' send 'Token' to Host '43' > [ 1.305670] (44:43@host-29.cluster2) Host '43' received 'Token' > [ 1.305670] (44:43@host-29.cluster2) Host '43' send 'Token' to Host '44' > [ 1.336035] (45:44@host-3.cluster1) Host '44' received 'Token' > [ 1.336035] (45:44@host-3.cluster1) Host '44' send 'Token' to Host '45' > [ 1.366399] (46:45@host-3.cluster2) Host '45' received 'Token' > [ 1.366399] (46:45@host-3.cluster2) Host '45' send 'Token' to Host '46' > [ 1.396763] (47:46@host-30.cluster1) Host '46' received 'Token' > [ 1.396763] (47:46@host-30.cluster1) Host '46' send 'Token' to Host '47' > [ 1.427128] (48:47@host-30.cluster2) Host '47' received 'Token' > [ 1.427128] (48:47@host-30.cluster2) Host '47' send 'Token' to Host '48' > [ 1.457492] (49:48@host-4.cluster1) Host '48' received 'Token' > [ 1.457492] (49:48@host-4.cluster1) Host '48' send 'Token' to Host '49' > [ 1.487857] (50:49@host-4.cluster2) Host '49' received 'Token' > [ 1.487857] (50:49@host-4.cluster2) Host '49' send 'Token' to Host '50' > [ 1.518221] (51:50@host-5.cluster1) Host '50' received 'Token' > [ 1.518221] (51:50@host-5.cluster1) Host '50' send 'Token' to Host '51' > [ 1.548586] (52:51@host-5.cluster2) Host '51' received 'Token' > [ 1.548586] (52:51@host-5.cluster2) Host '51' send 'Token' to Host '52' > [ 1.578950] (53:52@host-6.cluster1) Host '52' received 'Token' > [ 1.578950] (53:52@host-6.cluster1) Host '52' send 'Token' to Host '53' > [ 1.609314] (54:53@host-6.cluster2) Host '53' received 'Token' > [ 1.609314] (54:53@host-6.cluster2) Host '53' send 'Token' to Host '54' > [ 1.639679] (55:54@host-7.cluster1) Host '54' received 'Token' > [ 1.639679] (55:54@host-7.cluster1) Host '54' send 'Token' to Host '55' > [ 1.670043] (56:55@host-7.cluster2) Host '55' received 'Token' > [ 1.670043] (56:55@host-7.cluster2) Host '55' send 'Token' to Host '56' > [ 1.700408] (57:56@host-8.cluster1) Host '56' received 'Token' > [ 1.700408] (57:56@host-8.cluster1) Host '56' send 'Token' to Host '57' > [ 1.730772] (58:57@host-8.cluster2) Host '57' received 'Token' > [ 1.730772] (58:57@host-8.cluster2) Host '57' send 'Token' to Host '58' > [ 1.761137] (59:58@host-9.cluster1) Host '58' received 'Token' > [ 1.761137] (59:58@host-9.cluster1) Host '58' send 'Token' to Host '59' > [ 1.791501] (60:59@host-9.cluster2) Host '59' received 'Token' > [ 1.791501] (60:59@host-9.cluster2) Host '59' send 'Token' to Host '0' > [ 1.821865] (1:0@host-1.cluster1) Host '0' received 'Token' > [ 1.821865] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 1.821865] (0:maestro@) Simulation time 1.8218653608247406 SimGrid-3.18/examples/java/app/tokenring/RelayRunner.java0000644000175000017500000000374213217757325023761 0ustar mquinsonmquinson/* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.tokenring; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; public class RelayRunner extends Process { private static final int TASK_COMM_SIZE = 1000000; /* The token is 1MB long*/ public RelayRunner(Host host, String name, String[]args) { super(host,name,args); } /* This is the function executed by this kind of processes */ @Override public void main(String[] args) throws MsgException { // In this example, the processes are given numerical names: "0", "1", "2", and so on int rank = Integer.parseInt(this.getName()); if (rank == 0) { /* The root (rank 0) first sends the token then waits to receive it back */ String mailbox = "1"; Task token = new Task("Token", 0/* no computation associated*/ , TASK_COMM_SIZE ); Msg.info("Host '"+rank+"' send '"+token.getName()+"' to Host '"+mailbox+"'"); token.send(mailbox); token = Task.receive(this.getName()); // Get a message from the mailbox having the same name as the current processor Msg.info("Host '"+rank+"' received '"+token.getName()+"'"); } else { /* The others processes receive on their name (coming from their left neighbor -- rank-1) * and send to their right neighbor (rank+1) */ Task token = Task.receive(this.getName()); Msg.info("Host '"+rank+"' received '"+token.getName()+"'"); String mailbox = Integer.toString(rank+1); if (rank+1 == Host.getCount()) { /* The last process has no right neighbor, so it sends the token back to rank 0 */ mailbox = "0"; } Msg.info("Host '"+rank+"' send '"+token.getName()+"' to Host '"+mailbox+"'"); token.send(mailbox); } } }SimGrid-3.18/examples/java/app/bittorrent/0000755000175000017500000000000013217757343021036 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/app/bittorrent/Main.java0000644000175000017500000000170713217757325022572 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import org.simgrid.msg.Msg; import org.simgrid.msg.RngStream; class Main{ private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { int[] seed = { 12345, 12345, 12345, 12345, 12345, 12345 }; RngStream.setPackageSeed(seed); Msg.init(args); if(args.length < 2) { Msg.info("Usage : Bittorrent platform_file deployment_file"); Msg.info("example : Bittorrent ../platforms/cluster.xml bittorrent.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/app/bittorrent/generate.py0000755000175000017500000000313313217757325023205 0ustar mquinsonmquinson#!/usr/bin/env python # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. # This script generates a specific deployment file for the Bittorrent example. # It assumes that the platform will be a cluster. # Usage: python generate.py nb_nodes nb_bits end_date percentage # Example: python generate.py 10000 5000 import sys import random if len(sys.argv) != 4: print( "Usage: python generate.py nb_nodes end_date seed_percentage > deployment_file.xml") sys.exit(1) nb_nodes = int(sys.argv[1]) end_date = int(sys.argv[2]) seed_percentage = int(sys.argv[3]) nb_bits = 24 max_id = 2 ** nb_bits - 1 all_ids = [42] sys.stdout.write("\n" "\n" "\n" " \n" % end_date) for i in range(1, nb_nodes): ok = False while not ok: my_id = random.randint(0, max_id) ok = not my_id in all_ids start_date = i * 10 line = " " % ( i, my_id, end_date) if random.randint(0, 100) < seed_percentage: line += "" line += "\n" sys.stdout.write(line) all_ids.append(my_id) sys.stdout.write("") SimGrid-3.18/examples/java/app/bittorrent/Peer.java0000644000175000017500000005351413217757325022604 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import java.util.HashMap; import java.util.Iterator; import java.util.Map.Entry; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.RngStream; import org.simgrid.msg.MsgException; public class Peer extends Process { protected int round = 0; protected double beginReceiveTime; protected double deadline; protected static RngStream stream = new RngStream(); protected int id; protected String mailbox; protected String mailboxTracker; protected String hostname; protected int pieces = 0; protected char[] bitfield = new char[Common.FILE_PIECES]; protected char[][] bitfieldBlocks = new char[Common.FILE_PIECES][Common.PIECES_BLOCKS]; protected short[] piecesCount = new short[Common.FILE_PIECES]; protected int piecesRequested = 0; protected ArrayList currentPieces = new ArrayList<>(); protected int currentPiece = -1; protected HashMap activePeers = new HashMap<>(); protected HashMap peers = new HashMap<>(); protected Comm commReceived = null; public Peer(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { //Check arguments if (args.length != 3 && args.length != 2) { Msg.info("Wrong number of arguments"); } if (args.length == 3) { init(Integer.parseInt(args[0]),true); } else { init(Integer.parseInt(args[0]),false); } //Retrieve the deadline deadline = Double.parseDouble(args[1]); if (deadline < 0) { Msg.info("Wrong deadline supplied"); return; } Msg.info("Hi, I'm joining the network with id " + id); //Getting peer data from the tracker if (getPeersData()) { Msg.debug("Got " + peers.size() + " peers from the tracker"); Msg.debug("Here is my current status: " + getStatus()); beginReceiveTime = Msg.getClock(); if (hasFinished()) { pieces = Common.FILE_PIECES; sendHandshakeAll(); seedLoop(); } else { leechLoop(); seedLoop(); } } else { Msg.info("Couldn't contact the tracker."); } Msg.info("Here is my current status: " + getStatus()); } private void leechLoop() { double nextChokedUpdate = Msg.getClock() + Common.UPDATE_CHOKED_INTERVAL; Msg.debug("Start downloading."); // Send a "handshake" message to all the peers it got(it couldn't have gotten more than 50 peers anyway) sendHandshakeAll(); //Wait for at least one "bitfield" message. waitForPieces(); Msg.debug("Starting main leech loop"); while (Msg.getClock() < deadline && pieces < Common.FILE_PIECES) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { if (commReceived.test()) { handleMessage(commReceived.getTask()); commReceived = null; } else { //If the user has a pending interesting if (currentPiece != -1) { sendInterestedToPeers(); } else { if (currentPieces.size() < Common.MAX_PIECES) { updateCurrentPiece(); } } //We don't execute the choke algorithm if we don't already have a piece if (Msg.getClock() >= nextChokedUpdate && pieces > 0) { updateChokedPeers(); nextChokedUpdate += Common.UPDATE_CHOKED_INTERVAL; } else { waitFor(1); } } } catch (MsgException e) { e.printStackTrace(); commReceived = null; } } } private void seedLoop() { double nextChokedUpdate = Msg.getClock() + Common.UPDATE_CHOKED_INTERVAL; Msg.debug("Start seeding."); //start the main seed loop while (Msg.getClock() < deadline) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { if (commReceived.test()) { handleMessage(commReceived.getTask()); commReceived = null; } else { if (Msg.getClock() >= nextChokedUpdate) { updateChokedPeers(); //TODO: Change the choked peer algorithm when seeding nextChokedUpdate += Common.UPDATE_CHOKED_INTERVAL; } else { waitFor(1); } } } catch (MsgException e) { commReceived = null; } } } /** * @brief Initialize the various peer data * @param id id of the peer to take in the network * @param seed indicates if the peer is a seed */ private void init(int id, boolean seed) { this.id = id; this.mailbox = Integer.toString(id); this.mailboxTracker = "tracker_" + Integer.toString(id); if (seed) { for (int i = 0; i < bitfield.length; i++) { bitfield[i] = '1'; for (int j = 0; j < bitfieldBlocks[i].length; j++) { bitfieldBlocks[i][j] = '1'; } } } else { for (int i = 0; i < bitfield.length; i++) { bitfield[i] = '0'; for (int j = 0; j < bitfieldBlocks[i].length; j++) { bitfieldBlocks[i][j] = '0' ; } } } this.hostname = getHost().getName(); } private boolean getPeersData() { boolean success = false; boolean sendSuccess = false; double timeout = Msg.getClock() + Common.GET_PEERS_TIMEOUT; //Build the task to send to the tracker TrackerTask taskSend = new TrackerTask(hostname, mailboxTracker, id); while (!sendSuccess && Msg.getClock() < timeout) { try { Msg.debug("Sending a peer request to the tracker."); taskSend.send(Common.TRACKER_MAILBOX,Common.GET_PEERS_TIMEOUT); sendSuccess = true; } catch (MsgException e) { e.printStackTrace(); } } while (!success && Msg.getClock() < timeout) { commReceived = Task.irecv(this.mailboxTracker); try { commReceived.waitCompletion(Common.GET_PEERS_TIMEOUT); if (commReceived.getTask() instanceof TrackerTask) { TrackerTask task = (TrackerTask)commReceived.getTask(); for (Integer peerId: task.peers) { if (peerId != this.id) { peers.put(peerId, new Connection(peerId)); } } success = true; } } catch (MsgException e) { e.printStackTrace(); } commReceived = null; } commReceived = null; return success; } private void handleMessage(Task task) { MessageTask message = (MessageTask)task; Connection remotePeer = peers.get(message.peerId); switch (message.type) { case HANDSHAKE: Msg.debug("Received a HANDSHAKE message from " + message.mailbox); //Check if the peer is in our connection list if (remotePeer == null) { peers.put(message.peerId, new Connection(message.peerId)); sendHandshake(message.mailbox); } //Send our bitfield to the pair sendBitfield(message.mailbox); break; case BITFIELD: Msg.debug("Received a BITFIELD message from " + message.peerId + " (" + message.issuerHostname + ")"); //update the pieces list updatePiecesCountFromBitfield(message.bitfield); //Update the current piece if (currentPiece == -1 && pieces < Common.FILE_PIECES && currentPieces.size() < Common.MAX_PIECES) { updateCurrentPiece(); } remotePeer.bitfield = message.bitfield.clone(); break; case INTERESTED: Msg.debug("Received an INTERESTED message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.interested = true; break; case NOTINTERESTED: Msg.debug("Received a NOTINTERESTED message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.interested = false; break; case UNCHOKE: Msg.debug("Received an UNCHOKE message from " + message.peerId + "(" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.chokedDownload = false; activePeers.put(remotePeer.id,remotePeer); sendRequestsToPeer(remotePeer); break; case CHOKE: Msg.debug("Received a CHOKE message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.chokedDownload = true; activePeers.remove(remotePeer.id); break; case HAVE: if (remotePeer.bitfield == null) { return; } Msg.debug("Received a HAVE message from " + message.peerId + " (" + message.issuerHostname + ")"); assert message.index >= 0 && message.index < Common.FILE_PIECES; assert remotePeer.bitfield != null; remotePeer.bitfield[message.index] = '1'; piecesCount[message.index]++; //Send interested message to the peer if he has what we want if (!remotePeer.amInterested && currentPieces.contains(message.index) ) { remotePeer.amInterested = true; sendInterested(remotePeer.mailbox); } if (currentPieces.contains(message.index)) { int blockIndex = getFirstBlock(message.index); int blockLength = Common.PIECES_BLOCKS - blockIndex ; blockLength = blockLength > Common.BLOCKS_REQUESTED ? Common.BLOCKS_REQUESTED : blockLength; sendRequest(message.mailbox,message.index,blockIndex,blockLength); } break; case REQUEST: assert message.index >= 0 && message.index < Common.FILE_PIECES; if (!remotePeer.chokedUpload) { Msg.debug("Received a REQUEST from " + message.peerId + "(" + message.issuerHostname + ") for " + message.peerId); if (bitfield[message.index] == '1') { sendPiece(message.mailbox,message.index,false,message.blockIndex,message.blockLength); } else { Msg.debug("Received a REQUEST from " + message.peerId + " (" + message.issuerHostname + ") but he is choked" ); } } break; case PIECE: if (message.stalled) { Msg.debug("The received piece " + message.index + " from " + message.peerId + " (" + message.issuerHostname + ") is stalled"); } else { Msg.debug("Received piece " + message.index + " from " + message.peerId + " (" + message.issuerHostname + ")"); if (bitfield[message.index] == '0') { updateBitfieldBlocks(message.index,message.blockIndex,message.blockLength); if (pieceComplete(message.index)) { piecesRequested--; //Removing the piece from our piece list. currentPieces.remove((Object)Integer.valueOf(message.index)); //Setting the fact that we have the piece bitfield[message.index] = '1'; pieces++; Msg.debug("My status is now " + getStatus()); //Sending the information to all the peers we are connected to sendHave(message.index); //sending UNINTERESTED to peers that doesn't have what we want. updateInterestedAfterReceive(); } } else { Msg.debug("However, we already have it."); } } break; default: Msg.error("Unexpected message type: " + message.type); break; } if (remotePeer != null) { remotePeer.addSpeedValue(1 / (Msg.getClock() - beginReceiveTime)); } beginReceiveTime = Msg.getClock(); } private void waitForPieces() { boolean finished = false; while (Msg.getClock() < deadline && !finished) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { commReceived.waitCompletion(Common.TIMEOUT_MESSAGE); handleMessage(commReceived.getTask()); if (currentPiece != -1) { finished = true; } commReceived = null; } catch (MsgException e) { commReceived = null; } } } private boolean hasFinished() { for (int i = 0; i < bitfield.length; i++) { if (bitfield[i] == '1') { return true; } } return false; } /** * @brief Updates the list of who has a piece from a bitfield * @param bitfield bitfield */ private void updatePiecesCountFromBitfield(char[] bitfield) { for (int i = 0; i < Common.FILE_PIECES; i++) { if (bitfield[i] == '1') { piecesCount[i]++; } } } /** * Update the piece the peer is currently interested in. * There is two cases (as described in "Bittorrent Architecture Protocol", Ryan Toole : * If the peer has less than 3 pieces, he chooses a piece at random. * If the peer has more than pieces, he downloads the pieces that are the less * replicated */ private void updateCurrentPiece() { if (currentPieces.size() >= (Common.FILE_PIECES - pieces)) { return; } //TODO: trivial min algorithm when pieces >= 3 do { currentPiece = stream.randInt(0,Common.FILE_PIECES - 1); } while (!(bitfield[currentPiece] == '0' && !currentPieces.contains(currentPiece))); currentPieces.add(currentPiece); Msg.debug("New interested piece: " + currentPiece); assert currentPiece >= 0 && currentPiece < Common.FILE_PIECES; } // Update the list of current choked and unchoked peers, using the choke algorithm private void updateChokedPeers() { round = (round + 1) % 3; if (peers.size() == 0) { return; } //remove a peer from the list Iterator> it = activePeers.entrySet().iterator(); if (it.hasNext()) { Entry e = it.next(); Connection peerChoked = e.getValue(); peerChoked.chokedUpload = true; sendChoked(peerChoked.mailbox); activePeers.remove(e.getKey()); } Connection peerChoosed = null; //Separate the case from when the peer is seeding. if (pieces == Common.FILE_PIECES) { //Find the last unchoked peer. double unchokeTime = deadline + 1; for (Connection connection : peers.values()) { if (connection.lastUnchoke < unchokeTime && connection.interested) { peerChoosed = connection; unchokeTime = connection.lastUnchoke; } } } else { //Random optimistic unchoking if (round == 0) { int j = 0; do { int i = 0; int idChosen = stream.randInt(0,peers.size() - 1); for (Connection connection : peers.values()) { if (i == idChosen) { peerChoosed = connection; break; } i++; } //TODO: Not really the best way ever if (peerChoosed != null && !peerChoosed.interested) { peerChoosed = null; } j++; } while (peerChoosed == null && j < Common.MAXIMUM_PEERS); } else { Connection fastest = null; double fastestSpeed = 0; for (Connection c : peers.values()) { if (c.peerSpeed > fastestSpeed && c.interested && c.chokedUpload) { fastest = c; fastestSpeed = c.peerSpeed; } } peerChoosed = fastest; } } if (peerChoosed != null) { activePeers.put(peerChoosed.id,peerChoosed); peerChoosed.chokedUpload = false; peerChoosed.lastUnchoke = Msg.getClock(); sendUnchoked(peerChoosed.mailbox); } } // Updates our "interested" state about peers: send "not interested" to peers that don't have any more pieces we want. private void updateInterestedAfterReceive() { boolean interested; for (Connection connection : peers.values()) { interested = false; if (connection.amInterested) { for (Integer piece : currentPieces) { if (connection.bitfield[piece] == '1') { interested = true; break; } } if (!interested) { connection.amInterested = false; sendNotInterested(connection.mailbox); } } } } private void updateBitfieldBlocks(int index, int blockIndex, int blockLength) { for (int i = blockIndex; i < (blockIndex + blockLength); i++) { bitfieldBlocks[index][i] = '1'; } } // Returns if a piece is complete in the peer's bitfield. private boolean pieceComplete(int index) { for (int i = 0; i < bitfieldBlocks[index].length; i++) { if (bitfieldBlocks[index][i] == '0') { return false; } } return true; } // Returns the first block of a piece that we don't have. private int getFirstBlock(int piece) { int blockIndex = -1; for (int i = 0; i < Common.PIECES_BLOCKS; i++) { if (bitfieldBlocks[piece][i] == '0') { blockIndex = i; break; } } return blockIndex; } /** * @brief Send request messages to a peer that have unchoked us * @param remotePeer peer data to the peer we want to send the request */ private void sendRequestsToPeer(Connection remotePeer) { if (remotePeer.bitfield == null) { return; } for (Integer piece : currentPieces) { //Getting the block to send. int blockIndex = getFirstBlock(piece); int blockLength = Common.PIECES_BLOCKS - blockIndex ; blockLength = blockLength > Common.BLOCKS_REQUESTED ? Common.BLOCKS_REQUESTED : blockLength; if (remotePeer.bitfield[piece] == '1') { sendRequest(remotePeer.mailbox, piece, blockIndex, blockLength); } } } // Find the peers that have the current interested piece and send them the "interested" message private void sendInterestedToPeers() { if (currentPiece == -1) { return; } for (Connection connection : peers.values()) { if (connection.bitfield != null && connection.bitfield[currentPiece] == '1' && !connection.amInterested) { connection.amInterested = true; MessageTask task = new MessageTask(MessageTask.Type.INTERESTED, hostname, this.mailbox, this.id); task.dsend(connection.mailbox); } } currentPiece = -1; piecesRequested++; } // Send a "interested" message to a peer. private void sendInterested(String mailbox) { MessageTask task = new MessageTask(MessageTask.Type.INTERESTED, hostname, this.mailbox, this.id); task.dsend(mailbox); } /** * @brief Send a "not interested" message to a peer * @param mailbox mailbox destination mailbox */ private void sendNotInterested(String mailbox) { MessageTask task = new MessageTask(MessageTask.Type.NOTINTERESTED, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a handshake message to all the peers the peer has. private void sendHandshakeAll() { for (Connection remotePeer : peers.values()) { MessageTask task = new MessageTask(MessageTask.Type.HANDSHAKE, hostname, mailbox, id); task.dsend(remotePeer.mailbox); } } /** * @brief Send a "handshake" message to an user * @param mailbox mailbox where to we send the message */ private void sendHandshake(String mailbox) { Msg.debug("Sending a HANDSHAKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.HANDSHAKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "choked" message to a peer private void sendChoked(String mailbox) { Msg.debug("Sending a CHOKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.CHOKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "unchoked" message to a peer private void sendUnchoked(String mailbox) { Msg.debug("Sending a UNCHOKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.UNCHOKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "HAVE" message to all peers we are connected to private void sendHave(int piece) { Msg.debug("Sending HAVE message to all my peers"); for (Connection remotePeer : peers.values()) { MessageTask task = new MessageTask(MessageTask.Type.HAVE, hostname, this.mailbox, this.id, piece); task.dsend(remotePeer.mailbox); } } // Send a bitfield message to all the peers the peer has. private void sendBitfield(String mailbox) { Msg.debug("Sending a BITFIELD to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.BITFIELD, hostname, this.mailbox, this.id, this.bitfield); task.dsend(mailbox); } // Send a "request" message to a peer, containing a request for a piece private void sendRequest(String mailbox, int piece, int blockIndex, int blockLength) { Msg.debug("Sending a REQUEST to " + mailbox + " for piece " + piece + " and blocks " + blockIndex + "," + (blockIndex + blockLength)); MessageTask task = new MessageTask(MessageTask.Type.REQUEST, hostname, this.mailbox, this.id, piece, blockIndex, blockLength); task.dsend(mailbox); } // Send a "piece" message to a peer, containing a piece of the file private void sendPiece(String mailbox, int piece, boolean stalled, int blockIndex, int blockLength) { Msg.debug("Sending the PIECE " + piece + " to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.PIECE, hostname, this.mailbox, this.id, piece, stalled, blockIndex, blockLength); task.dsend(mailbox); } private String getStatus() { StringBuilder s = new StringBuilder(""); for (int i = 0; i < Common.FILE_PIECES; i++) s.append(bitfield[i]); return s.toString(); } } SimGrid-3.18/examples/java/app/bittorrent/TrackerTask.java0000644000175000017500000000241413217757325024120 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import org.simgrid.msg.Task; /* Task exchanged between the tracker and the peers. */ public class TrackerTask extends Task { public enum Type { REQUEST, ANSWER } protected Type type; protected String hostname; protected String mailbox; protected int peerId; protected int uploaded; protected int downloaded; protected int left; protected double interval; protected ArrayList peers; public TrackerTask(String hostname, String mailbox, int peerId) { this(hostname, mailbox, peerId, 0, 0, Common.FILE_SIZE); } public TrackerTask(String hostname, String mailbox, int peerId, int uploaded, int downloaded, int left) { super("", 0, Common.TRACKER_COMM_SIZE); this.type = Type.REQUEST; this.hostname = hostname; this.mailbox = mailbox; this.peerId = peerId; this.uploaded = uploaded; this.downloaded = downloaded; this.left = left; this.peers = new ArrayList<>(); } } SimGrid-3.18/examples/java/app/bittorrent/bittorrent.xml0000644000175000017500000000332113217757343023753 0ustar mquinsonmquinson SimGrid-3.18/examples/java/app/bittorrent/Connection.java0000644000175000017500000000303213217757325023776 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.Arrays; public class Connection { protected int id; protected char[] bitfield; protected String mailbox; // Indicates if we are interested in something this peer has protected boolean amInterested = false; // Indicates if the peer is interested in one of our pieces protected boolean interested = false; // Indicates if the peer is choked for the current peer protected boolean chokedUpload = true; // Indicates if the peer has choked the current peer protected boolean chokedDownload = true; // Number of messages we have received from the peer protected int messagesCount = 0; protected double peerSpeed = 0; protected double lastUnchoke = 0; public Connection(int id) { this.id = id; this.mailbox = Integer.toString(id); } // Add a new value to the peer speed average public void addSpeedValue(double speed) { peerSpeed = peerSpeed * 0.55 + speed * 0.45; } @Override public String toString() { return "Connection [id=" + id + ", bitfield=" + Arrays.toString(bitfield) + ", mailbox=" + mailbox + ", amInterested=" + amInterested + ", interested=" + interested + ", chokedUpload=" + chokedUpload + ", chokedDownload=" + chokedDownload + "]"; } } SimGrid-3.18/examples/java/app/bittorrent/MessageTask.java0000644000175000017500000000377413217757325024123 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import org.simgrid.msg.Task; public class MessageTask extends Task { public enum Type { HANDSHAKE, CHOKE, UNCHOKE, INTERESTED, NOTINTERESTED, HAVE, BITFIELD, REQUEST, PIECE } protected Type type; protected String issuerHostname; protected String mailbox; protected int peerId; protected char[] bitfield; protected int index; protected int blockIndex; protected int blockLength; protected boolean stalled; public MessageTask(Type type, String issuerHostname, String mailbox, int peerId) { this(type,issuerHostname,mailbox,peerId,-1,false,-1,-1); } public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index) { this(type,issuerHostname,mailbox,peerId,index,false,-1,-1); } // builds a new bitfield message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, char[] bitfield) { this(type,issuerHostname,mailbox,peerId,-1,false,-1,-1); this.bitfield = bitfield; } // build a new "request" message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index, int blockIndex, int blockLength) { this(type,issuerHostname,mailbox,peerId,index,false,blockIndex,blockLength); } // build a new "piece" message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index, boolean stalled, int blockIndex, int blockLength) { this.type = type; this.issuerHostname = issuerHostname; this.mailbox = mailbox; this.peerId = peerId; this.index = index; this.stalled = stalled; this.blockIndex = blockIndex; this.blockLength = blockLength; } } SimGrid-3.18/examples/java/app/bittorrent/app-bittorrent.tesh0000644000175000017500000000363013217757334024677 0ustar mquinsonmquinson#! tesh ! output sort 19 ! timeout 15 $ java -classpath ${classpath:=.} app/bittorrent/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/app/bittorrent/bittorrent.xml > [0.000000] [java/INFO] Using regular java threads. > [5000.046836] [java/INFO] MSG_main finished; Cleaning up the simulation... > [node-0.acme.org:app.bittorrent.Tracker:(1) 0.000000] [java/INFO] Tracker launched. > [node-0.acme.org:app.bittorrent.Tracker:(1) 3000.000000] [java/INFO] Tracker is leaving > [node-1.acme.org:app.bittorrent.Peer:(2) 0.000000] [java/INFO] Hi, I'm joining the network with id 2 > [node-1.acme.org:app.bittorrent.Peer:(2) 5000.007806] [java/INFO] Here is my current status: 1111111111 > [node-2.acme.org:app.bittorrent.Peer:(3) 0.000000] [java/INFO] Hi, I'm joining the network with id 3 > [node-2.acme.org:app.bittorrent.Peer:(3) 5000.023418] [java/INFO] Here is my current status: 1111111111 > [node-3.acme.org:app.bittorrent.Peer:(4) 0.000000] [java/INFO] Hi, I'm joining the network with id 4 > [node-3.acme.org:app.bittorrent.Peer:(4) 5000.023418] [java/INFO] Here is my current status: 1111111111 > [node-4.acme.org:app.bittorrent.Peer:(5) 0.000000] [java/INFO] Hi, I'm joining the network with id 5 > [node-4.acme.org:app.bittorrent.Peer:(5) 5000.007806] [java/INFO] Here is my current status: 1111111111 > [node-5.acme.org:app.bittorrent.Peer:(6) 0.000000] [java/INFO] Hi, I'm joining the network with id 6 > [node-5.acme.org:app.bittorrent.Peer:(6) 5000.023418] [java/INFO] Here is my current status: 1111111111 > [node-6.acme.org:app.bittorrent.Peer:(7) 0.000000] [java/INFO] Hi, I'm joining the network with id 7 > [node-6.acme.org:app.bittorrent.Peer:(7) 5000.046836] [java/INFO] Here is my current status: 1111111111 > [node-7.acme.org:app.bittorrent.Peer:(8) 0.000000] [java/INFO] Hi, I'm joining the network with id 8 > [node-7.acme.org:app.bittorrent.Peer:(8) 5000.031224] [java/INFO] Here is my current status: 1111111111 SimGrid-3.18/examples/java/app/bittorrent/Common.java0000644000175000017500000000330613217757325023133 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; /* Common constants for use in the simulation */ class Common { public static final String TRACKER_MAILBOX = "tracker_mailbox"; public static final int FILE_SIZE = 5120; public static final int FILE_PIECE_SIZE = 512; public static final int FILE_PIECES = 10; public static final int PIECES_BLOCKS = 5; public static final int BLOCKS_REQUESTED = 2; public static final int PIECE_COMM_SIZE = 1; public static final int MESSAGE_SIZE = 1; /* Information message size */ public static final int MAXIMUM_PEERS = 50; /* Max number of peers sent by the tracker to clients */ public static final int TRACKER_QUERY_INTERVAL = 1000; /* Interval of time where the peer should send a request to the tracker */ public static final double TRACKER_COMM_SIZE = 1; /* Communication size for a task to the tracker */ public static final int GET_PEERS_TIMEOUT = 10000; /* Timeout for the get peers data */ public static final int TIMEOUT_MESSAGE = 10; public static final int TRACKER_RECEIVE_TIMEOUT = 10; public static final int MAX_UNCHOKED_PEERS = 4; /* Number of peers that can be unchocked at a given time */ public static final int UPDATE_CHOKED_INTERVAL = 30; /* Interval between each update of the choked peers */ public static final int MAX_PIECES = 1; /* Number of pieces the peer asks for simultaneously */ private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.18/examples/java/app/bittorrent/Tracker.java0000644000175000017500000000477513217757325023311 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.RngStream; import org.simgrid.msg.MsgException; public class Tracker extends Process { protected RngStream stream; protected ArrayList peersList; protected double deadline; protected Comm commReceived = null; public Tracker(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { if (args.length != 1) { Msg.info("Wrong number of arguments for the tracker."); return; } //Build the RngStream object for randomness stream = new RngStream("tracker"); //Retrieve the end time deadline = Double.parseDouble(args[0]); //Building peers array peersList = new ArrayList<>(); Msg.info("Tracker launched."); while (Msg.getClock() < deadline) { if (commReceived == null) { commReceived = Task.irecv(Common.TRACKER_MAILBOX); } try { if (commReceived.test()) { Task task = commReceived.getTask(); if (task instanceof TrackerTask) { TrackerTask tTask = (TrackerTask)task; //Sending peers to the peer int nbPeers = 0; while (nbPeers < Common.MAXIMUM_PEERS && nbPeers < peersList.size()) { int nextPeer; do { nextPeer = stream.randInt(0, peersList.size() - 1); } while (tTask.peers.contains(peersList.get(nextPeer))); tTask.peers.add(peersList.get(nextPeer)); nbPeers++; } //Adding the peer to our list peersList.add(tTask.peerId); tTask.type = TrackerTask.Type.ANSWER; //Setting the interval tTask.interval = Common.TRACKER_QUERY_INTERVAL; //Sending the task back to the peer tTask.dsend(tTask.mailbox); } commReceived = null; } else { waitFor(1); } } catch (MsgException e) { commReceived = null; } } Msg.info("Tracker is leaving"); } } SimGrid-3.18/examples/java/app/masterworker/0000755000175000017500000000000013217757343021367 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/app/masterworker/Main.java0000644000175000017500000000223113217757325023114 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import java.io.File; import org.simgrid.msg.Msg; class Main { public static final int TASK_COMP_SIZE = 10000000; public static final int TASK_COMM_SIZE = 10000000; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { /* initialize the MSG simulation. Must be done before anything else (even logging). */ Msg.init(args); String platf = args.length > 1 ? args[0] : "../platforms/small_platform.xml"; String deploy = args.length > 1 ? args[1] : "app/masterworker/masterworker.xml"; Msg.verb("Platform: "+platf+"; Deployment:"+deploy+"; Current directory: "+new File(".").getAbsolutePath()); /* construct the platform and deploy the application */ Msg.createEnvironment(platf); Msg.deployApplication(deploy); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/app/masterworker/Worker.java0000644000175000017500000000266713217757325023516 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.TaskCancelledException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; import org.simgrid.msg.Process; public class Worker extends Process { public Worker(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { if (args.length < 1) { Msg.info("Worker needs 1 argument (its number)"); System.exit(1); } int num = Integer.parseInt(args[0]); Msg.debug("Receiving on 'worker_"+num+"'"); while(true) { Task task = Task.receive("worker_"+num); if ("finalize".equals(task.getName())) { break; } Msg.info("Received \"" + task.getName() + "\". Processing it (my pid is "+getPID()+")."); try { task.execute(); } catch (TaskCancelledException e) { e.printStackTrace(); } } Msg.info("Received Finalize. I'm done. See you!"); } } SimGrid-3.18/examples/java/app/masterworker/app-masterworker.tesh0000644000175000017500000000334113217757334025560 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} app/masterworker/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/app/masterworker/masterworker.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:app.masterworker.Master@Jacquelin) Hello! My PID is 1. Got 7 workers and 5 tasks to process > [ 0.860026] (2:app.masterworker.Worker@Tremblay) Received "Task_0". Processing it (my pid is 2). > [ 1.752187] (3:app.masterworker.Worker@Fafard) Received "Task_1". Processing it (my pid is 3). > [ 1.757531] (4:app.masterworker.Worker@Bourassa) Received "Task_2". Processing it (my pid is 4). > [ 2.806417] (5:app.masterworker.Worker@Boivin) Received "Task_3". Processing it (my pid is 5). > [ 2.811761] (6:app.masterworker.Worker@Ginette) Received "Task_4". Processing it (my pid is 6). > [ 2.811761] (1:app.masterworker.Master@Jacquelin) All tasks have been dispatched. Let's tell everybody the computation is over. > [ 3.671783] (2:app.masterworker.Worker@Tremblay) Received Finalize. I'm done. See you! > [ 4.563940] (3:app.masterworker.Worker@Fafard) Received Finalize. I'm done. See you! > [ 4.569280] (4:app.masterworker.Worker@Bourassa) Received Finalize. I'm done. See you! > [ 5.618161] (5:app.masterworker.Worker@Boivin) Received Finalize. I'm done. See you! > [ 5.623501] (6:app.masterworker.Worker@Ginette) Received Finalize. I'm done. See you! > [ 5.628842] (7:app.masterworker.Worker@Jupiter) Received Finalize. I'm done. See you! > [ 5.629037] (8:app.masterworker.Worker@Jacquelin) Received Finalize. I'm done. See you! > [ 5.629037] (1:app.masterworker.Master@Jacquelin) Goodbye now! > [ 5.629037] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/app/masterworker/masterworker.xml0000644000175000017500000000250013217757343024633 0ustar mquinsonmquinson SimGrid-3.18/examples/java/app/masterworker/README0000644000175000017500000000054313217757340022246 0ustar mquinsonmquinsonThis is a somehow basic master/workers example. There is 2 kind of processes: * Master: creates some tasks, and dispatches them to its workers * Worker: get tasks from the master and run them At the end of the execution: - the master sends a Task whose name is "finalize" to every known worker to stop them - On reception of such tasks workers stop.SimGrid-3.18/examples/java/app/masterworker/Master.java0000644000175000017500000000322113217757325023463 0ustar mquinsonmquinson/* Master of a basic master/worker example in Java */ /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Task; import org.simgrid.msg.Process; public class Master extends Process { public Master(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { if (args.length < 4) { Msg.info("Master needs 4 arguments"); System.exit(1); } int tasksCount = Integer.parseInt(args[0]); double taskComputeSize = Double.parseDouble(args[1]); double taskCommunicateSize = Double.parseDouble(args[2]); int workersCount = Integer.parseInt(args[3]); Msg.info("Hello! My PID is "+getPID()+". Got "+ workersCount + " workers and "+tasksCount+" tasks to process"); for (int i = 0; i < tasksCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.debug("Sending \"" + task.getName()+ "\" to \"worker_" + i % workersCount + "\""); task.send("worker_"+(i%workersCount)); } Msg.info("All tasks have been dispatched. Let's tell everybody the computation is over."); for (int i = 0; i < workersCount; i++) { Task task = new Task("finalize", 0, 0); task.send("worker_"+(i%workersCount)); } Msg.info("Goodbye now!"); } } SimGrid-3.18/examples/java/app/pingpong/0000755000175000017500000000000013217757334020463 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/app/pingpong/Main.java0000644000175000017500000000214513217757325022214 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import java.io.File; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; class Main { protected static final int TASK_COUNT = 3; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws HostNotFoundException { Msg.init(args); String platfFile = "../../examples/platforms/small_platform.xml"; if (args.length == 1) platfFile = args[0]; File f = new File(platfFile); if (!f.exists()) { Msg.error("File " + platfFile + " does not exist in " + System.getProperty("user.dir")); Msg.error("Usage : Main ../platforms/platform.xml"); } Msg.createEnvironment(platfFile); new Sender("Jacquelin", "Sender", new String[] {"Boivin"}).start(); new Receiver ("Boivin", "Receiver", null).start(); Msg.run(); } } SimGrid-3.18/examples/java/app/pingpong/app-pingpong.tesh0000644000175000017500000000302213217757334023744 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} app/pingpong/Main ${srcdir:=.}/../platforms/small_platform.xml > [0.000000] [java/INFO] Using regular java threads. > [Jacquelin:Sender:(1) 0.000000] [java/INFO] Host count: 1 > [Jacquelin:Sender:(1) 0.000000] [java/INFO] sender time: 0.0 > [Boivin:Receiver:(2) 0.000000] [java/INFO] Wait for a task > [Boivin:Receiver:(2) 1.048882] [java/INFO] Got one that was sent at time 0.0 > [Boivin:Receiver:(2) 1.048882] [java/INFO] Communication time : 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [java/INFO] --- bw 9.533962169004269E7 ---- > [Boivin:Receiver:(2) 1.048882] [java/INFO] Wait for a task > [Jacquelin:Sender:(1) 1.048882] [java/INFO] sender time: 1.0488818628325232 > [Boivin:Receiver:(2) 2.097764] [java/INFO] Got one that was sent at time 1.0488818628325232 > [Boivin:Receiver:(2) 2.097764] [java/INFO] Communication time : 1.0488818628325232 > [Boivin:Receiver:(2) 2.097764] [java/INFO] --- bw 9.533962169004269E7 ---- > [Boivin:Receiver:(2) 2.097764] [java/INFO] Wait for a task > [Jacquelin:Sender:(1) 2.097764] [java/INFO] sender time: 2.0977637256650463 > [Boivin:Receiver:(2) 3.146646] [java/INFO] Got one that was sent at time 2.0977637256650463 > [Boivin:Receiver:(2) 3.146646] [java/INFO] Communication time : 1.0488818628325234 > [Boivin:Receiver:(2) 3.146646] [java/INFO] --- bw 9.533962169004266E7 ---- > [Boivin:Receiver:(2) 3.146646] [java/INFO] Done. > [Jacquelin:Sender:(1) 3.146646] [java/INFO] Done. > [3.146646] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/app/pingpong/Sender.java0000644000175000017500000000221313217757325022544 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; public class Sender extends Process { private static final double COMM_SIZE_LAT = 1; public Sender(String hostname, String name, String[] args) throws HostNotFoundException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("Host count: " + args.length); for (int i = 0 ; i SimGrid-3.18/examples/java/app/centralizedmutex/Main.java0000644000175000017500000000143013217757325023756 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); String platf = args.length > 1 ? args[0] : "../platforms/small_platform.xml"; String deploy = args.length > 1 ? args[1] : "./centralizedmutex.xml"; /* construct the platform and deploy the application */ Msg.createEnvironment(platf); Msg.deployApplication(deploy); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/app/centralizedmutex/Coordinator.java0000644000175000017500000000304013217757325025354 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import java.util.LinkedList; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Coordinator extends Process { public Coordinator(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { int csToServe = Integer.parseInt(args[0]); LinkedList waitingQueue=new LinkedList<>(); while (csToServe >0) { Task task = Task.receive("coordinator"); if (task instanceof RequestTask) { RequestTask t = (RequestTask) task; if (waitingQueue.isEmpty()) { Msg.info("Got a request from "+t.from+". Queue empty: grant it"); GrantTask tosend = new GrantTask(); tosend.send(t.from); } else { waitingQueue.addFirst(t); } } else if (task instanceof ReleaseTask) { if (!waitingQueue.isEmpty()) { RequestTask req = waitingQueue.removeLast(); GrantTask tosend = new GrantTask(); tosend.send(req.from); } csToServe--; if (waitingQueue.isEmpty() && csToServe==0) { Msg.info("we should shutdown the simulation now"); } } } } } SimGrid-3.18/examples/java/app/centralizedmutex/RequestTask.java0000644000175000017500000000065613217757325025356 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Task; public class RequestTask extends Task { protected String from; public RequestTask(String name) { super(); from=name; } } SimGrid-3.18/examples/java/app/centralizedmutex/ReleaseTask.java0000644000175000017500000000050213217757325025274 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; public class ReleaseTask extends org.simgrid.msg.Task {} SimGrid-3.18/examples/java/app/centralizedmutex/Node.java0000644000175000017500000000206313217757325023762 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Node extends Process { public Node(Host host, String name, String[]args) { super(host,name,args); } public void request(double csTime) throws MsgException { RequestTask req = new RequestTask(getName()); Msg.info("Send a request to the coordinator"); req.send("coordinator"); Msg.info("Wait for a grant from the coordinator"); GrantTask.receive(getName()); Task compute = new Task("CS", csTime, 0); compute.execute(); ReleaseTask release = new ReleaseTask(); release.send("coordinator"); } public void main(String[] args) throws MsgException { request(Double.parseDouble(args[1])); } }SimGrid-3.18/examples/java/app/centralizedmutex/GrantTask.java0000644000175000017500000000050013217757325024765 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; public class GrantTask extends org.simgrid.msg.Task {} SimGrid-3.18/examples/java/app/centralizedmutex/app-centralizedmutex.tesh0000644000175000017500000000210713217757334027263 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} app/centralizedmutex/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/app/centralizedmutex/centralizedmutex.xml > [0.000000] [java/INFO] Using regular java threads. > [Jupiter:app.centralizedmutex.Node:(2) 0.000000] [java/INFO] Send a request to the coordinator > [Fafard:app.centralizedmutex.Node:(3) 0.000000] [java/INFO] Send a request to the coordinator > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.019014] [java/INFO] Got a request from app.centralizedmutex.Node. Queue empty: grant it > [Jupiter:app.centralizedmutex.Node:(2) 0.019014] [java/INFO] Wait for a grant from the coordinator > [Fafard:app.centralizedmutex.Node:(3) 0.063737] [java/INFO] Wait for a grant from the coordinator > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.063737] [java/INFO] Got a request from app.centralizedmutex.Node. Queue empty: grant it > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.134167] [java/INFO] we should shutdown the simulation now > [0.134167] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/async/0000755000175000017500000000000013217757325017177 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/async/dsend/0000755000175000017500000000000013217757334020274 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/async/dsend/async-dsend.tesh0000644000175000017500000000333613217757334023376 0ustar mquinsonmquinson#! tesh ! timeout 30 $ java -classpath ${classpath:=.} async/dsend/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Sender@Boivin) Hello! Got 6 receivers to contact > [ 0.000000] (1:Sender@Boivin) Sending "Task_1" to "Bourassa" > [ 0.000000] (2:Receiver@Bourassa) Receiving on 'Bourassa' > [ 0.000000] (3:Receiver@Fafard) Receiving on 'Fafard' > [ 0.000000] (4:Receiver@Ginette) Receiving on 'Ginette' > [ 0.000000] (5:Receiver@Jacquelin) Receiving on 'Jacquelin' > [ 0.000000] (6:Receiver@Jupiter) Receiving on 'Jupiter' > [ 0.000000] (7:Receiver@Tremblay) Receiving on 'Tremblay' > [ 0.000000] (1:Sender@Boivin) Sending "Task_2" to "Fafard" > [ 0.000000] (1:Sender@Boivin) Sending "Task_3" to "Ginette" > [ 0.000000] (1:Sender@Boivin) Sending "Task_4" to "Jacquelin" > [ 0.000000] (1:Sender@Boivin) Sending "Task_5" to "Jupiter" > [ 0.000000] (1:Sender@Boivin) Sending "Task_6" to "Tremblay" > [ 0.000000] (1:Sender@Boivin) All tasks have been (asynchronously) dispatched. Let's sleep for 20s so that nobody gets a message from a terminated process. > [ 1.933362] (6:Receiver@Jupiter) Received a task. I'm done. See you! > [ 1.933362] (4:Receiver@Ginette) Received a task. I'm done. See you! > [ 1.933362] (2:Receiver@Bourassa) Received a task. I'm done. See you! > [ 2.449247] (7:Receiver@Tremblay) Received a task. I'm done. See you! > [ 2.964768] (3:Receiver@Fafard) Received a task. I'm done. See you! > [ 4.162002] (5:Receiver@Jacquelin) Received a task. I'm done. See you! > [ 20.000000] (1:Sender@Boivin) Done sleeping. Goodbye now! > [ 20.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/async/dsend/Main.java0000644000175000017500000000244613217757325022031 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; /** This example demonstrates the use of the Task.dsend() method. * * This way, the sender can be detached from the communication: it is not blocked as with Task.send() * and has nothing to do at the end as with Task.isend() where it must do a Comm.wait(). */ import org.simgrid.msg.Host; import org.simgrid.msg.Msg; class Main { private Main() { /* This is just to ensure that nobody creates an instance of this singleton */ throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); String platform = "../platforms/small_platform.xml"; if (args.length >= 1) { platform = args[0]; // Override the default value if passed on the command line } /* construct the platform and deploy the application */ Msg.createEnvironment(platform); Host[] hosts = Host.all(); new Sender(hosts[0],"Sender").start(); for (int i=1; i < hosts.length; i++){ new Receiver(hosts[i], "Receiver").start(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/async/dsend/Sender.java0000644000175000017500000000250313217757325022357 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Sender extends Process { public Sender (Host host, String name){ super(host,name); } @Override public void main(String[] args) throws MsgException { double taskComputeSize =0; double taskCommunicateSize = 5000000; Host[] hosts = Host.all(); int receiverCount = hosts.length - 1; Msg.info("Hello! Got "+ receiverCount + " receivers to contact"); for (int i = 1; i <= receiverCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.info("Sending \"" + task.getName()+ "\" to \"" + hosts[i].getName() + "\""); task.dsend(hosts[i].getName()); } Msg.info("All tasks have been (asynchronously) dispatched."+ " Let's sleep for 20s so that nobody gets a message from a terminated process."); waitFor(20); Msg.info("Done sleeping. Goodbye now!"); } } SimGrid-3.18/examples/java/async/dsend/Receiver.java0000644000175000017500000000170713217757325022710 0ustar mquinsonmquinson/* Copyright (c) 2006-2007, 2010, 2013-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; public class Receiver extends Process { public Receiver (Host host, String name) { super(host,name); } @Override public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { Msg.info("Receiving on '"+ getHost().getName() + "'"); Task.receive(getHost().getName()); Msg.info("Received a task. I'm done. See you!"); } }SimGrid-3.18/examples/java/async/waitall/0000755000175000017500000000000013217757334020634 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/async/waitall/Main.java0000644000175000017500000000256713217757325022375 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; /** This example demonstrates the use of the asynchrounous communications * * Task.isend() and Task.irecv() are used to start the communications in non-blocking mode. * * The sends are then blocked onto with Comm.waitCompletion(), that locks until the given * communication terminates. * * The receives are packed into an array, and the sender blocks until all of them terminate * with Comm.waitAll(). */ import org.simgrid.msg.Msg; import org.simgrid.msg.Host; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); String platform = "../platforms/small_platform.xml"; if (args.length >= 1) { platform = args[0]; // Override the default value if passed on the command line } /* construct the platform and deploy the application */ Msg.createEnvironment(platform); Host[] hosts = Host.all(); new Sender(hosts[0],"Sender").start(); for (int i=1; i < hosts.length; i++){ new Receiver(hosts[i], "Receiver").start(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/async/waitall/Sender.java0000644000175000017500000000256013217757325022722 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; public class Sender extends Process { public Sender (Host host, String name){ super(host,name); } @Override public void main(String[] args) throws MsgException { double taskComputeSize =0; double taskCommunicateSize = 5000000; Host[] hosts = Host.all(); int receiverCount = hosts.length - 1; Msg.info("I have "+ receiverCount + " receivers to contact"); Comm[] communicators = new Comm[receiverCount]; for (int i = 1; i <= receiverCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.info("Start the Sending '" + task.getName()+ "' to '" + hosts[i].getName() + "'"); communicators[i-1] = task.isend(hosts[i].getName()); } Msg.info("All tasks have been (asynchronously) dispatched. Let's wait for their completion."); Comm.waitAll(communicators); Msg.info("Goodbye now!"); } } SimGrid-3.18/examples/java/async/waitall/async-waitall.tesh0000644000175000017500000000376013217757334024277 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} async/waitall/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Sender@Boivin) I have 6 receivers to contact > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_1' to 'Bourassa' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_2' to 'Fafard' > [ 0.000000] (2:Receiver@Bourassa) I started receiving on 'Bourassa. Wait 0.1 second, and block on the communication. > [ 0.000000] (3:Receiver@Fafard) I started receiving on 'Fafard. Wait 0.1 second, and block on the communication. > [ 0.000000] (4:Receiver@Ginette) I started receiving on 'Ginette. Wait 0.1 second, and block on the communication. > [ 0.000000] (5:Receiver@Jacquelin) I started receiving on 'Jacquelin. Wait 0.1 second, and block on the communication. > [ 0.000000] (6:Receiver@Jupiter) I started receiving on 'Jupiter. Wait 0.1 second, and block on the communication. > [ 0.000000] (7:Receiver@Tremblay) I started receiving on 'Tremblay. Wait 0.1 second, and block on the communication. > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_3' to 'Ginette' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_4' to 'Jacquelin' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_5' to 'Jupiter' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_6' to 'Tremblay' > [ 0.000000] (1:Sender@Boivin) All tasks have been (asynchronously) dispatched. Let's wait for their completion. > [ 1.933362] (6:Receiver@Jupiter) I got my task, good bye. > [ 1.933362] (4:Receiver@Ginette) I got my task, good bye. > [ 1.933362] (2:Receiver@Bourassa) I got my task, good bye. > [ 2.449247] (7:Receiver@Tremblay) I got my task, good bye. > [ 2.964768] (3:Receiver@Fafard) I got my task, good bye. > [ 4.162002] (5:Receiver@Jacquelin) I got my task, good bye. > [ 4.162002] (1:Sender@Boivin) Goodbye now! > [ 4.162002] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/async/waitall/Receiver.java0000644000175000017500000000232713217757325023247 0ustar mquinsonmquinson/* Copyright (c) 2006-2007, 2010, 2013-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; public class Receiver extends Process { public Receiver (Host host, String name) { super(host,name); } @Override public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { Comm comm = Task.irecv(getHost().getName()); Msg.info("I started receiving on '"+ getHost().getName() +". Wait 0.1 second, and block on the communication."); waitFor(0.1); try { comm.waitCompletion(); } catch (TimeoutException e) { Msg.info("Timeout while waiting for my task"); throw e; // Stop this process } Msg.info("I got my task, good bye."); } }SimGrid-3.18/examples/java/async/yield/0000755000175000017500000000000013217757334020305 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/async/yield/Main.java0000644000175000017500000000244213217757325022036 0ustar mquinsonmquinson/* Copyright (c) 2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.yield; /** This example demonstrates the use of the Task.dsend() method. * * This way, the sender can be detached from the communication: it is not blocked as with Task.send() * and has nothing to do at the end as with Task.isend() where it must do a Comm.wait(). */ import org.simgrid.msg.Host; import org.simgrid.msg.Msg; class Main { private Main() { /* This is just to ensure that nobody creates an instance of this singleton */ throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); String platform = "../platforms/small_platform.xml"; if (args.length >= 1) { platform = args[0]; // Override the default value if passed on the command line } /* construct the platform and deploy the application */ Msg.createEnvironment(platform); Host[] hosts = Host.all(); new Yielder(hosts[0],"Yielder", new String[] {"10"}).start(); new Yielder(hosts[1],"Yielder", new String[] {"15"}).start(); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/async/yield/async-yield.tesh0000644000175000017500000000064113217757334023414 0ustar mquinsonmquinson#! tesh ! timeout 30 $ java -classpath ${classpath:=.} async/yield/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Yielder@Boivin) Yielded 10. Good bye now! > [ 0.000000] (2:Yielder@Bourassa) Yielded 15. Good bye now! > [ 0.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/async/yield/Yielder.java0000644000175000017500000000126013217757325022544 0ustar mquinsonmquinson/* Copyright (c) 2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.yield; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; public class Yielder extends Process { public Yielder (Host host, String name, String[] args) { super(host, name, args); } @Override public void main(String[] args) { int yieldsCount = Integer.parseInt(args[0]); for (int i=0; i [0.000000] [java/INFO] Using regular java threads. > [Jacquelin:killer:(1) 0.000000] [java/INFO] Hello! > [Boivin:victim:(2) 0.000000] [java/INFO] Hello! > [Boivin:victim:(2) 0.000000] [java/INFO] Suspending myself > [Jacquelin:killer:(1) 10.000000] [java/INFO] Resume Process > [Boivin:victim:(2) 10.000000] [java/INFO] OK, OK. Let's work > [Jacquelin:killer:(1) 11.000000] [java/INFO] Kill Process > [Jacquelin:killer:(1) 11.000000] [java/INFO] Ok, goodbye now. > [11.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/process/kill/Killer.java0000644000175000017500000000261713217757326022567 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.kill; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; import process.kill.Victim; public class Killer extends Process { public Killer(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { Victim poorVictim = null; Msg.info("Hello!"); try { poorVictim = new Victim("Boivin","victim"); poorVictim.start(); } catch (MsgException e){ e.printStackTrace(); Msg.error("Cannot create the victim process!"); return; } sleep(10000); Msg.info("Resume Process"); poorVictim.resume(); sleep(1000); Msg.info("Kill Process"); poorVictim.kill(); Msg.info("Ok, goodbye now."); // The actor can also commit a suicide with the following command exit(); // This will forcefully stop the current actor // Of course, it's not useful here at the end of the main function, but that's for the example (and to check that this still works in the automated tests) } } SimGrid-3.18/examples/java/process/startkilltime/0000755000175000017500000000000013217757343022430 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/process/startkilltime/Main.java0000644000175000017500000000161613217757326024164 0ustar mquinsonmquinson/* Copyright (c) 2006-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.startkilltime; import org.simgrid.msg.Msg; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 2) { Msg.info("Usage : StartKilltime platform_file deployment_file"); Msg.info("example : StartKilltime ../platforms/platform.xml deployment_start_kill.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/process/startkilltime/Sleeper.java0000644000175000017500000000150013217757326024667 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.startkilltime; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Sleeper extends Process { public Sleeper(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { Msg.info("Hello! I go to sleep."); try { waitFor(Integer.parseInt(args[0])); Msg.info("Done sleeping"); } catch (MsgException e) { Msg.debug("Wait cancelled."); } } } SimGrid-3.18/examples/java/process/startkilltime/startkilltime.xml0000644000175000017500000000173113217757343026044 0ustar mquinsonmquinson SimGrid-3.18/examples/java/process/startkilltime/process-startkilltime.tesh0000644000175000017500000000203013217757334027654 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} process/startkilltime/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/process/startkilltime/startkilltime.xml > [0.000000] [java/INFO] Using regular java threads. > [node-0.acme.org:process.startkilltime.Sleeper:(1) 0.000000] [java/INFO] Hello! I go to sleep. > [node-1.acme.org:process.startkilltime.Sleeper:(2) 1.000000] [java/INFO] Hello! I go to sleep. > [node-2.acme.org:process.startkilltime.Sleeper:(3) 2.000000] [java/INFO] Hello! I go to sleep. > [node-3.acme.org:process.startkilltime.Sleeper:(4) 3.000000] [java/INFO] Hello! I go to sleep. > [node-4.acme.org:process.startkilltime.Sleeper:(5) 4.000000] [java/INFO] Hello! I go to sleep. > [node-5.acme.org:process.startkilltime.Sleeper:(6) 5.000000] [java/INFO] Hello! I go to sleep. > [node-2.acme.org:process.startkilltime.Sleeper:(3) 6.000000] [java/INFO] Done sleeping > [node-3.acme.org:process.startkilltime.Sleeper:(4) 7.000000] [java/INFO] Done sleeping > [10.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/process/suspend/0000755000175000017500000000000013217757334021221 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/process/suspend/Main.java0000644000175000017500000000203613217757326022752 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Suspend platform_file"); Msg.info("example : Suspend ../platforms/platform.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); try { DreamMaster process1 = new DreamMaster("Jacquelin","DreamMaster"); process1.start(); } catch (MsgException e){ Msg.error("Create processes failed!"); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/process/suspend/LazyGuy.java0000644000175000017500000000142113217757326023467 0ustar mquinsonmquinson/* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class LazyGuy extends Process { public LazyGuy(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { Msg.info("Nobody's watching me ? Let's go to sleep."); suspend(); Msg.info("Uuuh ? Did somebody call me ?"); Msg.info("Mmmh, goodbye now."); } }SimGrid-3.18/examples/java/process/suspend/DreamMaster.java0000644000175000017500000000171313217757326024273 0ustar mquinsonmquinson/* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class DreamMaster extends Process { public DreamMaster(String hostname, String name) throws HostNotFoundException{ super(hostname,name); } public void main(String[] args) throws MsgException { Msg.info("Let's create a lazy guy."); Process lazyGuy = new LazyGuy(getHost(),"Lazy",null); lazyGuy.start(); Msg.info("Let's wait a little bit..."); waitFor(10); Msg.info("Let's wake the lazy guy up! >:) BOOOOOUUUHHH!!!!"); lazyGuy.resume(); Msg.info("OK, goodbye now."); } }SimGrid-3.18/examples/java/process/suspend/process-suspend.tesh0000644000175000017500000000141213217757334025241 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} process/suspend/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:DreamMaster@Jacquelin) Let's create a lazy guy. > [ 0.000000] (1:DreamMaster@Jacquelin) Let's wait a little bit... > [ 0.000000] (2:Lazy@Jacquelin) Nobody's watching me ? Let's go to sleep. > [ 10.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 10.000000] (1:DreamMaster@Jacquelin) Let's wake the lazy guy up! >:) BOOOOOUUUHHH!!!! > [ 10.000000] (1:DreamMaster@Jacquelin) OK, goodbye now. > [ 10.000000] (2:Lazy@Jacquelin) Uuuh ? Did somebody call me ? > [ 10.000000] (2:Lazy@Jacquelin) Mmmh, goodbye now. SimGrid-3.18/examples/java/process/migration/0000755000175000017500000000000013217757334021531 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/process/migration/Main.java0000644000175000017500000000254413217757326023266 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.Mutex; import org.simgrid.msg.Process; class Main { protected static Mutex mutex; protected static Process processToMigrate = null; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Migration platform_file"); Msg.info("example : Migration ../platforms/platform.xml"); System.exit(1); } /* Create the mutex */ mutex = new Mutex(); /* construct the platform*/ Msg.createEnvironment(args[0]); /* bypass deploymemt */ try { Policeman policeman = new Policeman("Boivin","policeman"); policeman.start(); Emigrant emigrant = new Emigrant("Jacquelin","emigrant"); emigrant.start(); } catch (HostNotFoundException e){ Msg.error("Create processes failed!"); e.printStackTrace(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.18/examples/java/process/migration/Policeman.java0000644000175000017500000000166013217757326024307 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class Policeman extends Process { public Policeman(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { waitFor(1); Msg.info("Wait a bit before migrating the emigrant."); Main.mutex.acquire(); Main.processToMigrate.migrate(Host.getByName("Jacquelin")); Msg.info("I moved the emigrant"); Main.processToMigrate.resume(); } }SimGrid-3.18/examples/java/process/migration/Emigrant.java0000644000175000017500000000252013217757326024142 0ustar mquinsonmquinson/* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class Emigrant extends Process { public Emigrant(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { Main.mutex.acquire(); Msg.info("I'll look for a new job on another machine where the grass is greener."); migrate(Host.getByName("Boivin")); Msg.info("Yeah, found something to do"); Task task = new Task("job", 98095000, 0); task.execute(); waitFor(2); Msg.info("Moving back to home after work"); migrate(Host.getByName("Jacquelin")); migrate(Host.getByName("Boivin")); waitFor(4); Main.processToMigrate = this; Main.mutex.release(); suspend(); Msg.info("I've been moved on this new host:" + getHost().getName()); Msg.info("Uh, nothing to do here. Stopping now"); } }SimGrid-3.18/examples/java/process/migration/process-migration.tesh0000644000175000017500000000150313217757334026062 0ustar mquinsonmquinson#! tesh ! output sort 19 $ java -classpath ${classpath:=.} process/migration/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (2:emigrant@Jacquelin) I'll look for a new job on another machine where the grass is greener. > [ 0.000000] (2:emigrant@Boivin) Yeah, found something to do > [ 1.000000] (1:policeman@Boivin) Wait a bit before migrating the emigrant. > [ 3.000000] (2:emigrant@Boivin) Moving back to home after work > [ 7.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 7.000000] (2:emigrant@Jacquelin) I've been moved on this new host:Jacquelin > [ 7.000000] (2:emigrant@Jacquelin) Uh, nothing to do here. Stopping now > [ 7.000000] (1:policeman@Boivin) I moved the emigrant SimGrid-3.18/examples/java/io/0000755000175000017500000000000013217757326016472 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/io/storage/0000755000175000017500000000000013217757334020135 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/io/storage/Main.java0000644000175000017500000000145613217757326021673 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package io.storage; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); Msg.fileSystemInit(); if(args.length < 1) { Msg.info("Usage : Storage platform_file "); Msg.info("example : Storage ../platforms/storage/storage.xml "); System.exit(1); } Msg.createEnvironment(args[0]); Host[] hosts = Host.all(); new io.storage.Client(hosts[3],0).start(); Msg.run(); } } SimGrid-3.18/examples/java/io/storage/Client.java0000644000175000017500000000461113217757326022221 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /********************* Files and Storage handling **************************** * This example implements some storage related functions of the MSG API * * Scenario : * - display information on the disks mounted by the current host * - attach some properties to a disk * - list all the storage elements in the platform * ******************************************************************************/ package io.storage; import java.util.Arrays; import java.util.Comparator; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.Storage; import org.simgrid.msg.MsgException; public class Client extends Process { public Client(Host host, int number) { super(host, Integer.toString(number), null); } public void main(String[] args) throws MsgException { // Retrieve all mount points of current host Storage[] storages = getHost().getMountedStorage(); Arrays.sort(storages, new Comparator() { public int compare(Storage a, Storage b) { return a.getName().compareTo(b.getName()); } }); for (int i = 0; i < storages.length; i++) { // For each disk mounted on host Msg.info("------------------------------------"); Msg.info("Disk name: "+storages[i].getName()); Msg.info("Size: "+storages[i].getSize()+" bytes."); Msg.info("Free Size: "+storages[i].getFreeSize()+" bytes."); Msg.info("Used Size: "+storages[i].getUsedSize()+" bytes."); } Storage st = Storage.getByName("Disk2"); Msg.info("Disk name: "+st.getName()); Msg.info("Attached to host:"+st.getHost()); st.setProperty("key","Pierre"); Msg.info("Property key: "+st.getProperty("key")); Host h = Host.currentHost(); h.setProperty("key2","Pierre"); Msg.info("Property key2: "+h.getProperty("key2")); String[] attach = h.getAttachedStorage(); for (int j = 0; j < attach.length; j++) { Msg.info("Disk attached: "+attach[j]); } Msg.info("**************** ALL *************************"); Storage[] stos = Storage.all(); for (int i = 0; i < stos.length; i++) { Msg.info("Disk: "+ stos[i].getName()); } } } SimGrid-3.18/examples/java/io/storage/io-storage.tesh0000644000175000017500000000267513217757334023105 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} io/storage/Main ${srcdir:=.}/../platforms/storage/storage.xml > [0.000000] [java/INFO] Using regular java threads. > [denise:0:(1) 0.000000] [java/INFO] ------------------------------------ > [denise:0:(1) 0.000000] [java/INFO] Disk name: Disk2 > [denise:0:(1) 0.000000] [java/INFO] Size: 536870912000 bytes. > [denise:0:(1) 0.000000] [java/INFO] Free Size: 534479374867 bytes. > [denise:0:(1) 0.000000] [java/INFO] Used Size: 2391537133 bytes. > [denise:0:(1) 0.000000] [java/INFO] ------------------------------------ > [denise:0:(1) 0.000000] [java/INFO] Disk name: Disk4 > [denise:0:(1) 0.000000] [java/INFO] Size: 536870912000 bytes. > [denise:0:(1) 0.000000] [java/INFO] Free Size: 536857690006 bytes. > [denise:0:(1) 0.000000] [java/INFO] Used Size: 13221994 bytes. > [denise:0:(1) 0.000000] [java/INFO] Disk name: Disk2 > [denise:0:(1) 0.000000] [java/INFO] Attached to host:alice > [denise:0:(1) 0.000000] [java/INFO] Property key: Pierre > [denise:0:(1) 0.000000] [java/INFO] Property key2: Pierre > [denise:0:(1) 0.000000] [java/INFO] Disk attached: Disk4 > [denise:0:(1) 0.000000] [java/INFO] **************** ALL ************************* > [denise:0:(1) 0.000000] [java/INFO] Disk: Disk1 > [denise:0:(1) 0.000000] [java/INFO] Disk: Disk2 > [denise:0:(1) 0.000000] [java/INFO] Disk: Disk3 > [denise:0:(1) 0.000000] [java/INFO] Disk: Disk4 > [0.000000] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/io/file/0000755000175000017500000000000013217757334017410 5ustar mquinsonmquinsonSimGrid-3.18/examples/java/io/file/Main.java0000644000175000017500000000161213217757326021140 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package io.file; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); Msg.fileSystemInit(); if(args.length < 1) { Msg.info("Usage : IO platform_file "); Msg.info("example : IO ../platforms/storage/storage.xml "); System.exit(1); } Msg.createEnvironment(args[0]); Host[] hosts = Host.all(); Msg.info("Number of hosts:" + hosts.length); for (int i = 0; i < hosts.length && i < 4; i++) { new io.file.Node(hosts[i],i).start(); } Msg.run(); } }SimGrid-3.18/examples/java/io/file/io-file.tesh0000644000175000017500000000434313217757334021625 0ustar mquinsonmquinson#! tesh $ java -classpath ${classpath:=.} io/file/Main ${srcdir:=.}/../platforms/storage/storage.xml > [0.000000] [java/INFO] Using regular java threads. > [0.000000] [java/INFO] Number of hosts:4 > [alice:0:(1) 0.000000] [java/INFO] Open file c:\Windows\setupact.log > [bob:1:(2) 0.000000] [java/INFO] Open file /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.000000] [java/INFO] Open file /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [denise:3:(4) 0.000000] [java/INFO] Open file /home/doc/simgrid/examples/platforms/g5k.xml > [bob:1:(2) 0.000040] [java/INFO] Having read 4028 on /home/doc/simgrid/examples/platforms/nancy.xml > [alice:0:(1) 0.000050] [java/INFO] Having read 10000 on c:\Windows\setupact.log > [denise:3:(4) 0.000050] [java/INFO] Having read 10000 on /home/doc/simgrid/examples/platforms/g5k.xml > [carl:2:(3) 0.000100] [java/INFO] Having read 10000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [alice:0:(1) 0.001717] [java/INFO] Having write 100000 on c:\Windows\setupact.log > [alice:0:(1) 0.001717] [java/INFO] Seek back to the beginning of c:\Windows\setupact.log > [denise:3:(4) 0.001717] [java/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/g5k.xml > [denise:3:(4) 0.001717] [java/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/g5k.xml > [alice:0:(1) 0.002267] [java/INFO] Having read 110000 on c:\Windows\setupact.log > [denise:3:(4) 0.002267] [java/INFO] Having read 110000 on /home/doc/simgrid/examples/platforms/g5k.xml > [bob:1:(2) 0.003374] [java/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/nancy.xml > [bob:1:(2) 0.003374] [java/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.003433] [java/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [carl:2:(3) 0.003433] [java/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [bob:1:(2) 0.004414] [java/INFO] Having read 104028 on /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.004533] [java/INFO] Having read 110000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [0.004533] [java/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.18/examples/java/io/file/Node.java0000644000175000017500000000430613217757326021144 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /********************* Files and Storage handling **************************** * This example implements all main file functions of the MSG API * * Scenario: Each host * - opens a file * - tries to read 10,000 bytes in this file * - writes 100,000 bytes in the file * - seeks back to the beginning of the file * - tries to read 150,000 bytes in this file * ******************************************************************************/ package io.file; import org.simgrid.msg.Msg; import org.simgrid.msg.File; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Node extends Process { private static String file1 = "/doc/simgrid/examples/platforms/g5k.xml"; private static String file2 = "\\Windows\\setupact.log"; private static String file3 = "/doc/simgrid/examples/platforms/g5k_cabinets.xml"; private static String file4 = "/doc/simgrid/examples/platforms/nancy.xml"; protected int rank; public Node(Host host, int number) { super(host, Integer.toString(number), null); this.rank = number; } public void main(String[] args) throws MsgException { String mount = "/home"; String fileName; switch (rank) { case 4: fileName = mount + file1; break; case 0: mount = "c:"; fileName = mount + file2; break; case 2: fileName = mount + file3; break; case 1: fileName = mount + file4; break; default: fileName = mount + file1; break; } Msg.info("Open file " + fileName); File file = new File(fileName); long read = file.read(10000,1); Msg.info("Having read " + read + " on " + fileName); long write = file.write(100000,1); Msg.info("Having write " + write + " on " + fileName); Msg.info("Seek back to the beginning of " + fileName); file.seek(0,File.SEEK_SET); read = file.read(150000,1); Msg.info("Having read " + read + " on " + fileName); } } SimGrid-3.18/examples/simdag/0000755000175000017500000000000013217757327016407 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/schedule-dotload/0000755000175000017500000000000013217757340021622 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/schedule-dotload/sd_schedule-dotload.tesh0000644000175000017500000003160513217757335026426 0ustar mquinsonmquinson#! ./tesh p Test the loader of DAG written in the DOT format ! expect return 2 $ $SG_TEST_EXENV ${bindir:=.}/schedule-dotload/sd_schedule-dotload --log=no_loc "--log=sd_dotparse.thres:verbose" ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/schedule-dotload/dag_with_bad_schedule.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task 'end' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task '1' can not be scheduled on 0 hosts > [0.000000] [sd_dotparse/VERBOSE] Task '0' wants to start on performer '1' at the same position '1' as task '2' > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task '3' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task 'root' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/WARNING] The scheduling is ignored > [0.000000] [test/CRITICAL] The dot file with the provided scheduling is wrong, more information with the option : --log=sd_dotparse.thres:verbose # The order differ when executed with gcc's thread sanitizer ! output sort $ $SG_TEST_EXENV ${bindir:=.}/schedule-dotload/sd_schedule-dotload --log=no_loc ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] Displaying task 0 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] 0->2 > [0.000000] [sd_task/INFO] Displaying task 1 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000131 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] Displaying task 2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000121 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] 0->2 > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] Displaying task 3 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000231 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 4 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] Displaying task 5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000046 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] Displaying task 6 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000092 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] Displaying task 7 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000041 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [sd_task/INFO] Displaying task 8 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000250 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] Displaying task 9 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000079 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 0->2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10001 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 1->2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10004 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 2->3 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10002 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] Displaying task 4->5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10029 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 6->7 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] Displaying task 7->8 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 7->end > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [100.003561] [test/INFO] ------------------- Produce the trace file--------------------------- > [100.003561] [test/INFO] Producing the trace of the run into dag_with_good_schedule.trace $ cat ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.trace > [0.000000->0.000000] node-0.acme.org compute 0.000000 flops # root > [0.000000->10.000000] node-1.acme.org compute 10000000129.452715 flops # 0 > [0.000000->10.000000] node-0.acme.org compute 10000000131.133657 flops # 1 > [10.000760->20.000760] node-1.acme.org compute 10000000121.124870 flops # 2 > [20.001440->30.001441] node-1.acme.org compute 10000000230.608025 flops # 3 > [30.001441->40.001441] node-0.acme.org compute 10000000004.994019 flops # 4 > [40.002121->50.002121] node-0.acme.org compute 10000000046.016401 flops # 5 > [50.002121->60.002121] node-0.acme.org compute 10000000091.598791 flops # 6 > [60.002801->70.002801] node-0.acme.org compute 10000000040.679438 flops # 7 > [70.003561->80.003561] node-1.acme.org compute 10000000250.490017 flops # 8 > [80.003561->90.003561] node-1.acme.org compute 10000000079.267649 flops # 9 > [0.000000 -> 0.080712] node-0.acme.org -> node-0.acme.org transfer of 10014000 bytes # root->5 > [10.000000 -> 10.000760] node-1.acme.org -> node-1.acme.org transfer of 10001 bytes # 0->2 > [10.000000 -> 10.000760] node-0.acme.org -> node-1.acme.org transfer of 10004 bytes # 1->2 > [20.000760 -> 20.001440] node-1.acme.org -> node-1.acme.org transfer of 10002 bytes # 2->3 > [40.001441 -> 40.002121] node-0.acme.org -> node-0.acme.org transfer of 10029 bytes # 4->5 > [60.002121 -> 60.002801] node-0.acme.org -> node-0.acme.org transfer of 10005 bytes # 6->7 > [70.002801 -> 70.003561] node-0.acme.org -> node-1.acme.org transfer of 10000 bytes # 7->8 > [70.002801 -> 70.083593] node-0.acme.org -> node-0.acme.org transfer of 10014000 bytes # 7->end > [90.003561->100.003561] node-0.acme.org compute 10000000129.452715 flops # end $ rm -f ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.trace SimGrid-3.18/examples/simdag/schedule-dotload/dag_with_bad_schedule.dot0000644000175000017500000000206713217757340026607 0ustar mquinsonmquinsondigraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA" performer="1" order="1"]; 1 [size="10000000131.133657" category="taskA" performer="0"]; 2 [size="10000000121.12487" category="taskA" performer="1" order="1"]; 3 [size="10000000230.608025" category="taskA" order="1"]; 4 [size="10000000004.994019" category="taskB" performer="1" order="2"]; 5 [size="10000000046.016401" category="taskB" performer="1" order="3"]; 6 [size="10000000091.598791" category="taskB" performer="1" order="4"]; 7 [size="10000000040.679438" category="taskB" performer="1" order="5"]; 8 [size="10000000250.490017" category="taskC" performer="1" order="6"]; 9 [size="10000000079.267649" category="taskC" performer="1" order="9"]; 0->1 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.18/examples/simdag/schedule-dotload/dag_with_good_schedule.dot0000644000175000017500000000220713217757340027005 0ustar mquinsonmquinsondigraph G { end [size="10000000129.452715" performer="0" order="7"]; root [performer="0" order="1"]; 0 [size="10000000129.452715" category="taskA" performer="1" order="1"]; 1 [size="10000000131.133657" category="taskA" performer="0" order="2"]; 2 [size="10000000121.12487" category="taskA" performer="1" order="2"]; 3 [size="10000000230.608025" category="taskA" performer="1" order="3"]; 4 [size="10000000004.994019" category="taskB" performer="0" order="3"]; 5 [size="10000000046.016401" category="taskB" performer="0" order="4"]; 6 [size="10000000091.598791" category="taskB" performer="0" order="5"]; 7 [size="10000000040.679438" category="taskB" performer="0" order="6"]; 8 [size="10000000250.490017" category="taskC" performer="1" order="4"]; 9 [size="10000000079.267649" category="taskC" performer="1" order="5"]; 0->2 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.18/examples/simdag/schedule-dotload/sd_schedule-dotload.c0000644000175000017500000000557213217757327025712 0ustar mquinsonmquinson/* simple test trying to load a DOT file. */ /* Copyright (c) 2010-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "simgrid/simdag.h" #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dot_file [trace_file]" "example: %s ../2clusters.xml dag.dot dag.mytrace", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file and schedule tasks */ dot = SD_dotload_with_sched(argv[2]); if(!dot){ XBT_CRITICAL("The dot file with the provided scheduling is wrong," " more information with the option : --log=sd_dotparse.thres:verbose"); exit(2); } char *tracefilename; char *last = strrchr(argv[2], '.'); tracefilename = bprintf("%.*s.trace", (int) (last == NULL ? strlen(argv[2]) : last - argv[2]),argv[2]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing the trace of the run into %s", basename(tracefilename)); FILE *out = fopen(tracefilename, "w"); xbt_assert(out, "Cannot write to %s", tracefilename); free(tracefilename); xbt_dynar_foreach(dot, cursor, task) { int kind = SD_task_get_kind(task); sg_host_t *wsl = SD_task_get_workstation_list(task); switch (kind) { case SD_TASK_COMP_SEQ: fprintf(out, "[%f->%f] %s compute %f flops # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), SD_task_get_amount(task), SD_task_get_name(task)); break; case SD_TASK_COMM_E2E: fprintf(out, "[%f -> %f] %s -> %s transfer of %.0f bytes # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), sg_host_get_name(wsl[1]), SD_task_get_amount(task), SD_task_get_name(task)); break; default: xbt_die("Task %s is of unknown kind %u", SD_task_get_name(task), SD_task_get_kind(task)); } SD_task_destroy(task); } fclose(out); xbt_dynar_free_container(&dot); return 0; } SimGrid-3.18/examples/simdag/CMakeLists.txt0000644000175000017500000000620213217757323021143 0ustar mquinsonmquinsonforeach(x availability daxload fail typed_tasks properties throttling scheduling) add_executable (sd_${x} ${x}/sd_${x}.c) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.c) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() foreach(x test) add_executable (sd_${x} ${x}/sd_${x}.cpp) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.cpp) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() foreach(x dag-dotload ptg-dotload schedule-dotload) if(HAVE_GRAPHVIZ) add_executable (sd_${x} ${x}/sd_${x}.c) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) endif() set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.c) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() add_executable (goal_test goal/goal_test.c) target_link_libraries(goal_test simgrid) set_target_properties(goal_test PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/goal) set(tesh_files ${tesh_files} PARENT_SCOPE) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/goal/goal_test.c PARENT_SCOPE) set(xml_files ${xml_files} ${CMAKE_CURRENT_SOURCE_DIR}/scheduling/Montage_25.xml ${CMAKE_CURRENT_SOURCE_DIR}/daxload/simple_dax_with_cycle.xml ${CMAKE_CURRENT_SOURCE_DIR}/daxload/smalldax.xml PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/dag-dotload/dag_with_cycle.dot ${CMAKE_CURRENT_SOURCE_DIR}/dag-dotload/dag.dot ${CMAKE_CURRENT_SOURCE_DIR}/ptg-dotload/ptg.dot ${CMAKE_CURRENT_SOURCE_DIR}/schedule-dotload/dag_with_bad_schedule.dot ${CMAKE_CURRENT_SOURCE_DIR}/schedule-dotload/dag_with_good_schedule.dot ${CMAKE_CURRENT_SOURCE_DIR}/scheduling/expected_output.jed PARENT_SCOPE) foreach(x availability daxload fail typed_tasks properties throttling scheduling test) ADD_TESH(simdag-${x} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/simdag --cd ${CMAKE_BINARY_DIR}/examples/simdag ${CMAKE_HOME_DIRECTORY}/examples/simdag/${x}/sd_${x}.tesh) endforeach() if(HAVE_GRAPHVIZ) foreach(x dag-dotload ptg-dotload schedule-dotload) ADD_TESH(simdag-${x} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/simdag --cd ${CMAKE_BINARY_DIR}/examples/simdag ${CMAKE_HOME_DIRECTORY}/examples/simdag/${x}/sd_${x}.tesh) endforeach() endif() SimGrid-3.18/examples/simdag/dag-dotload/0000755000175000017500000000000013217757340020561 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/dag-dotload/sd_dag-dotload.tesh0000644000175000017500000003044713217757335024327 0ustar mquinsonmquinson#! ./tesh p Test the loader of DAG written in the DOT format # The order differ when executed with gcc's thread sanitizer ! output sort $ $SG_TEST_EXENV ${bindir:=.}/dag-dotload/sd_dag-dotload --log=no_loc ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dag-dotload/dag.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: schedulable not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] Displaying task 0 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0->1 > [0.000000] [sd_task/INFO] Displaying task 1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000131 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0->1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] Displaying task 2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000121 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] Displaying task 3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000231 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] Displaying task 4 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] Displaying task 5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000046 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] Displaying task 6 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000092 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] Displaying task 7 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000041 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [sd_task/INFO] Displaying task 8 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000250 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] Displaying task 9 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000079 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task 0->1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10001 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] Displaying task 1->2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10004 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 2->3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10002 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] Displaying task 4->5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10029 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 6->7 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] Displaying task 7->8 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 7->end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [test/INFO] ------------------- Schedule tasks --------------------------- > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [110.004162] [test/INFO] ------------------- Produce the trace file--------------------------- > [110.004162] [test/INFO] Producing the trace of the run into dag.trace $ cat ${srcdir:=.}/dag-dotload/dag.trace > [0.000000->0.000000] node-0.acme.org compute 0.000000 flops # root > [0.000000->10.000000] node-1.acme.org compute 10000000129.452715 flops # 0 > [10.000680->20.000680] node-10.acme.org compute 10000000131.133657 flops # 1 > [20.001360->30.001360] node-11.acme.org compute 10000000121.124870 flops # 2 > [30.002040->40.002041] node-12.acme.org compute 10000000230.608025 flops # 3 > [40.002041->50.002041] node-13.acme.org compute 10000000004.994019 flops # 4 > [50.002721->60.002721] node-14.acme.org compute 10000000046.016401 flops # 5 > [60.002721->70.002721] node-15.acme.org compute 10000000091.598791 flops # 6 > [70.003401->80.003401] node-16.acme.org compute 10000000040.679438 flops # 7 > [80.004161->90.004161] node-17.acme.org compute 10000000250.490017 flops # 8 > [90.004161->100.004161] node-18.acme.org compute 10000000079.267649 flops # 9 > [10.000000 -> 10.000680] node-1.acme.org -> node-10.acme.org transfer of 10001 bytes # 0->1 > [20.000680 -> 20.001360] node-10.acme.org -> node-11.acme.org transfer of 10004 bytes # 1->2 > [30.001360 -> 30.002040] node-11.acme.org -> node-12.acme.org transfer of 10002 bytes # 2->3 > [50.002041 -> 50.002721] node-13.acme.org -> node-14.acme.org transfer of 10029 bytes # 4->5 > [70.002721 -> 70.003401] node-15.acme.org -> node-16.acme.org transfer of 10005 bytes # 6->7 > [80.003401 -> 80.004161] node-16.acme.org -> node-17.acme.org transfer of 10000 bytes # 7->8 > [80.003401 -> 80.084193] node-16.acme.org -> node-0.acme.org transfer of 10014000 bytes # 7->end > [0.000000 -> 0.080712] node-0.acme.org -> node-14.acme.org transfer of 10014000 bytes # root->5 > [100.004161->110.004162] node-0.acme.org compute 10000000129.452715 flops # end $ rm -f ${srcdir:=.}/dag-dotload/dag.trace ${srcdir:=.}/dot.dot ! expect return 2 $ $SG_TEST_EXENV ${bindir:=.}/dag-dotload/sd_dag-dotload --log=no_loc ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dag-dotload/dag_with_cycle.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_daxparse/WARNING] the task root is not marked > [0.000000] [sd_daxparse/WARNING] the task 1 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 2 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 3 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 4 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 5 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 6 is in a cycle > [0.000000] [sd_dotparse/ERROR] The DOT described in dag_with_cycle.dot is not a DAG. It contains a cycle. > [0.000000] [test/CRITICAL] No dot loaded. Do you have a cycle in your graph? SimGrid-3.18/examples/simdag/dag-dotload/sd_dag-dotload.c0000644000175000017500000000670713217757327023611 0ustar mquinsonmquinson/* simple test trying to load a DOT file. */ /* Copyright (c) 2010-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/log.h" #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dot_file [trace_file]" "example: %s ../2clusters.xml dag.dot dag.mytrace", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file */ dot = SD_dotload(argv[2]); if(dot == NULL){ XBT_CRITICAL("No dot loaded. Do you have a cycle in your graph?"); exit(2); } char *tracefilename; char *last = strrchr(argv[2], '.'); tracefilename = bprintf("%.*s.trace", (int) (last == NULL ? strlen(argv[2]) : last - argv[2]),argv[2]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } FILE *dotout = fopen("dot.dot", "w"); fprintf(dotout, "digraph A {\n"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dotty(task, dotout); } fprintf(dotout, "}\n"); fclose(dotout); /* Schedule them all on the first workstation */ XBT_INFO("------------------- Schedule tasks ---------------------------"); sg_host_t *hosts = sg_host_list(); int count = sg_host_count(); xbt_dynar_foreach(dot, cursor, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_SEQ) { if (!strcmp(SD_task_get_name(task), "end")) SD_task_schedulel(task, 1, hosts[0]); else SD_task_schedulel(task, 1, hosts[cursor % count]); } } xbt_free(hosts); XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing the trace of the run into %s", basename(tracefilename)); FILE *out = fopen(tracefilename, "w"); xbt_assert(out, "Cannot write to %s", tracefilename); free(tracefilename); xbt_dynar_foreach(dot, cursor, task) { int kind = SD_task_get_kind(task); sg_host_t *wsl = SD_task_get_workstation_list(task); switch (kind) { case SD_TASK_COMP_SEQ: fprintf(out, "[%f->%f] %s compute %f flops # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), SD_task_get_amount(task), SD_task_get_name(task)); break; case SD_TASK_COMM_E2E: fprintf(out, "[%f -> %f] %s -> %s transfer of %.0f bytes # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), sg_host_get_name(wsl[1]), SD_task_get_amount(task), SD_task_get_name(task)); break; default: xbt_die("Task %s is of unknown kind %u", SD_task_get_name(task), SD_task_get_kind(task)); } SD_task_destroy(task); } xbt_dynar_free_container(&dot); fclose(out); /* exit */ return 0; } SimGrid-3.18/examples/simdag/dag-dotload/dag.dot0000644000175000017500000000153013217757340022023 0ustar mquinsonmquinsondigraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA"]; 1 [size="10000000131.133657" category="taskA"]; 2 [size="10000000121.12487" category="taskA"]; 3 [size="10000000230.608025" category="taskA"]; 4 [size="10000000004.994019" category="taskB"]; 5 [size="10000000046.016401" category="taskB"]; 6 [size="10000000091.598791" category="taskB"]; 7 [size="10000000040.679438" category="taskB"]; 8 [size="10000000250.490017" category="taskC"]; 9 [size="10000000079.267649" category="taskC"]; 0->1 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.18/examples/simdag/dag-dotload/dag_with_cycle.dot0000644000175000017500000000153613217757340024243 0ustar mquinsonmquinsondigraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA"]; 1 [size="10000000131.133657" category="taskA"]; 2 [size="10000000121.12487" category="taskA"]; 3 [size="10000000230.608025" category="taskA"]; 4 [size="10000000004.994019" category="taskB"]; 5 [size="10000000046.016401" category="taskB"]; 6 [size="10000000091.598791" category="taskB"]; 7 [size="10000000040.679438" category="taskB"]; 8 [size="10000000250.490017" category="taskC"]; 9 [size="10000000079.267649" category="taskC"]; 0->2 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 6->1; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.18/examples/simdag/properties/0000755000175000017500000000000013217757335020602 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/properties/sd_properties.c0000644000175000017500000000470413217757327023636 0ustar mquinsonmquinson/* Copyright (c) 2007-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Property test"); int main(int argc, char **argv) { xbt_dict_cursor_t cursor = NULL; char *key; char *data; char noexist[] = "NoProp"; const char *value; char exist[] = "Hdd"; /* SD initialization */ SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\tExample: %s ../../platforms/prop.xml", argv[0], argv[0]); SD_create_environment(argv[1]); /* init of platform elements */ sg_host_t h1 = sg_host_by_name("host1"); sg_host_t h2 = sg_host_by_name("host2"); const char *name1 = sg_host_get_name(h1); const char *name2 = sg_host_get_name(h2); /* Trying to set a new property */ sg_host_set_property_value(h1, "NewProp", "newValue"); /* Get the property list of 'host1'. This is only a copy of the internal data structure.*/ XBT_INFO("Property list for host %s", name1); xbt_dict_t props = sg_host_get_properties(h1); /* Print the properties of 'host1' */ xbt_dict_foreach (props, cursor, key, data) XBT_INFO("\tProperty: %s has value: %s", key, data); /* Try to get a property that does not exist */ value = sg_host_get_property_value(h1, noexist); XBT_INFO("\tProperty: %s has value: %s", noexist, value?value:"Undefined (NULL)"); xbt_dict_free(&props); /* Get the property list of 'host2' */ XBT_INFO("Property list for host %s", name2); props = sg_host_get_properties(h2); /* Print the properties of 'host2' */ xbt_dict_foreach (props, cursor, key, data) XBT_INFO("\tProperty: %s on host: %s", key, data); xbt_dict_free(&props); /* Modify an existing property test. First check it exists */ XBT_INFO("Modify an existing property"); value = sg_host_get_property_value(h2, exist); if (value == NULL) XBT_INFO("\tProperty: %s is undefined", exist); else { XBT_INFO("\tProperty: %s old value: %s", exist, value); sg_host_set_property_value(h2, exist, "250"); } /* Test if we have changed the value */ value = sg_host_get_property_value(h2, exist); XBT_INFO("\tProperty: %s new value: %s", exist, value?value:"Undefined (NULL)"); /* Test if properties are displayed by sg_host_dump */ sg_host_dump(h2); return 0; } SimGrid-3.18/examples/simdag/properties/sd_properties.tesh0000644000175000017500000000177013217757335024356 0ustar mquinsonmquinson#! ./tesh p Simple test of simdag with properties $ $SG_TEST_EXENV properties/sd_properties ${srcdir:=.}/../platforms/prop.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] Property list for host host1 > [0.000000] [test/INFO] Property: mem has value: 4 > [0.000000] [test/INFO] Property: NewProp has value: newValue > [0.000000] [test/INFO] Property: Hdd has value: 180 > [0.000000] [test/INFO] Property: NoProp has value: Undefined (NULL) > [0.000000] [test/INFO] Property list for host host2 > [0.000000] [test/INFO] Property: Hdd on host: 120 > [0.000000] [test/INFO] Modify an existing property > [0.000000] [test/INFO] Property: Hdd old value: 120 > [0.000000] [test/INFO] Property: Hdd new value: 250 > [0.000000] [sg_host/INFO] Displaying host host2 > [0.000000] [sg_host/INFO] - speed: 1000000000 > [0.000000] [sg_host/INFO] - available speed: 1.00 > [0.000000] [sg_host/INFO] - properties: > [0.000000] [sg_host/INFO] Hdd->250 SimGrid-3.18/examples/simdag/ptg-dotload/0000755000175000017500000000000013217757340020620 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/ptg-dotload/sd_ptg-dotload.c0000644000175000017500000000345513217757327023704 0ustar mquinsonmquinson/* Copyright (c) 2013-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/log.h" XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); /* simple test trying to load a Parallel Task Graph (PTG) as a DOT file. */ int main(int argc, char **argv){ xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert (argc > 1,"Usage: %s platform_file dot_file example: %s ../2clusters.xml ptg.dot", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file */ dot = SD_PTG_dotload(argv[2]); if(dot == NULL){ xbt_die("No dot load may be you have a cycle in your graph"); } /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } /* Schedule them all on all the first host*/ XBT_INFO("------------------- Schedule tasks ---------------------------"); sg_host_t *hosts = sg_host_list(); int count = sg_host_count(); xbt_dynar_foreach(dot, cursor, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_PAR_AMDAHL) { SD_task_schedulev(task, count, hosts); } } xbt_free(hosts); XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("Makespan: %f", SD_get_clock()); xbt_dynar_foreach(dot, cursor, task) { SD_task_destroy(task); } xbt_dynar_free_container(&dot); return 0; } SimGrid-3.18/examples/simdag/ptg-dotload/sd_ptg-dotload.tesh0000644000175000017500000001211413217757335024414 0ustar mquinsonmquinson#! ./tesh p Test the loader of PTG (Parallel Task Graph) written in the DOT format # The order differ when executed with gcc's thread sanitizer ! output sort $ $SG_TEST_EXENV ${bindir:=.}/ptg-dotload/sd_ptg-dotload ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/ptg-dotload/ptg.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: schedulable not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] root->c1 > [0.000000] [sd_task/INFO] root->c2 > [0.000000] [sd_task/INFO] Displaying task c1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 1000000000 > [0.000000] [sd_task/INFO] - alpha: 0.20 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root->c1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c1->c3 > [0.000000] [sd_task/INFO] Displaying task c2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 5000000000 > [0.000000] [sd_task/INFO] - alpha: 0.50 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root->c2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] Displaying task c3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 2000000000 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c2 > [0.000000] [sd_task/INFO] c1->c3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3->end > [0.000000] [sd_task/INFO] Displaying task c1->c3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 500000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] Displaying task c3->end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 200000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->c1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 200000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c1 > [0.000000] [sd_task/INFO] Displaying task root->c2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 100000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c2 > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c3->end > [0.000000] [test/INFO] ------------------- Schedule tasks --------------------------- > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [2.931978] [test/INFO] Makespan: 2.931978 SimGrid-3.18/examples/simdag/ptg-dotload/ptg.dot0000644000175000017500000000033613217757340022124 0ustar mquinsonmquinsondigraph G { c1 [size="1e9", alpha="0.2"]; c2 [size="5e9", alpha="0.5"]; c3 [size="2e9"]; root->c1 [size="2e8"]; root->c2 [size="1e8"]; c1->c3 [size="5e8"]; c2->c3 [size="-1."]; c3->end [size="2e8"]; } SimGrid-3.18/examples/simdag/fail/0000755000175000017500000000000013217757335017321 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/fail/sd_fail.c0000644000175000017500000000541113217757327021070 0ustar mquinsonmquinson/* Copyright (c) 2006-2010, 2012-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(sd_fail, "Logging specific to this SimDag example"); int main(int argc, char **argv) { double computation_amount[1]; double communication_amount[2] = { 0 }; sg_host_t hosts[1]; /* initialization of SD */ SD_init(&argc, argv); /* creation of the environment */ SD_create_environment(argv[1]); /* creation of a single task that will poorly fail when the workstation will stop */ XBT_INFO("First test: COMP_SEQ task"); SD_task_t task = SD_task_create_comp_seq("Poor task", NULL, 2e10); SD_task_watch(task, SD_FAILED); SD_task_watch(task, SD_DONE); XBT_INFO("Schedule task '%s' on 'Faulty Host'", SD_task_get_name(task)); SD_task_schedulel(task, 1, sg_host_by_name("Faulty Host")); SD_simulate(-1.0); SD_task_dump(task); XBT_INFO("Task '%s' has failed. %.f flops remain to be done", SD_task_get_name(task), SD_task_get_remaining_amount(task)); XBT_INFO("let's unschedule task '%s' and reschedule it on the 'Safe Host'", SD_task_get_name(task)); SD_task_unschedule(task); SD_task_schedulel(task, 1, sg_host_by_name("Safe Host")); XBT_INFO("Run the simulation again"); SD_simulate(-1.0); SD_task_dump(task); XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); SD_task_destroy(task); XBT_INFO("Second test: NON TYPED task"); task = SD_task_create("Poor parallel task", NULL, 2e10); SD_task_watch(task, SD_FAILED); SD_task_watch(task, SD_DONE); computation_amount[0] = 2e10; XBT_INFO("Schedule task '%s' on 'Faulty Host'", SD_task_get_name(task)); hosts[0] = sg_host_by_name("Faulty Host"); SD_task_schedule(task, 1, hosts, computation_amount, communication_amount,-1); SD_simulate(-1.0); SD_task_dump(task); XBT_INFO("Task '%s' has failed. %.f flops remain to be done", SD_task_get_name(task), SD_task_get_remaining_amount(task)); XBT_INFO("let's unschedule task '%s' and reschedule it on the 'Safe Host'", SD_task_get_name(task)); SD_task_unschedule(task); hosts[0] = sg_host_by_name("Safe Host"); SD_task_schedule(task, 1, hosts, computation_amount, communication_amount,-1); XBT_INFO("Run the simulation again"); SD_simulate(-1.0); SD_task_dump(task); XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); SD_task_destroy(task); return 0; } SimGrid-3.18/examples/simdag/fail/sd_fail.tesh0000644000175000017500000000431413217757335021611 0ustar mquinsonmquinson#! ./tesh p Test of the management of failed tasks simdag $ $SG_TEST_EXENV ${bindir:=.}/fail/sd_fail ${srcdir:=.}/../platforms/faulty_host.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_fail/INFO] First test: COMP_SEQ task > [0.000000] [sd_fail/INFO] Schedule task 'Poor task' on 'Faulty Host' > [10.000000] [sd_task/INFO] Displaying task Poor task > [10.000000] [sd_task/INFO] - state: not runnable failed > [10.000000] [sd_task/INFO] - kind: sequential computation > [10.000000] [sd_task/INFO] - amount: 20000000000 > [10.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [10.000000] [sd_fail/INFO] Task 'Poor task' has failed. 20000000000 flops remain to be done > [10.000000] [sd_fail/INFO] let's unschedule task 'Poor task' and reschedule it on the 'Safe Host' > [10.000000] [sd_fail/INFO] Run the simulation again > [50.000000] [sd_task/INFO] Displaying task Poor task > [50.000000] [sd_task/INFO] - state: not runnable done > [50.000000] [sd_task/INFO] - kind: sequential computation > [50.000000] [sd_task/INFO] - amount: 20000000000 > [50.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [50.000000] [sd_fail/INFO] Task 'Poor task' start time: 10.000000, finish time: 50.000000 > [50.000000] [sd_fail/INFO] Second test: NON TYPED task > [50.000000] [sd_fail/INFO] Schedule task 'Poor parallel task' on 'Faulty Host' > [60.000000] [sd_task/INFO] Displaying task Poor parallel task > [60.000000] [sd_task/INFO] - state: not runnable failed > [60.000000] [sd_task/INFO] - amount: 20000000000 > [60.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [60.000000] [sd_fail/INFO] Task 'Poor parallel task' has failed. 20000000000 flops remain to be done > [60.000000] [sd_fail/INFO] let's unschedule task 'Poor parallel task' and reschedule it on the 'Safe Host' > [60.000000] [sd_fail/INFO] Run the simulation again > [100.000000] [sd_task/INFO] Displaying task Poor parallel task > [100.000000] [sd_task/INFO] - state: not runnable done > [100.000000] [sd_task/INFO] - amount: 20000000000 > [100.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [100.000000] [sd_fail/INFO] Task 'Poor parallel task' start time: 60.000000, finish time: 100.000000 SimGrid-3.18/examples/simdag/goal/0000755000175000017500000000000013217757327017331 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/goal/goal_test.c0000644000175000017500000000536113217757327021463 0ustar mquinsonmquinson/* Example of scatter communication, accepting a large amount of processes. * This based the experiment of Fig. 4 in http://hal.inria.fr/hal-00650233/ * That experiment is a comparison to the LogOPSim simulator, that takes * GOAL files as an input, thus the file name. But there is no actual link * to the GOAL formalism beside of this. */ /* Copyright (c) 2011-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "simgrid/simdag.h" #include "xbt/xbt_os_time.h" XBT_LOG_NEW_DEFAULT_CATEGORY(goal, "The GOAL loader into SimDag"); typedef struct { int i; int j; int k; } s_bcast_task_t; typedef s_bcast_task_t *bcast_task_t; const sg_host_t* ws_list; int count = 0; xbt_dynar_t reclaimed; static void send_one(int from, int to) { if (count %100000 == 0) XBT_INFO("Sending task #%d",count); count++; bcast_task_t bt; if (!xbt_dynar_is_empty(reclaimed)) { bt = xbt_dynar_pop_as(reclaimed,bcast_task_t); } else { bt = xbt_new(s_bcast_task_t,1); } bt->i=from; bt->j=(from+to)/2; bt->k=to; SD_task_t task = SD_task_create_comm_e2e(NULL,bt,424242); XBT_DEBUG("Schedule task between %d and %d",bt->i,bt->j); SD_task_schedulel(task,2,ws_list[bt->i],ws_list[bt->j]); SD_task_watch(task,SD_DONE); } int main(int argc, char **argv) { xbt_os_timer_t timer = xbt_os_timer_new(); /* initialization of SD */ SD_init(&argc, argv); if (argc > 1) { SD_create_environment(argv[1]); } else { SD_create_environment("../../platforms/One_cluster_no_backbone.xml"); } ws_list = sg_host_list(); reclaimed = xbt_dynar_new(sizeof(bcast_task_t),xbt_free_ref); xbt_dynar_t done = xbt_dynar_new(sizeof(SD_task_t), NULL); xbt_os_cputimer_start(timer); send_one(0,sg_host_count()); do { if (!xbt_dynar_is_empty(done)) { unsigned int cursor; SD_task_t task; xbt_dynar_foreach(done, cursor, task) { bcast_task_t bt = SD_task_get_data(task); if (bt->i != bt->j -1) send_one(bt->i,bt->j); if (bt->j != bt->k -1) send_one(bt->j,bt->k); if (xbt_dynar_length(reclaimed)<100) { xbt_dynar_push_as(reclaimed,bcast_task_t,bt); } else { free(bt); } SD_task_destroy(task); } xbt_dynar_free_container(&done); } SD_simulate_with_update(-1, done); } while(!xbt_dynar_is_empty(done)); xbt_os_cputimer_stop(timer); printf("exec_time:%f\n", xbt_os_timer_elapsed(timer) ); xbt_dynar_free(&done); xbt_dynar_free(&reclaimed); XBT_INFO("Done. Bailing out"); return 0; } SimGrid-3.18/examples/simdag/throttling/0000755000175000017500000000000013217757335020604 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/throttling/sd_throttling.tesh0000644000175000017500000000207313217757335024357 0ustar mquinsonmquinson#! ./tesh p Modify the rate of communication tasks even when they are auto-scheduled # We need to sort this out because the order changes with the sanitizers (at least) ! output sort $ $SG_TEST_EXENV ./throttling/sd_throttling ${srcdir:=.}/../platforms/cluster.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [5.000000] [sd_comm_throttling/INFO] Simulation stopped after 5.0000 seconds > [5.000000] [sd_comm_throttling/INFO] Task 'Task A' start time: 0.000000, finish time: 5.000000 > [10.080600] [sd_comm_throttling/INFO] Simulation stopped after 10.0806 seconds > [10.080600] [sd_comm_throttling/INFO] Task 'Task B' start time: 5.000000, finish time: 5.080600 > [10.080600] [sd_comm_throttling/INFO] Task 'Task C' start time: 5.080600, finish time: 10.080600 > [15.241200] [sd_comm_throttling/INFO] Simulation stopped after 15.2412 seconds > [15.241200] [sd_comm_throttling/INFO] Task 'Task D' start time: 10.080600, finish time: 10.241200 > [15.241200] [sd_comm_throttling/INFO] Task 'Task E' start time: 10.241200, finish time: 15.241200 SimGrid-3.18/examples/simdag/throttling/sd_throttling.c0000644000175000017500000000600413217757327023635 0ustar mquinsonmquinson/* Copyright (c) 2006-2010, 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(sd_comm_throttling, "Logging specific to this SimDag example"); int main(int argc, char **argv) { unsigned int ctr; SD_task_t task; xbt_dynar_t changed_tasks = xbt_dynar_new(sizeof(SD_task_t), NULL); SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\nExample: %s two_clusters.xml", argv[0], argv[0]); SD_create_environment(argv[1]); sg_host_t *hosts = sg_host_list(); /* creation of some typed tasks and their dependencies */ /* chain of five tasks, three compute tasks with two data transfers in between */ SD_task_t taskA = SD_task_create_comp_seq("Task A", NULL, 5e9); SD_task_t taskB = SD_task_create_comm_e2e("Task B", NULL, 1e7); SD_task_t taskC = SD_task_create_comp_seq("Task C", NULL, 5e9); SD_task_t taskD = SD_task_create_comm_e2e("Task D", NULL, 1e7); SD_task_t taskE = SD_task_create_comp_seq("Task E", NULL, 5e9); SD_task_dependency_add(NULL, NULL, taskA, taskB); SD_task_dependency_add(NULL, NULL, taskB, taskC); SD_task_dependency_add(NULL, NULL, taskC, taskD); SD_task_dependency_add(NULL, NULL, taskD, taskE); /* Add watchpoints on completion of compute tasks */ SD_task_watch(taskA, SD_DONE); SD_task_watch(taskC, SD_DONE); SD_task_watch(taskE, SD_DONE); /* Auto-schedule the compute tasks on three different workstations */ /* Data transfer tasks taskB and taskD are automagically scheduled */ SD_task_schedulel(taskA, 1, hosts[0]); SD_task_schedulel(taskC, 1, hosts[1]); SD_task_schedulel(taskE, 1, hosts[0]); SD_simulate_with_update(-1.0, changed_tasks); while (!xbt_dynar_is_empty(changed_tasks)) { XBT_INFO("Simulation stopped after %.4f seconds", SD_get_clock()); xbt_dynar_foreach(changed_tasks, ctr, task) { XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); } xbt_dynar_reset(changed_tasks); /* let throttle the communication for taskD if its parent is SD_DONE */ /* the bandwidth is 1.25e8, the data size is 1e7, and we want to throttle the bandwidth by a factor 2. * The rate is then 1.25e8/(2*1e7)=6.25 * Changing the rate is possible before the task execution starts (in SD_RUNNING state). */ if (SD_task_get_state(taskC) == SD_DONE && SD_task_get_state(taskD) < SD_RUNNING) SD_task_set_rate(taskD, 6.25); SD_simulate_with_update(-1.0, changed_tasks); } xbt_dynar_free(&changed_tasks); XBT_DEBUG("Destroying tasks..."); SD_task_destroy(taskA); SD_task_destroy(taskB); SD_task_destroy(taskC); SD_task_destroy(taskD); SD_task_destroy(taskE); XBT_DEBUG("Tasks destroyed. Exiting SimDag..."); xbt_free(hosts); return 0; } SimGrid-3.18/examples/simdag/test/0000755000175000017500000000000013217757335017365 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/test/sd_test.tesh0000644000175000017500000000365713217757335021732 0ustar mquinsonmquinson#! ./tesh p Simple test of simdag ! output sort $ $SG_TEST_EXENV ./test/sd_test ${srcdir:=.}/../platforms/small_platform.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_test/INFO] Computation time for 2000000.000000 flops on Jacquelin: 0.014563 > [0.000000] [sd_test/INFO] Computation time for 1000000.000000 flops on Fafard: 0.013107 > [0.000000] [sd_test/INFO] Route between Jacquelin and Fafard: > [0.000000] [sd_test/INFO] Link 145: latency = 0.000410, bandwidth = 2583375.000000 > [0.000000] [sd_test/INFO] Link 59: latency = 0.000371, bandwidth = 11845375.000000 > [0.000000] [sd_test/INFO] Link 56: latency = 0.029589, bandwidth = 21414750.000000 > [0.000000] [sd_test/INFO] Link 54: latency = 0.035083, bandwidth = 15376875.000000 > [0.000000] [sd_test/INFO] Link 17: latency = 0.000137, bandwidth = 118682500.000000 > [0.000000] [sd_test/INFO] Link 16: latency = 0.000514, bandwidth = 34285625.000000 > [0.000000] [sd_test/INFO] Link 10: latency = 0.000514, bandwidth = 34285625.000000 > [0.000000] [sd_test/INFO] Link 6: latency = 0.000060, bandwidth = 41279125.000000 > [0.000000] [sd_test/INFO] Link 9: latency = 0.001462, bandwidth = 7209750.000000 > [0.000000] [sd_test/INFO] Link 79: latency = 0.000156, bandwidth = 8427250.000000 > [0.000000] [sd_test/INFO] Link 78: latency = 0.000278, bandwidth = 27946250.000000 > [0.000000] [sd_test/INFO] Route latency = 0.068575, route bandwidth = 2583375.000000 > [0.000000] [sd_test/INFO] Communication time for 2000000.000000 bytes between Jacquelin and Fafard: 0.842756 > [0.000000] [sd_test/INFO] Communication time for 3000000.000000 bytes between Fafard and Jacquelin: 1.229846 > [0.000000] [sd_test/INFO] Estimated time for 'Task D': 1.242953 > [4.008055] [sd_test/INFO] Task 'Task B' start time: 2.004027, finish time: 4.008055 > [4.008055] [sd_test/INFO] Task 'Task D' start time: 0.000000, finish time: 2.004027 SimGrid-3.18/examples/simdag/test/sd_test.cpp0000644000175000017500000001300213217757327021533 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u.hpp" #include "simgrid/simdag.h" #include "xbt/ex.h" #include "xbt/log.h" #include #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(sd_test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { sg_host_t host_list[2]; double computation_amount[2]; double communication_amount[4] = { 0 }; /* initialization of SD */ SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\nExample: %s two_clusters.xml", argv[0], argv[0]); SD_create_environment(argv[1]); /* test the estimation functions */ const sg_host_t* hosts = sg_host_list(); simgrid::s4u::Host* h1 = hosts[4]; simgrid::s4u::Host* h2 = hosts[2]; double comp_amount1 = 2000000; double comp_amount2 = 1000000; double comm_amount12 = 2000000; double comm_amount21 = 3000000; XBT_INFO("Computation time for %f flops on %s: %f", comp_amount1, h1->getCname(), comp_amount1 / h1->getSpeed()); XBT_INFO("Computation time for %f flops on %s: %f", comp_amount2, h2->getCname(), comp_amount2 / h2->getSpeed()); XBT_INFO("Route between %s and %s:", h1->getCname(), h2->getCname()); std::vector route; double latency = 0; h1->routeTo(h2, route, &latency); for (auto const& link : route) XBT_INFO(" Link %s: latency = %f, bandwidth = %f", sg_link_name(link), sg_link_latency(link), sg_link_bandwidth(link)); XBT_INFO("Route latency = %f, route bandwidth = %f", latency, sg_host_route_bandwidth(h1, h2)); XBT_INFO("Communication time for %f bytes between %s and %s: %f", comm_amount12, h1->getCname(), h2->getCname(), sg_host_route_latency(h1, h2) + comm_amount12 / sg_host_route_bandwidth(h1, h2)); XBT_INFO("Communication time for %f bytes between %s and %s: %f", comm_amount21, h2->getCname(), h1->getCname(), sg_host_route_latency(h2, h1) + comm_amount21 / sg_host_route_bandwidth(h2, h1)); /* creation of the tasks and their dependencies */ SD_task_t taskA = SD_task_create("Task A", NULL, 10.0); SD_task_t taskB = SD_task_create("Task B", NULL, 40.0); SD_task_t taskC = SD_task_create("Task C", NULL, 30.0); SD_task_t taskD = SD_task_create("Task D", NULL, 60.0); /* try to attach and retrieve user data to a task */ SD_task_set_data(taskA, static_cast(&comp_amount1)); if (fabs(comp_amount1 - (*(static_cast(SD_task_get_data(taskA))))) > 1e-12) XBT_ERROR("User data was corrupted by a simple set/get"); SD_task_dependency_add(NULL, NULL, taskB, taskA); SD_task_dependency_add(NULL, NULL, taskC, taskA); SD_task_dependency_add(NULL, NULL, taskD, taskB); SD_task_dependency_add(NULL, NULL, taskD, taskC); SD_task_dependency_add(NULL, NULL, taskB, taskC); try { SD_task_dependency_add(NULL, NULL, taskA, taskA); /* shouldn't work and must raise an exception */ xbt_die("Hey, I can add a dependency between Task A and Task A!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; /* this is a serious error */ } try { SD_task_dependency_add(NULL, NULL, taskB, taskA); /* shouldn't work and must raise an exception */ xbt_die("Oh oh, I can add an already existing dependency!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } try { SD_task_dependency_remove(taskA, taskC); /* shouldn't work and must raise an exception */ xbt_die("Dude, I can remove an unknown dependency!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } try { SD_task_dependency_remove(taskC, taskC); /* shouldn't work and must raise an exception */ xbt_die("Wow, I can remove a dependency between Task C and itself!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } /* if everything is ok, no exception is forwarded or rethrown by main() */ /* watch points */ SD_task_watch(taskD, SD_DONE); SD_task_watch(taskB, SD_DONE); SD_task_unwatch(taskD, SD_DONE); /* scheduling parameters */ host_list[0] = h1; host_list[1] = h2; computation_amount[0] = comp_amount1; computation_amount[1] = comp_amount2; communication_amount[1] = comm_amount12; communication_amount[2] = comm_amount21; /* estimated time */ SD_task_t task = taskD; XBT_INFO("Estimated time for '%s': %f", SD_task_get_name(task), SD_task_get_execution_time(task, 2, host_list, computation_amount, communication_amount)); SD_task_schedule(taskA, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskB, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskC, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskD, 2, host_list, computation_amount, communication_amount, -1); std::set *changed_tasks = simgrid::sd::simulate(-1.0); for (auto const& task : *changed_tasks) { XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); } XBT_DEBUG("Destroying tasks..."); SD_task_destroy(taskA); SD_task_destroy(taskB); SD_task_destroy(taskC); SD_task_destroy(taskD); XBT_DEBUG("Tasks destroyed. Exiting SimDag..."); xbt_free((sg_host_t*)hosts); return 0; } SimGrid-3.18/examples/simdag/scheduling/0000755000175000017500000000000013217757343020532 5ustar mquinsonmquinsonSimGrid-3.18/examples/simdag/scheduling/Montage_25.xml0000644000175000017500000005505713217757343023170 0ustar mquinsonmquinson SimGrid-3.18/examples/simdag/scheduling/sd_scheduling.c0000644000175000017500000002036213217757327023516 0ustar mquinsonmquinson/* Copyright (c) 2009-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* simple test to schedule a DAX file with the Min-Min algorithm. */ #include #include "simgrid/simdag.h" #if SIMGRID_HAVE_JEDULE #include "simgrid/jedule/jedule_sd_binding.h" #endif XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); typedef struct _HostAttribute *HostAttribute; struct _HostAttribute { /* Earliest time at which a host is ready to execute a task */ double available_at; SD_task_t last_scheduled_task; }; static double sg_host_get_available_at(sg_host_t host) { HostAttribute attr = (HostAttribute) sg_host_user(host); return attr->available_at; } static void sg_host_set_available_at(sg_host_t host, double time) { HostAttribute attr = (HostAttribute) sg_host_user(host); attr->available_at = time; sg_host_user_set(host, attr); } static SD_task_t sg_host_get_last_scheduled_task( sg_host_t host){ HostAttribute attr = (HostAttribute) sg_host_user(host); return attr->last_scheduled_task; } static void sg_host_set_last_scheduled_task(sg_host_t host, SD_task_t task){ HostAttribute attr = (HostAttribute) sg_host_user(host); attr->last_scheduled_task=task; sg_host_user_set(host, attr); } static xbt_dynar_t get_ready_tasks(xbt_dynar_t dax) { unsigned int i; xbt_dynar_t ready_tasks; SD_task_t task; ready_tasks = xbt_dynar_new(sizeof(SD_task_t), NULL); xbt_dynar_foreach(dax, i, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_SEQ && SD_task_get_state(task) == SD_SCHEDULABLE) { xbt_dynar_push(ready_tasks, &task); } } XBT_DEBUG("There are %lu ready tasks", xbt_dynar_length(ready_tasks)); return ready_tasks; } static double finish_on_at(SD_task_t task, sg_host_t host) { double result; xbt_dynar_t parents = SD_task_get_parents(task); if (!xbt_dynar_is_empty(parents)) { unsigned int i; double data_available = 0.; double redist_time = 0; double last_data_available; /* compute last_data_available */ SD_task_t parent; last_data_available = -1.0; xbt_dynar_foreach(parents, i, parent) { /* normal case */ if (SD_task_get_kind(parent) == SD_TASK_COMM_E2E) { sg_host_t * parent_host= SD_task_get_workstation_list(parent); /* Estimate the redistribution time from this parent */ if (SD_task_get_amount(parent) <= 1e-6){ redist_time= 0; } else { redist_time = sg_host_route_latency(parent_host[0], host) + SD_task_get_amount(parent) / sg_host_route_bandwidth(parent_host[0], host); } data_available = SD_task_get_start_time(parent) + redist_time; } /* no transfer, control dependency */ if (SD_task_get_kind(parent) == SD_TASK_COMP_SEQ) { data_available = SD_task_get_finish_time(parent); } if (last_data_available < data_available) last_data_available = data_available; } xbt_dynar_free_container(&parents); result = MAX(sg_host_get_available_at(host), last_data_available) + SD_task_get_amount(task)/sg_host_speed(host); } else { xbt_dynar_free_container(&parents); result = sg_host_get_available_at(host) + SD_task_get_amount(task)/sg_host_speed(host); } return result; } static sg_host_t SD_task_get_best_host(SD_task_t task) { sg_host_t *hosts = sg_host_list(); int nhosts = sg_host_count(); sg_host_t best_host = hosts[0]; double min_EFT = finish_on_at(task, hosts[0]); for (int i = 1; i < nhosts; i++) { double EFT = finish_on_at(task, hosts[i]); XBT_DEBUG("%s finishes on %s at %f", SD_task_get_name(task), sg_host_get_name(hosts[i]), EFT); if (EFT < min_EFT) { min_EFT = EFT; best_host = hosts[i]; } } xbt_free(hosts); return best_host; } int main(int argc, char **argv) { unsigned int cursor; double min_finish_time = -1.0; SD_task_t task; SD_task_t selected_task = NULL; xbt_dynar_t ready_tasks; sg_host_t selected_host = NULL; char * tracefilename = NULL; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dax_file [jedule_file]\n" "\tExample: %s simulacrum_7_hosts.xml Montage_25.xml Montage_25.jed", argv[0], argv[0]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* creation of the environment */ SD_create_environment(argv[1]); /* Allocating the host attribute */ int total_nhosts = sg_host_count(); sg_host_t *hosts = sg_host_list(); for (cursor = 0; cursor < total_nhosts; cursor++) sg_host_user_set(hosts[cursor], xbt_new0(struct _HostAttribute, 1)); /* load the DAX file */ xbt_dynar_t dax = SD_daxload(argv[2]); xbt_dynar_foreach(dax, cursor, task) { SD_task_watch(task, SD_DONE); } /* Schedule the root first */ xbt_dynar_get_cpy(dax, 0, &task); sg_host_t host = SD_task_get_best_host(task); SD_task_schedulel(task, 1, host); xbt_dynar_t changed_tasks = xbt_dynar_new(sizeof(SD_task_t), NULL); SD_simulate_with_update(-1.0, changed_tasks); while (!xbt_dynar_is_empty(changed_tasks)) { /* Get the set of ready tasks */ ready_tasks = get_ready_tasks(dax); xbt_dynar_reset(changed_tasks); if (xbt_dynar_is_empty(ready_tasks)) { xbt_dynar_free_container(&ready_tasks); /* there is no ready task, let advance the simulation */ SD_simulate_with_update(-1.0, changed_tasks); continue; } /* For each ready task: * get the host that minimizes the completion time. * select the task that has the minimum completion time on its best host. */ xbt_dynar_foreach(ready_tasks, cursor, task) { XBT_DEBUG("%s is ready", SD_task_get_name(task)); host = SD_task_get_best_host(task); double finish_time = finish_on_at(task, host); if (min_finish_time < 0 || finish_time < min_finish_time) { min_finish_time = finish_time; selected_task = task; selected_host = host; } } XBT_INFO("Schedule %s on %s", SD_task_get_name(selected_task), sg_host_get_name(selected_host)); SD_task_schedulel(selected_task, 1, selected_host); /* * SimDag allows tasks to be executed concurrently when they can by default. * Yet schedulers take decisions assuming that tasks wait for resource availability to start. * The solution (well crude hack is to keep track of the last task scheduled on a host and add a special type of * dependency if needed to force the sequential execution meant by the scheduler. * If the last scheduled task is already done, has failed or is a predecessor of the current task, no need for a * new dependency */ SD_task_t last_scheduled_task = sg_host_get_last_scheduled_task(selected_host); if (last_scheduled_task && (SD_task_get_state(last_scheduled_task) != SD_DONE) && (SD_task_get_state(last_scheduled_task) != SD_FAILED) && !SD_task_dependency_exists(sg_host_get_last_scheduled_task(selected_host), selected_task)) SD_task_dependency_add("resource", NULL, last_scheduled_task, selected_task); sg_host_set_last_scheduled_task(selected_host, selected_task); sg_host_set_available_at(selected_host, min_finish_time); xbt_dynar_free_container(&ready_tasks); /* reset the min_finish_time for the next set of ready tasks */ min_finish_time = -1.; xbt_dynar_reset(changed_tasks); SD_simulate_with_update(-1.0, changed_tasks); } XBT_INFO("Simulation Time: %f", SD_get_clock()); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing a jedule output (if active) of the run into %s", tracefilename?tracefilename:"minmin_test.jed"); #if SIMGRID_HAVE_JEDULE jedule_sd_dump(tracefilename); #endif free(tracefilename); xbt_dynar_free_container(&ready_tasks); xbt_dynar_free(&changed_tasks); xbt_dynar_foreach(dax, cursor, task) { SD_task_destroy(task); } xbt_dynar_free_container(&dax); for (cursor = 0; cursor < total_nhosts; cursor++) { free(sg_host_user(hosts[cursor])); sg_host_user_set(hosts[cursor], NULL); } xbt_free(hosts); return 0; } SimGrid-3.18/examples/simdag/scheduling/expected_output.jed0000644000175000017500000012312213217757340024435 0ustar mquinsonmquinson )) { next if ($line =~ /^#/); # 278 7.2 -9.4 h 2.3 if($line =~ /^([^ ]*) ([^ ]*) ([^ ]*) h ([^ ]*) *$/) { print "\t\t\n"; next; } die "Parse error: $line\n"; } print "\t\n"; print "\n"; SimGrid-3.18/examples/platforms/cluster_dragonfly.xml0000644000175000017500000000067313217757324023425 0ustar mquinsonmquinson SimGrid-3.18/examples/platforms/griffon.xml0000644000175000017500000000360413217757325021327 0ustar mquinsonmquinson SimGrid-3.18/examples/platforms/two_hosts.xml0000644000175000017500000000120013217757325021714 0ustar mquinsonmquinson 1.0 0.5 4.0 1.0 SimGrid-3.18/examples/platforms/routing_cluster.lua0000644000175000017500000000423213217757325023104 0ustar mquinsonmquinson-- See routing_cluster.xml for description. -- require "simgrid" simgrid.engine.open() simgrid.engine.AS_open{id="AS0",mode="Full"} simgrid.engine.AS_open{id="my_cluster1",mode="Cluster"} simgrid.engine.router_new{id="router1"} simgrid.engine.host_new{id="host1",speed="1Gf"}; simgrid.engine.link_new{id="l1_UP",bandwidth="125MBps",lat="100us"}; simgrid.engine.link_new{id="l1_DOWN",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host1",up="l1_UP",down="l1_DOWN"}; simgrid.engine.host_new{id="host2",speed="1Gf"}; simgrid.engine.link_new{id="l2",bandwidth="125MBps",lat="100us",sharing_policy="FULLDUPLEX"}; simgrid.engine.host_link_new{id="host2",up="l2_UP",down="l2_DOWN"}; simgrid.engine.host_new{id="host3",speed="1Gf"}; simgrid.engine.link_new{id="l3",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host3",up="l3",down="l3"}; simgrid.engine.backbone_new{id="backbone1",bandwidth="2.25GBps",lat="500us"}; simgrid.engine.AS_seal() simgrid.engine.AS_open{id="my_cluster2",mode="Cluster"} simgrid.engine.router_new{id="router2"} simgrid.engine.host_new{id="host4",speed="1Gf"}; simgrid.engine.link_new{id="l4_UP",bandwidth="125MBps",lat="100us"}; simgrid.engine.link_new{id="l4_DOWN",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host4",up="l4_UP",down="l4_DOWN"}; simgrid.engine.host_new{id="host5",speed="1Gf"}; simgrid.engine.link_new{id="l5",bandwidth="125MBps",lat="100us",sharing_policy="FULLDUPLEX"}; simgrid.engine.host_link_new{id="host5",up="l5_UP",down="l5_DOWN"}; simgrid.engine.host_new{id="host6",speed="1Gf"}; simgrid.engine.link_new{id="l6",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host6",up="l6",down="l6"}; simgrid.engine.backbone_new{id="backbone2",bandwidth="2.25GBps",lat="500us"} simgrid.engine.AS_seal() simgrid.engine.link_new{id="link1-2",bandwidth="2.25GBps",lat="500us"}; simgrid.engine.ASroute_new{src="my_cluster1", dst="my_cluster2", gw_src="router1", gw_dst="router2", links="link1-2"} simgrid.engine.AS_seal() simgrid.engine.close() SimGrid-3.18/examples/platforms/energy_cluster.xml0000644000175000017500000000116213217757325022724 0ustar mquinsonmquinson SimGrid-3.18/examples/platforms/cluster_no_backbone.xml0000644000175000017500000000111213217757324023665 0ustar mquinsonmquinson SimGrid-3.18/examples/platforms/two_hosts_platform_with_availability.xml0000644000175000017500000000077113217757325027421 0ustar mquinsonmquinson SimGrid-3.18/INSTALL0000644000175000017500000002372713217757340014364 0ustar mquinsonmquinsonThis page summarizes how to compile SimGrid. The full Install documentation is available in doc/html/install.html or online at http://simgrid.gforge.inria.fr/ Getting the Dependencies ------------------------ SimGrid only uses very standard tools: - C compiler, C++ compiler, make and friends. - perl (but you may try to go without it) - cmake (version 2.8.8 or higher). You may want to use ccmake for a graphical interface over cmake. - boost: - Max OS X: with fink: fink install boost1.53.nopython, or with homebrew: brew install boost - Debian / Ubuntu: apt-get install libboost-dev libboost-context-dev - Java (if you want to build the Java bindings): - Mac OS X or Windows: Grab a full JDK - Debian / Ubuntu: apt-get install default-jdk Build Configuration ------------------- Note that compile-time options are very different from run-time options. The default configuration should be fine for most usages, but if you need to change something, there are several ways to do so. First, you can use environment variables. For example, you can change the compilers used by issuing these commands before launching cmake: export CC=gcc-4.7 export CXX=g++-4.7 Note that other variables are available, such as CFLAGS and CXXFLAGS to add options respectively for the C and C++ compilers. Another way to do so is to use the -D argument of cmake as follows. Note that the ending dot is mandatory (see Out of Tree Compilation). cmake -DCC=clang -DCXX=clang++ . Finally, you can use the ccmake graphical interface to change these settings. ccmake . Existing compilation options ---------------------------- CMAKE_INSTALL_PREFIX (path) Where to install SimGrid (/opt/simgrid, /usr/local, or elsewhere). enable_compile_optimizations (ON/OFF) Request the compiler to produce efficient code. You want to activate it, unless you plan to debug SimGrid itself. Indeed, efficient code may be appear mangled to debuggers. enable_compile_warnings (ON/OFF) Request the compiler to issue error messages whenever the source code is not perfectly clean. If you are a SimGrid developer, you have to activate this option to enforce the code quality. As a regular user, this option will bring you nothing. enable_debug (ON/OFF) Disable this option toto discard all log messages of gravity debug or below at compile time. The resulting code is faster than if you discarding these messages at runtime. However, it obviously becomes impossible to get any debug info from SimGrid if something goes wrong. enable_documentation (ON/OFF) Generate the documentation pages. enable_java (ON/OFF) To enjoy the java bindings of SimGrid. enable_jedule (ON/OFF) To get SimDag producing execution traces that can then be visualized with the Jedule external tool. enable_lua (ON/OFF) To enjoy the lua bindings to the SimGrid internals. enable_lib_in_jar (ON/OFF) Bundle the native java bindings in the jar file. enable_lto (ON/OFF) Enable the Link Time Optimization of the C compiler. This feature really speeds up the produced code, but it is fragile with some versions of GCC. enable_maintainer_mode (ON/OFF) Only needed if you plan to modify very specific parts of SimGrid (e.g., the XML parsers and other related elements). Moreover, this adds an extra dependency on flex and flexml. enable_mallocators (ON/OFF) Disabled this when tracking memory issues within SimGrid, or our internal memory caching mechanism will fool the debuggers. enable_model-checking (ON/OFF) This execution gear is very usable now, but enabling this option at compile time will hinder simulation speed even when the model-checker is not activated at run time. enable_ns3 (ON/OFF) Allow to use ns-3 as a SimGrid network model. enable_smpi (ON/OFF) Allow to run MPI code on top of SimGrid. enable_smpi_ISP_testsuite (ON/OFF) Add many extra tests for the model-checker module. enable_smpi_MPICH3_testsuite (ON/OFF) Add many extra tests for the MPI module. Reset the build configuration ----------------------------- To empty the cmake cache (either when you add a new library or when things go seriously wrong), simply delete your CMakeCache.txt. You may also want to directly edit this file in some circumstances. Out of Tree Compilation ----------------------- By default, the files produced during the compilation are placed in the source directory. It is however often better to put them all in a separate directory: cleaning the tree becomes as easy as removing this directory, and you can have several such directories to test several parameter sets or architectures. For that, go to the directory where the files should be produced, and invoke cmake (or ccmake) with the full path to the SimGrid source as last argument. mkdir build cd build cmake [options] .. make Mac OS X Builds --------------- SimGrid compiles like a charm with clang (version 3.0 or higher) on Mac OS X: cmake -DCMAKE_C_COMPILER=/path/to/clang -DCMAKE_CXX_COMPILER=/path/to/clang++ . make With the XCode version of clang 4.1, you may get the following error message: CMake Error: Parse error in cache file build_dir/CMakeCache.txt. Offending entry: /SDKs/MacOSX10.8.sdk In that case, edit the CMakeCache.txt file directly, so that the CMAKE_OSX_SYSROOT is similar to the following. Don't worry about the warning that the "-pthread" argument is not used, if it appears. CMAKE_OSX_SYSROOT:PATH=/Applications/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer In the El Capitan version of Max OS X, Apple decided that users don't need no /usr/include directory anymore. If you are hit by this pure madness, just run the following command to restore that classical UNIX directory: xcode-select -install Windows Builds -------------- Building SimGrid on Windows may be something of an adventure: We only manage to do so ourselves with MinGW-64, ActiveState Perl and msys git). Have a look at out configuration scripts in appveyor.yml, but don't expect too much from us: we are really not fluent with Windows. Actually your help is welcome. The drawback of MinGW-64 is that the produced DLL are not compatible with MS Visual C. clang-cl sounds promising to fix this. If you get something working, please tell us. Build the Java bindings ----------------------- Once you have the full JDK installed (on Debian/Ubuntu, grab the package default-jdk for that), things should be as simple as: cmake -Denable_java=ON . make After the compilation, the file simgrid.jar is produced in the root directory. If you only want to build the jarfile and its dependencies, type make simgrid-java_jar. It will save you the time of building every C examples and other things that you don't need for Java. Sometimes, the build system fails to find the JNI headers: Error: jni could not be found. In this case, you need to first locate them as follows: $ locate jni.h /usr/lib/jvm/java-7-openjdk-amd64/include/jni.h /usr/lib/jvm/java-8-openjdk-amd64/include/jni.h Then, set the JAVA_INCLUDE_PATH environment variable to the right path, and relaunch cmake. If you have several version of jni installed (as above), use the right one (check the java version you use with javac -version). export JAVA_INCLUDE_PATH=/usr/lib/jvm/java-8-openjdk-amd64/include/ cmake -Denable_java=ON . make Note that the filename jni.h was removed from the path. 32 bits Builds on Multi-arch Linux ---------------------------------- On a multiarch x86_64 Linux, it should be possible to compile a 32 bit version of SimGrid with something like: CFLAGS=-m32 \ CXXFLAGS=-m32 \ PKG_CONFIG_LIBDIR=/usr/lib/i386-linux-gnu/pkgconfig/ \ cmake . \ -DCMAKE_SYSTEM_PROCESSOR=i386 \ -DCMAKE_Fortran_COMPILER=/some/path/to/i686-linux-gnu-gfortran \ -DGFORTRAN_EXE=/some/path/to/i686-linux-gnu-gfortran \ -DCMAKE_Fortran_FLAGS=-m32 If needed, implement i686-linux-gnu-gfortran as a script: #!/bin/sh exec gfortran -m32 "$@" Existing Compilation Targets ---------------------------- In most cases, compiling and installing SimGrid is enough: make make install # try "sudo make install" if you don't have the permission to write In addition, several compilation targets are provided in SimGrid. If your system is well configured, the full list of targets is available for completion when using the Tab key. Note that some of the existing targets are not really for public consumption so don't worry if some stuff doesn't work for you. make simgrid Build only the SimGrid library and not any example make app-masterworker Build only this example (works for any example) make clean Clean the results of a previous compilation make install Install the project (doc/ bin/ lib/ include/) make uninstall Uninstall the project (doc/ bin/ lib/ include/) make dist Build a distribution archive (tgz) make distcheck Check the dist (make + make dist + tests on the distribution) make documentation Create SimGrid documentation If you want to see what is really happening, try adding VERBOSE=1 to your compilation requests: make VERBOSE=1 Testing your build ------------------ Once everything is built, you may want to test the result. SimGrid comes with an extensive set of regression tests (as described in the insider manual). The tests are run with ctest, that comes with CMake. We run them every commit and the results are on our Jenkins. ctest # Launch all tests ctest -R msg # Launch only the tests which name match the string "msg" ctest -j4 # Launch all tests in parallel, at most 4 at the same time ctest --verbose # Display all details on what's going on ctest --output-on-failure # Only get verbose for the tests that fail ctest -R msg- -j5 --output-on-failure # You changed MSG and want to check that you didn't break anything, huh? # That's fine, I do so all the time myself. SimGrid-3.18/ChangeLog0000644000175000017500000050032113217757340015073 0ustar mquinsonmquinsonSimGrid (3.18) Released December 24 2017 The "Ho Ho Ho! SimGrid 4 beta is coming to town" release. MSG IS NOW DEPRECATED. Please use S4U for new projects. - Support for MSG should not be removed from SimGrid before 2020, but future evolutions will be limited to the integration of user-provided patches. S4U New features - s4u::onDeadlock() signal, run before stopping the simulation. - s4u::Exec: asynchronous executions (abstraction of background threads) - s4u::Host->getLoad() returns the achieved speed in flops/s S4U API changes - Link::name() is deprecated and will be removed in v3.21, use Link::getCname() instead. - Mailbox::getName() changed to return a std::string, use Mailbox::getCname() instead to get a char*. - Storage::getName() changed to return a std::string, use Storage::getCname() instead to get a char*. - s4u::allStorages() becomes s4u::getStorageList(whereTo) to have both allocation and deallocation of the map in user space. MSG - Forbid the use of MSG_task_get_flops_amount() on parallel tasks: The amount of flops remaining to do is a vector, not a scalar. - Introduce MSG_task_get_remaining_work_ratio(), which does what its name implies on both sequential and parallel tasks. - Both changes fix GitHub's #223 using PR #237 as a basis. Thanks Michael Mercier. - Most examples were converted to S4U and hidden elsewhere at tests. Rational: we still want MSG to work; we want newcomers to use S4U. SURF - LMM stuff moved to its own namespace: simgrid::kernel::lmm. - Renamed LMM classes (e.g. s_lmm_system_t -> System). SMPI - Switch to the faster dlopen privatization mechanism by default - Documentation improvments Virtual Machines - Live migration is getting moved to a plugin. Dirty page tracking is the first part of this plugin. This imply that VM migration is now only possible if one this function is called : - C/MSG: MSG_vm_live_migration_plugin_init() - C/C++: sg_vm_live_migration_plugin_init() - Java: Msg.liveMigrationInit() For C and C++, "simgrid/plugins/live_migration.h" also has to be included. XBT - Define class simgrid::xbt::Path to manage file names. - Removed unused functions: - xbt/file.h: xbt_basename(), xbt_dirname(), xbt_getline() - xbt/graph.h: xbt_graph_edge_get_length(), xbt_graph_edge_set_length, xbt_graph_export_graphviz() - xbt/str.h: xbt_str_join() - Remove unused datatypes: - xbt/heap.h: use std::priority_queue or boost::heap instead - xbt/swag.h: use boost::intrusive::list instead PLUGINS: - New link_energy plugin for the consumption of the links. - All of the operations on files and storage contents have been packaged into a plugin (src/plugins/file_system). The current public interface can be found in include/simgrid/plugins/file_system.h To use these functions you now have to initialize the plugin by calling MSG_storage_file_system_init() just after calling MSG_init() or sg_storage_file_system_init() just after creating the Engine. XML - Remove the undocumented/untested tag TRACE - Remove viva specific tracing as the tool is no longer maintained Fixed bugs: - #248: Kill of finished processes leads to segfault - #240: xbt_cond_wait_timeout should gracefully return for C apps - #239: please implement signal s4u::onDeadlock() - #230: segfaults when exit() without run() - #225: s4u::Actor::kill() doesn not really kill victims in a join() - #223: MSG_task_get_flops_amount() not working with parallel tasks - #222: Actor::kill() doesn't really kill and segfaults - #221: odd LMM warning when killing an actor - #120: Memory leak when the processes are forcefully killed SimGrid (3.17) Released October 8 2017 The Drained Leaks release: (almost) no known leaks despite the tests. Even very long simulations will preserve your memory: our testsuite stresses SimGrid for over 45mn on fast machines for a coverage of over 80%, and there is only one single known leak, of about 4kb. S4U - Comm.detach(): start and forget about asynchronous emission. A cleanup handler may be given to free resources if the comm cannot be completed. - this_actor::send(mailbox) is now mailbox->put() - New: simgrid::s4u::Comm::wait_all() - New: Host.execute() for remote executions. SURF - Unused option network/sender-gap is removed. MSG - Deprecate MSG_task_isend_with_matching(): This unused feature really complicates our internals. Will be removed in v3.20. Simix - Improved context termination. It is now possible to free resources, even when a process is forcibly killed. - ContextBoost: add support for Boost versions above 1.61. XBT - Bring back run-time option --cfg=exception/cutpath to remove exception backtraces. - Removed unused functions: - xbt/str.h: xbt_str_split_str(), xbt_str_subst(), xbt_str_ltrim(), xbt_str_rtrim(), xbt_str_trim(). - xbt/xbt_os_thread.h: xbt_os_thread_cancel(), xbt_os_thread_detach(). Misc - Removed header files obsolete since SimGrid 3.12: msg/datatypes.h, msg/msg.h, simdag/datatypes.h, simdag/simdag.h. - Fix many bugs: - #3: SD_exit should be made optional - #120: Memory leak when the processes are forcefully killed - #159: Threading test regression in Actor refcounting - #170: simgrid::s4u::Comm::wait_any() returns too many comms - #185: simgrid::s4u::Engine::instance()->shutdown() segfaults - #186: Actor::killAll() segfaults if some process is blocked on wait() - #191: VM migration and pstate - #192: Updating the energy consumptions of all hosts crashes with VMs - #195: All actors have PID=0 in the logs - #204: Sometimes segfault with thread contexts and mmap privatization - #222: Actor::kill() doesn't really kill and segfaults - #225: Actor::kill() doesn't really kill when victims are doing a join() ---------------------------------------------------------------------------- SimGrid (3.16) Released June 22. 2017. The Blooming Spring Release: developments are budding. Portability status: - FreeBSD: Disable SMPI mmap privatization, switch automatically to dlopen. - Mac, BSD: dlopen+thread broken, switch automatically to raw contexts. - Java JAR file should be finally fully working on Mac OSX too. XML platforms: Switch to platform v4.1 format. * This is (mainly) a backward compatible change: v4 are valid v4.1 files - can be used as a synonym for the now deprecated - can be used as a synonym for the now deprecated - an be used as a synonym for the now deprecated - can be used as a synonym for the now deprecated - state_file and avail_file periodicity is now easier to express (check the documentation) the old behavior should still work. * Storage: not backward compatible, but it seems that nobody ever used it - Remove attribute 'content_type' of : was never used - Make attribute 'model' of optional: for future usage - Remove Bconnection model property: was never (in)validated. Replaced by the maximum of the read and write bandwidth as a resource constraint. SimDag - New and Backwards Compatibility break: SD_simulate_with_update (double how_long, xbt_dynar_t changed_tasks_dynar) When one wants to get the list of tasks whose states have changed during a simulation round, s/he has to allocate and free a dynar and use it as argument to this function. The former SD_simulate (double how_long) now returns void. Virtual Machines - Allow multicore VMs, with the correct sharing (unless you overcommit) BUG: vCPU overcommitting still leads to buggy sharing, though. WIP. Energy - New function to update the consumption of all hosts at once. - Fix the model for multi-core hosts, linear in the amount of busy cores with an abnormality for pIdle (see paper). BUG: Still not satisfactory for ptask on mono-cores. MSG - The netzone are now available from the MSG API. The old names still work, but are now deprecated. SMPI - New algorithm to privatize globals: dlopen, with dynamic loading tricks - New option: smpi/keep-temps to not cleanup temp files - New option : smpi/shared-malloc-blocksize . Relevant only when global shared mallocs mode is used, allows to change the size of the fake file used (default 1MB), to potentially limit the number of mappings for large runs. - Support for sparse privatized malloc with SMPI_PARTIAL_SHARED_MALLOC() - Fortran ifort and flang compilers support - New RMA calls supported (experimental) : - MPI_Win_allocate, MPI_Win_create_dynamic, MPI_Win_attach - MPI_Win_detach, MPI_Win_set_info, MPI_Win_get_info - MPI_Win_lock_all, MPI_Win_unlock_all, MPI_Win_flush - MPI_Win_flush_local, MPI_Win_flush_all, MPI_Win_flush_local_all - MPI_Op_commutative, MPI_Fetch_and_Op, MPI_Compare_and_swap - MPI_Rput, MPI_Rget, MPI_Raccumulate, MPI_Rget_accumulate S4U - New: this_actor::isMaestro() returns whether we are in kernel mode. - Behind the scene, ActivityImpl are now automatically refcounted. Removing this nasty bottleneck will greatly simplify our internals. XBT - Replay: New function xbt_replay_action_get(): Retrieve the function previously associated to an event type. - DROPPED FUNCTION: xbt_str_varsubst() - DROPPED MODULE: strbuff. We don't need it anymore. - DROPPED MODULE: matrix. We don't need it anymore. - DROPPED MODULE: lib. We don't need it anymore. -- Thu Jun 22 22:48:12 CEST 2017 -- Da SimGrid team SimGrid (3.15) Released March 22. 2017 The Spring Release: continuous integration servers become green We fixed even the transient bugs on all target architectures: Linux (CentOS, Debian, Ubuntu, Fedora), Mac OSX (Mavericks, El Capitan) Windows, FreeBSD, NetBSD. - Rename NetCards to NetPoints. This was intended to help NS3 users, but that's not a netcard. That's a point in the routing algorithm, let's avoid wrong simplifications. MSG - New: MSG_process_yield(). Stop and yield to other processes. - New: MSG_process_daemon(). Daemon processes are automatically killed when the last non-daemon process terminates - New: MSG_process_ref/unref(). Fiddle with the process refcounting. - Renamed MSG_energy_plugin_init() -> MSG_host_energy_plugin_init() to make room for the upcoming network energy plugin. - Drop MSG_host_get_current_power_peak: duplicates MSG_host_get_speed Java - Ensure that an actor can kill itself with Process::exit() - Kill the obscure NativeException. Nobody want to survive the issues it denotes, so use JniException that is a RuntimeException (not to be caught explicitly). - Partial bug fix in initialization. SimGrid flags on command line were consumed at C level but stayed in the original Java String[] args. This could mess users' args[i] if SG flags were not put at the end of the command line. The SimGrid flags are now removed from the Java arguments. However, the number of arguments REMAINS UNCHANGED. It is then UNSAFE to test if args.length is greater than the number of YOUR OWN ARGUMENTS. It might be if you have --log or --cfg flags in the command line. - Fix numerous memleaks all around the place. In particular, around VMs. S4U - New callbacks: - s4u::onPlatformCreated: right before the simulation starts - s4u::onSimulationEnd: right after the main simulation loop - s4u::onTimeAdvance: right after a clock change (time is discrete) - s4u::Host::onSpeedChange: when the pstate is changed, or when an event from the availability_file changes the avail speed. - Links are now usable from s4u - New: Engine::hostList() and Engine::hostCount(). Still clumsy. - New: Actor::suspend(), Actor::resume(), and Actor::migrate(new_host) - New examples: The conversion of MSG examples to S4U has begun - Actors: create, kill, migration, and suspend - Applications: master-worker and token-ring - Action replay: communications and storage - Drop Host::getPstateSpeedCurrent() which duplicates Host::speed() SimDag - Backwards Compatibility breaks - The SD_route_get_list and SD_route_get_size functions have been removed. They are replaced by the unique void sg_host_route(sg_host_t src, sg_host_t dst, xbt_dynar_t links) The route from src to dst is built in the links dynar whose size can be obtained with xbt_dynar_length. - The SD_route_bandwidth and SD_route_latency functions have been replaced by sg_host_route_bandwidth and sg_host_route_latency. Macros ensure the backwards compatibility, but you should fix your code SMPI - Major C++ rewrite ongoing (SMPI used to be C compiled in C++). This can break codes using SMPI internals (from private.h instead of the public smpi.h). - Bump our claim of support from MPI 1.1 to MPI 2.2. We don't support 100% of it, but it should be enough. Contact us if not. - MPI_Win_lock/unlock, MPI_Get_accumulate support added (as for all RMA, implementation is naive and probably inaccurate) - New algorithm for SMPI_SHARED_MALLOC: global, where all blocks are mapped onto a unique small file using some system magic. - Bugfix: smpirun was sometimes misusing hostfile when wrongly terminated - Fortran: cleanups, fixes, support of user-added operations - RMA: MPI_Accumulate are applied in correct order. - RMA: MPI_Win_{complete/post} shouldn't miss messages anymore. - Better support for MPI_IN_PLACE - Support for MPI_Win attrs and keyvals. - Support MPI_Comm_group_create, MPI_Type_size_x - MPI ops should be properly applied to all allowed types - Replace implementation of MPI_Bcast "scatter followed by rdb allgather" algorithm by a non failing one from mpich. XBT - Kill the fifo data container: we don't use it anymore. XML - A can now be created with different speed values to represent pstates. see examples/platforms/energy_cluster.xml for an example. SimGrid (3.14.159) Released December 28. 2016 The previous release was only a crude approximation of a Pi release; this one should be a bit better. - Revert a last minute change that broke on Mac OSX. - Fix the changelog and NEWS files. - Improve the documentation, in particular of the routing module. - Integrate some patches flying around in Debian. SimGrid (3.14) Released December 24. 2016 Documentation * The whole documentation was reworked and reorganized. There is still much room for improvement here, but we're on it. Infrastructure * We now need python3 for our testing infrastructure. * Model checking needs libevent MC * Now works on FreeBSD SMPI * Privatization now works on FreeBSD * Privatization is now activated by default in smpirun You can turn it off with -no-privatize if something goes wrong. * Call-location tracing for SMPI. You can add the exact location (filename / linenumber) of an MPI call to your trace files and slow down or speed up the simulation between two consecutive calls by using an adjustment file (see the documentation). * Fixed computation of timings for MPI_Send, MPI_Recv & possibly also others We've found a bug that prevented SMPI to account for MPI_Send, MPI_Recv and others (in some cases) in a correct way. That is, the smpi/os, smpi/or values were ignored in some cases. The timings of these functions can now be significantly different. * smpi/cpu-threshold:-1 should become smpi/simulate-computation:no smpi/running-power is renamed to smpi/host-speed * smpi/grow-injected-times option to enable or disable multiplication of the timings injected in MPI_Iprobe, or MPI_Test. Enabled by default, which can make simulation less precise (but also much faster). * smpirun script should be (much) faster for large deployments. * SMPI tracing : fixed issue with poor matching of send/receives. * Replay : Fix broken waitall New functions and features * MSG_parallel_task_execute_with_timeout, to timeout computations. Dropped / renamed functions and features * msg_mailbox_t and associated functions. Use s4u::Mailbox instead. - MSG_mailbox_is_empty() -> Mailbox::empty() - MSG_mailbox_front() -> Mailbox::front() - MSG_mailbox_get_by_alias() -> simgrid::s4u::Mailbox::byName(name) - MSG_mailbox_get_task_ext() -> MSG_task_receive_ext() - MSG_mailbox_get_task_ext_bounded -> MSG_task_receive_ext_bounded - MSG_host_(get/set)_params -> MSG_vm_(get/set)_params * Don't pass the free_f parameter to property related functions: - MSG_host_set_property_value() - MSG_as_router_set_property_value() - MSG_storage_set_property_value() * VM properties. Since msg_vm_t are msg_host_t, just use MSG_host_get_property_value() and friends * VM I/O related things: - Ignored parameter of vm_create: core_nb, disk_path and disk_size. - Unimplemented save/restore methods * MSG_as_router_get_property_value() was redundent with MSG_environment_as_get_property_value(). - Removed MSG_as_router_*propert*() functions - Added MSG_environment_as_set_property_value() for consistency * xbt heterogeneous dictionnaries (created with xbt_dict_new()). Well, they are still there for now, but deprecated with a warning. Please switch to xbt_dict_new_homogeneous() before this is removed for real. * Task affinity. Its intended behavior (that was very badly tested and probably not really working) was deceiving what most users would have hoped here. * xbt_os_sem_get_value: unused internally, deprecated on OS X El Capitan * Option network/coordinates is now useless and should be dropped. Storage: * Added option storage/max_file_descriptors to allow more than 1024 files opened SURF: * Added option maxmin/concurrency_limit to allow more than 100 processes per host * Added Dragonfly topology support XML: * Add Exa- and Peta- units such as EiB, EB, Eib, Eb for size, and EiBps, EBps, Eibps, Ebps for bandwidth. They may become useful to some lucky ones. Java: * New functions: msg.Comm.waitAll() and msg.Comm.waitAny() * ex/app_tokenring: new example, very similar to the MSG Token Ring * ex/async_waitAll: new example, on asynchronous communications MSG: * Memory usage should be decreased for simulations with a large number of processes. This also helps for SMPI. SimGrid (3.13) stable; urgency=low The Half Release, a.k.a. the Zealous Easter Trim. - We removed half of the lines, that were mostly experimental cruft. v3.12 lasted 286000 lines of code, v3.13 is only 142000 lines (not counting blanks and comments -- according to openhub.net) - The internals are now compiled in C++ (and will soon be clean C++) - We removed 75 klines of XML, 12 klines of Java, 5 klines of cmake, 59 klines of C, etc. We added only 29 klines of C++ in replacement. * Backwards Compatibility breaks - Removed Lua simulation bindings (switch to C or Java for that). Lua can still be used to describe platforms - Removed Java kernel plug-ins. Will be reintroduced after the ongoing major internals reorg. - In MSG - the following functions were removed. They were too specific and should be reimplemented in a generic way, with filter function. - MSG_task_listen_from_host - MSG_mailbox_get_count_host_waiting_tasks - MSG_mailbox_put_with_timeout was removed. Please use MSG_task_send_with_timeout instead. - In SimDag - the SD_application_reinit function was removed. It has been a noop for a while. - The ACCESS_MODE of SD_workstation has been removed. This feature was not really usable and should soon be replaced by a more flexible mechanism. - The following functions thus do not exist anymore - SD_workstation_get_access_mode - SD_workstation_set_access_mode - SD_workstation_get_current_task - Basic estimation functions have been removed but can easily be replaced - SD_route_get_communication_time => SG_route_get_latency() + amount / SD_route_get_bandwidth() - SD_workstation_get_computation_time => amount / sg_host_speed() - In Java - VM.setBound(int load) is now VM.setBound(double bound) to meet the MSG semantics. Use VM.getSpeed()*load/100 for the legacy behavior. - In CMake - option enable_tracing was removed. It was not doing anything for a while. - In the ModelChecker: - the model-checker now ptraces the model-checked process which means you cannot use a debugger on the latter anymore (we might make this optional in the feature); - removed soft-dirty page tracking; - remove model-checked side snapshot management, MC_snapshot() and MC_compare_snapshot(); - keep the MC_cut() function as a stub (it was not really working in the previous release). Options: * All options are consistently in kebab-case. Old names are kept as alias. XML platforms: * Switch to platform v4 format. - Rename from 'power' to 'speed' the attributes describing the amount of flops that a , , or can deliver per second. - In , attribute kind="POWER" is now kind="SPEED". - In and , attributes availability and state are gone. It was redundent with state and availability traces, and with peak values. - In , attributes availability_file and state_file are gone. It was too complex and unused. - Kill . Was not doing anything. - The DOCTYPE points to the right URL: http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd (the file at this address now documents the changelog since its v1) - A warning is emitted for unit-less values (they are still accepted). - speed. Default: 'f' or 'flops'. Also defined: 'Yf', 'Zf', 'Ef', 'Pf', 'Tf', 'Gf', 'Mf', 'kf' 'yottaflops', 'zettaflops', 'exaflops', 'petaflops', 'teraflops', 'gigaflops', 'megaflops', 'kiloflops' - bandwidth. Default: 'Bps' bytes per second (or 'bps' for bits but 1 Bps = 8 bps) Also defined in bytes: 'TiBps', 'GiBps', 'MiBps', 'KiBps', 'TBps', 'GBps', 'MBps', 'kBps', 'Bps' And the same in bits: 'Tibps', 'Gibps', 'Mibps', 'Kibps', 'Tbps', 'Gbps', 'Mbps', 'kbps', 'bps' - latency. Default: 's' second. Also defined: 'w' week, 'd' day, 'h' hour, 'm' minute, 'ms' millisecond, 'us' microsecond, 'ns' nanosecond, 'ps' picosecond * bin/simgrid_update_xml can upgrade your files automatically (won't convert unit-less values) tools/sg_xml_unit_converter.py may help (but it's just a warning and will probably ever be). S4U * s4u::Host is now the preferred public interface to the Host features. sg_host_* functions are C bindings to the exact same behavior MSG_host_* and SD_workstation_* are #define to the sg_host_* ones MSG * The examples were completely reorganized (in C and Java), for your browsing pleasure. * Kill all deprecated functions (the ones you had when declaring MSG_DEPRECATED). They were deprecated since a few years, and probably did not even compile anymore. SimDag * The API has been profoundly modified to directly use the core objects instead of redefining its own. SD_Workstation_t and SD_link_t are now sg_host_t and sg_link_t respectively. Some functions have also been renamed for consistency. Backward compatibility is maintained, but users are encouraged to update their codes. A list of the modified functions can be found at the end of include/simgrid/simdag.h Simix * simgrid::simix::kernelImmediate() is the closure callback. It ensures that the lambda or closure passed as a parameter will run in kernel mode. All the callback functions should be rewritten to that interface at some point. Surf * Reorganizing and cleaning the internals all around the place. SMPI * Remove old default barrier/bcast buggy algorithms (see #18407) * Various bug fixes to handle more codes * Remove the need for the --foreground option of smpirun (it is still accepted for backward compatibility). XBT * Kill the setset data container: MC don't use it anymore. * Kill the queue data container: it made more sense with GRAS. * Kill the xbt_peer_t data type: it's useless without GRAS. * Kill rm_cb feature of config sets: it was never useful. * Kill graphxml parsing feature. It was not used. * Kill the deprecated code protected by XBT_USE_DEPRECATED * New functions: - xbt_dynar_sort_strings(), when the content is char* - xbt_str_parse_int / xbt_str_parse_double, wrapping strtol/strtod They throw exceptions on invalid input; * C++ support for declaring CLI flags (simgrid::config::Flag); * class for abstracting different signal backends (simgrid::xbt::signal). with no external dependencies (we need very simple signals). MC * refactoring and cleanup of the code; * ongoing process to cleanly separate the model-checking algorithms from the code model-checking support. -- Wed Apr 27 21:00:53 CEST 2016 Da SimGrid team SimGrid (3.12) stable; urgency=low The Facelift Release. Build System * Require g++ v4.7 at least to not speak prehistorical C++. * Require Boost 1.48 (for signal2 component). * Java must be version 7 at least when activated. * Builds on Windows again (including Java bindings). * Tracing is now always enabled (no way to turn it out) * Remove GTNetS. It was not working anyway. * Various cleanups in the cmake scripts. * Move headers around to sort them out on installed systems: - instr/instr.h -> simgrid/instr.h - instr/jedule/* -> simgrid/jedule - simdag/datatypes.h was removed - simdag/simdag.h -> simgrid/simdag.h - msg/datatypes.h was removed - msg/msg.h -> simgrid/msg.h cMSG: * Interface improvement: - Rename MSG_host_is_avail(h) to MSG_host_is_on(h) - Sanitize the interface in MSG_task_ module: - Merge two functions that were close enough but misleading: set_compute_duration(t) -> set_flops_amount(t) get_remaining_computation(t) -> get_flops_amount(t) - set_data_size(t) -> set_bytes_amount(t) get_data_size(t) -> get_bytes_amount(t) - Massive cleanups in the functions related to the energy - MSG_host_get_pstate_number() -> MSG_host_get_nb_pstates() - New: MSG_host_get_pstate() * New example: - msg/energy/onoff: switching hosts on and off jMSG: * Interface improvement: - Rename Host.isAvail() to Host.isOn() - Rename Process.currentProcess() to Process.getCurrentProcess() - Rename Task.setDataSize() to Task.setBytesAmount() - Merge Task.getRemainingDuration() and Task.getComputeDuration() into Task.getFlopsAmount() * Bug fixes: - #18874: Actually allows the GC to reclaim tasks SIMIX: * New functions - SIMIX_process_throw: raises an exception in a remote process * Refactoring: Separate sub-modules - libsmx: the public interface, as libc in a real system - popping: the strange dance that converts a user request into a kernel handling - smx_context_*: the virtualization mechanisms that embed the user code - smx_*: the handling of each simcalls * Interface cleanup: - simcall_host_set_power_peak_at -> simcall_host_set_pstate * Rename smx_action_t into smx_synchro_t, making explicit that these things are used to synchronize processes with their environment. For example, a communication is a sort of synchronization involving the communicating processes (that may block until the exchange) and the platform. The same can be said from computations, etc. * Bug fixes: - #18888: segfault when a process ends before its kill_time SMPI: * New functions - Onesided early support for : MPI_Win_(create, free, fence, get_name, set_name, get_group), MPI_Get, MPI_Put, MPI_Accumulate, MPI_Alloc_mem, MPI_Free_mem. - MPI_Keyval*, MPI_Attr* functions, as well as MPI_Comm_attr*, MPI_Type_attr* variants (C only, no Fortran support yet) - MPI_Type_set_name, MPI_Type_get_name - MPI_*_c2f and MPI_*_f2c functions - MPI_Info_* functions (beware, get_nthkey may not follow the insertion order) - MPI_Pack, MPI_Unpack and MPI_Pack_size functions - Activate a lot of new tests from the mpich 3 testsuite * Features - Constant times can be injected inside MPI_Wtime and MPI_Test through options smpi/wtime and smpi/test - InfiniBand network model added : Based on the works of Jerome Vienne http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf - When smpi/display_timing is set, also display global simulation time and application times - Have smpirun, smpicc and friends display the simgrid git hash version on --git-version * Collective communications - SMP-aware algorithms are now dynamically handled. An internal communicator is created for each node, and an external one to handle communications between "leaders" of each node - MVAPICH2 (1.9) collective algorithms selector : normal and SMP algorithms are handled, and selection logic is based on the one used on TACC's Stampede cluster (https://www.tacc.utexas.edu/stampede/). - Support for Rabenseifner Reduce/Allreduce algorithms (https://fs.hlrs.de/projects/par/mpi//myreduce.html) * Replay - Replay now uses algorithms from wanted collective selector - Replay can be used with SMP-aware algorithms - Memory occupation of replay should now be contained (temporary buffers allocated in collective algorithms should be shared between processes) - Replay can now replay several traces at the same time (check examples/smpi/replay_multiple example), to simulate interactions between several applications on a given platform. User can specify the start time of each instance. This should also allow replay + actual applications to run. * Bug fixes - [#17799] : have mpi_group_range_incl and mpi_group_range_excl better test some corner cases - Correctly use loopback on fat-tree clusters - Asynchronous small messages shouldn't trigger deadlocks anymore * Energy/DVFS cleanup and improvement - smpi_set_host_power_peak_at() -> smpi_set_host_pstate() - new: smpi_get_host_pstate() SURF * Bug fixes - "Full" network optimization flag was broken since Surf++ - Better handling of precision flags in maxmin - Fix bug causing sometimes "Impossible" errors - Properly pass cluster properties to included hosts * Improvement of the Energy plugin. - Always update the consumption before returning that value - New property: watt_off to denote the dissipation when the host is off - New functions getWattMinAt and getWattMaxAt to retrieve the dissipation of pstates that we are not currently at. * Java: class NetworkLink renamed to Link * New function: simcall_process_get_kill_time() * Massive rename s/workstation/host/ - That's intrusive, but that's good for the project consistency. Sorry. - Change config option "workstation/model" into "host/model" XBT * New functions - Add a xbt_heap_update function, to avoid costly xbt_heap_remove+xbt_heap_insert use - Add a xbt wrapper for simcall_mutex_trylock (asked in [#17878]) - Add two new log appenders : rollfile and splitfile. Patch by Fabien Chaix. - xbt_dirname and xbt_basename for non-POSIX systems MC * The model checker now runs as a separate process. * The model checker runs must now be launched with the new simgrid-mc program. * Record/Replay: the MC can display a textual representation of a path in the execution graph. It can then be replayed outside of the model checker. -- Mon Oct 12 06:02:41 CEST 2015 Da SimGrid team SimGrid (3.11) stable; urgency=low The Class Release. Tools: * Normalizing pointers addresses tool for better diff between logs Examples: * Add cloud examples using new VMs - examples/msg/cloud/two_tasks_vm.tesh - examples/msg/cloud/simple_vm.tesh - examples/java/cloud/cloud.tesh - examples/java/cloud/migration/migration.tesh * Add java surf examples: - examples/java/surfPlugin/surf_plugin.tesh - examples/java/reservationSurfPlugin/reservation_surf_plugin.tes - examples/java/surfCpuModel/surf_cpu_model.tesh * Add SMPI+MSG example: - examples/smpi/smpi_msg_masterslave/ TeshSuite: * Add tests: - msg process test - msg task destroy cancel test - msg_host on/off test * Move all tests in testsuite to teshsuite (adding tesh files) * Restructure teshsuites - one folder for each kind of test * Restructure AddTests.cmake - unify tests names - structure the order of tests (with sections) MSG: * Add virtual machine - creation of a VM on a PM - migration of a VM from a PM to another PM * New functions - MSG_process_join(msg_process_t process, double timeout) - msg_bar_t MSG_barrier_init(unsigned int count) - int MSG_barrier_wait(msg_bar_t barrier) - void MSG_barrier_destroy(msg_bar_t barrier) - msg_as_t MSG_environment_as_get_by_name(const char * name) * New option "msg/debug_multiple_use" to help debugging when a task is used several times * MSG IO - Improvements and finalization of MSG_storage, MSG_file APIs and their documentation - Increase code coverage in test suites - Bug fixes SIMIX: * Protect context stacks against stack overflow. The number of protected memory pages allocated on the top of each stack (1 by default) can be configured with the new command line option --cfg=contexts/guard_size:. * Simcalls are now generated by a python script that - generates files included by SimGrid - checks that all the functions exist, and proposes prototypes * Clean simcalls - remove sem_destroy, file_set_data, comm_destroy, vm_set_state, host_set_data, host_get_data * New simcalls - simcall_process_join(smx_process_t process, double timeout) * Fix bug where sleeping processing could not be suspended. SURF: * Translate surf models from C to C++ - Generic classes for all models: Model, Resource, Action - A generic interface for each kind of model (CPU, Network, Storage Workstation, WorkstationVM) - C bindings * Translate surf routings from C to C++ * Add callbacks using sigc++ or boost::signals2 - Add callback functions for resource creation/destruction - Add callback functions for action state change - Handle Energy as a plugin * Replace swag by boost::intrusive * Add new routing models for clusters. For documentation, see http://simgrid.gforge.inria.fr/simgrid/latest/doc/platform.html#pf_cluster - tori, with topology="TORUS" and topo_parameters="ndim1,ndim2,...,ndimn" parameters for cluster tag - Fat trees, with topology="FAT_TREE" and topo_parameters="h;m1,...,mh;w1,...,wh;p1,...,ph" parameters for cluster tag - see examples/platforms/torus_cluster.xml and examples/platforms/fat_tree_cluster.xml * More documentation SMPI: * Hostfiles support host:nb_processes construct to deploy several processes on one node. * Collective communication algorithms should not crash if used with improper number of nodes and report the error. * SMPI now partially supports MPI_Topologies : MPI_Cart_create, MPI_Cart_shift, MPI_Cart_rank, MPI_Cart_get, MPI_Cart_coords, MPI_Cartdim_get, MPI_Dims_create, MPI_Cart_sub are supported. * New interface to use SMPI programmatically (still depends on MSG for some parts, see examples/smpi/smpi_msg_masterslave) : - SMPI_app_instance_register(const char *name, xbt_main_func_t code, int num_processes) - SMPI_init() - SMPI_finalize(); * Global variables privatization in MPI executables is now performed at runtime with the option smpi/privatize_global_variables (default:no). Limitations : Linux/BSD only, with mmap enabled. Global variables inside dynamic libraries loaded by the application are not privatized (static linking with these libraries is advised in this case) Tracing: * Options defined in XML work correctly now. Java: * New cmake option, enable_lib_in_jar, to control whether native libraries are copied into simgrid.jar or not (ON by default). Use this option if you want to reduce the size of the installed simgrid.jar, *and* the native libraries are kept installed elsewhere. * Surf binding with SWIG (code generated in maintainer mode only): - plugin to handle callbacks - CPU model only for the moment Build System: * Supernovae build mode is definitively removed. It was used to improve inlining and inter-module optimizations. It is nowadays superseded by link-time optimizations commonly available in compilers. * Update ns-3 find lib. Bindings for ns-3 should work again now. * Add boost dependency for surf++ * Add new macro for tests - ADD_TESH(name ) - ADD_TESH_FACTORIES(name "thread;ucontext;raw" ) XBT: * New functions - xbt_bar_t XBT_barrier_init(unsigned int count) - int XBT_barrier_wait(xbt_bar_t barrier) - void XBT_barrier_destroy(xbt_bar_t barrier) * Make the xbt_os_time module public -- Sat May 31 22:39:38 CEST 2014 Da SimGrid team SimGrid (3.10) stable; urgency=low The Clean Diaper Release, a.k.a. SimGrid is leak-free. Java: * Reintegrate Java to the main archive as desynchronizing these package is not acceptable anymore (Java is now considered stable) * Add explicit synchronization facilities through semaphores * Bug fix: Task.setDataSize() only changed the C world, not the value cached in the Java world MSG: * Dramatically change the way files are handled. API and internals changed, but this part of MSG was not considered as production grade either. * Add explicit synchronization facilities through semaphores * Add a new function MSG_host_get_process_list() * Preliminary DVFS support (see examples/msg/energy/ for details) SMPI: * SMPI is now included directly in the libsimgrid as the windows linker doesn't force us on splitting it anymore. * Improvements of the SMPI replay tool: - Most of the collective communications are now rooted in the same process as in the original application. - Traces now rely on the same MPI data type as the application (MPI_BYTE was used until now). Multiple data types can now be used in a trace. - The replay tool now supports traces produce either by TAU or a modified version of MPE. - Bug Fix: the compute part of the reduce action is now taken into account. - Gatherv collective is now supported - SimGrid (SMPI for now) can generate replay traces as well. Option -trace-ti of smpirun outputs time independent traces for the current run. One file is created per process. If too many processes are simulated, this behavior can be changed to one file for all processes by using the tracing/smpi/format/ti_one_file flag * smpirun generates the host file if needed (with given host count and platform) * Integration of more than 100 STAR-MPI, MPICH, OpenMPI collective algorithms - allows to select one in particular with --cfg=smpi/coll_name:algorithm - allows to use the decision logic of OpenMPI(1.7) or MPICH(3.0.4) by setting --cfg=smpi/coll_selector:(mpich/ompi) * Support for new functions : MPI_Issend, MPI_Ssend, Commutative operations in Reduce * Add a --cfg:tracing/smpi/internals option, to trace internal communications happening inside a collective SMPI call. * Fix the behavior of complex data types handling. * Make MPI_Wtime another synchronization point to take computations into account. * Replace MPICH-1 test suite by the one from MPICH 3.0.4. Can be built using enable_smpi_MPICH3_testsuite flag in cmake. Run with ctest. * Add all missing Fortran bindings, SMPI should work with Fortran 90 (no privatization of global variables yet) * Preliminary DVFS support (see examples/smpi/energy/ for details) Model-Checking; * Verification of liveness properties is now available for SMPI applications (in addition to MSG applications) * Bugged examples using SMPI in examples/smpi/mc/ * Add --cfg=model-check/visited option. Allows the verification of infinite programs. Detection of loops in the execution thanks to the system state comparison and reduction of the state space to explore. Can be combined with DPOR for safety properties. SimDag: * Allow to change SimGrid configuration (see --help) within the code thanks to SD_config() as it can be done in MSG. * Add a new function SD_task_set_amount() upon user request. PLATFORM: * Handle units for values (10ms, 10kiloflops, 10Bps, 1GB, ...) * Remove rule based routing (no more PCRE dependency) * Add a limiter_link option to cluster tag, to specify a maximum reachable bandwidth in fullduplex mode when it is less than twice the nominal bandwidth. * Add a loopback_bw and loopback_lat options to cluster tag. * Fix the peer tag that could not be mixed with other AS within a Vivaldi routing. Now peers are encapsulated in an AS and have their own private router but this is transparent. XBT: * Our own implementation of getline is renamed xbt_getline, and gets used even if the OS provide a getline(). This should reduce the configuration complexity by using the same code on all platforms. * New type: xbt_cfg_elm_boolean. * Allow to use yes/no for boolean configuration options in the command line. * Allow to disable SimGrid cleanups at exit from command line option. There are situations where one may want a simulation to end with an exit. Unfortunately, calling exit may cause SimGrid to segfault, which is quite annoying when scripting around the simulator. Adding a --cfg=clean_atexit:no allows to circumvent this issue. Build System: * Lots of memory leaks were corrected in this release. There is no known memory leaks anymore, in all of our 600+ tests. * New command line option --version, to get SimGrid version information. Packagers may want to add extra words to SIMGRID_VERSION_EXTRA defined in CMakeLists.txt. * Supernovae builds are deprecated, and expected to be removed in the next version of SimGrid. -- Sun Nov 17 00:26:44 CET 2013 Da SimGrid team SimGrid (3.9) stable; urgency=low The Grasgory release: GRAS is really dead now. * Complete overhaul of the internal host structures scheme. GRAS: * If you use GRAS, you should stay at SimGrid 3.5 (at most) since it was considered as experimental and badly maintained since then. * Keeping it was thus a trap to our potential users, that could take it instead of MSG or SMPI by mistake despite is pity state. * GRAS seems to have very few users (if any), and no one volunteered to maintain it further. It also induces a lot of XBT code (for portability sake), that must be maintained too. * For all these reasons, we killed GRAS. If someone wants to revive it in the future, don't cry, our git history still remembers of GRAS. Documentation: * Major overhaul. Merge our documentation again as time proved that splitting it was really not helping our users. * Further improve the developer documentation to help newcomers hacking on SimGrid itself. The user documentation (and in particular, the beginner documentation) is still in a sorry state. SMPI: * Now works on Windows too! * Much more extensive test suite, from MPICH SIMDAG: * Add a new loader (SD_PTG_dotload) that creates a parallel task graph (i.e., a DAG whose nodes are parallel tasks) from a dot file. Creates a dynar of SD_TASK_COMP_PAR_AMDAHL and SD_TASK_COMM_MXN_1D_BLOCK tasks. * Bug fix: let task be scheduled when the last dependency to be solved is a control dependency. * Remove SD_load_environment_script function. Use the C sg_platf function if you want to declare a platform programmatically. MSG: * New function: MSG_process_get_number() * Old function documented: MSG_config() * Remove MSG_load_platform_script function Use the C sg_platf function if you want to declare a platform programmatically. SURF: * Change the default value of the TCP_gamma constant (maximal size of TCP congestion window) to a more realistic 4MiB value. If you notice changes in your simulation results, you can fall back to the previous 20k tiny window by adding --cfg=network/TCP_gamma:20000 on command line. * (Hopefully) fix a bug wrt periodic availability and state traces * Bug fix: use default values at start when first event in availability/state trace is not at time 0. PLATFORM: * remove the "new_" part of function name sg_platf_new_trace_connect (resulting in sg_platf_trace_connect), since it does not create anything new XBT: * Kill synchronized dynars, and xbt_dynar_dopar(). We cannot think of a use case where it's really mandatory, and maintaining it was a pain in our code base. * New: xbt_fifo_search(), search an item with a user-provided comparison function instead of dumb pointer comparison. LUA: * Fix the lua deployment: Use `simgrid.init_application()` before deployment instead of `simgrid.msg_register_application()` after. TRACING: * Transfer the tracing files into the corresponding modules. -- Tue Feb 5 11:31:43 CET 2013 Da SimGrid team SimGrid (3.8.1) stable; urgency=low The "we are told that some people want to also *install* the simgrid framework" release. * Add missing file "tesh.1" to the archive. -- Sat Oct 27 16:12:11 CEST 2012 Da SimGrid team SimGrid (3.8) stable; urgency=low The Psssshiiiit release: SimGrid jumps into the Cloud. MSG: * Add an experimental interface to manipulate VMs. They are mainly process groups with very few intrinsic semantic, but they should allow you to build the semantic you want easily. * New function: MSG_host_set_property_value() * New function: MSG_process_on_exit(). To clean memory in all cases. * Bug fixes that made the host (and link) failures unusable. * Add a way to auto-restart process when the host in which they are executing comes back (ON_FAILURE="RESTART" on deployment file, MSG_process_auto_restart_set). * Use the "msg_" prefix for all datatypes (instead of m_, msg_ and MSG_), please stop using the old ones, they are DEPRECATED. * Deprecate functions MSG_global_init() / MSG_global_init_args() Please use MSG_init() instead. (reducing the amount of entry points in the library helps us). * Make it impossible to link against the wrong version of the lib * Deprecate MSG_clean(). No need to call it anymore. * Function MSG_get_host_number() is not deprecated anymore. Documentation: * Split the doc into a user guide and a reference guide. * Start a developper guide to help people hacking on SimGrid. Cmake: * Enable tracing by default. This modules rocks you should use it. * Remove option custom_flags. Now use environment variables CFLAGS and LDFLAGS. * Use default cmake things to detect lua instead of home grown ones. * New option "enable_mallocators" to disable mallocators, for debugging purpose ("on" by default). SIMIX: * Bug fixes around the resource failures: don't let the processes survive the host they are running onto. * Add an interface to auto-restart processes when the host in which they are executing comes back. * Ensures that SIMIX_clean is called automatically. It's not part of the public interface anymore (bindings should be updated). SimDag: * Bug fix for when SD_Simulate is called with a positive value: be careful when comparing doubles. Sometimes they are different for non significant digits only. * New types of typed tasks. SD_TASK_COMP_PAR_AMDAHL represents a parallel task whose initial work is distributed among host according to the Amdahl's law. Such tasks are created with a parameter alpha that corresponds to the non-parallelizable part of the computation. SD_TASK_COMM_PAR_MXN_1D_BLOCK represents a complex data redistribution between two sets of workstations assuming a 1D block distribution (each workstation owns a similar share of data) on both sides. These tasks can be scheduled with SD_task_schedulel or SD_task_schedulev. Data redistribution will be automatically scheduled once parent and child are both scheduled. The filling of computation_amount and communication_amount structures is now done seamlessly thanks to the chosen assumptions. * New function SD_workstation_dump to display various information * New function SD_task_set_rate to throttle the bandwidth allowed to be used by a SD_TASK_COMM_E2E typed task. This rate depends on both the nominal bandwidth on the route onto which the task is scheduled and the amount of data to transfer. To divide the nominal bandwidth by 2, the rate then has to be : rate = bandwidth/(2*amount) * Compute tasks that have failed can now be rescheduled and executed again (from their beginning) * Increasing source code coverage (src/simdag is now covered at 95.8% on average) SMPI: * Re-implement time-independent trace replay using SMPI (at the smpi_smp_* level) instead of MSG. This should replace examples/msg/actions/actions.c * Implement support of MPI Datatypes (vectors, hvectors, indexed, hindexed and structs) * Implement the exchange of non-contiguous data. [Khalid Hasanov & Jean-Noel Quintin] Thanks for the patch, guys. * Correct behavior of smpi/sender_gap and set its default value to 0 * Add option to asynchronously send small messages to allow better simulation of pt2pt communications. --cfg=smpi/async_small_threshold:value specifies the size in bytes under which messages will be asynchronously sent. * Add support of MPI_Iprobe, MPI_Probe, MPI_Testall, MPI_Wtick functions * SMPI now handles more MPI specific values in input. Closes [#14389] and [#14388] SimGrid: * New C interface to define a platform: XML is now optional. For more info, please check include/simgrid/platf.h. * New interface to define random platforms from the C: For more info, please check include/simgrid/platf_generator.h and examples/msg/masterslave/masterslave_platfgen.c * Export a sg_cmdline dynar containing all the arguments we got from the command line. TRACE: * Two new tracing options for adding comments to trace file so you can track your experiments (see --help-tracing for details). * New option to generate a impoverished trace file (--cfg=tracing/basic:1) * Adding the SimGrid version that generated the trace file as a comment. * Instrumenting other MSG functions (MSG_task_isend_with_matching and MSG_task_dsend) * Fix to avoid key clashes on Paje links * Other minor fixes related to the Paje specification XBT: * Functions xbt_dict_hash() and xbt_dict_hash_ext() are made public, and renamed to xbt_str_hash() and xbt_str_hash_ext(). * New function: xbt_os_timer_resume() to restart a timer w/o resetting it. * Greatly improve the robustness of mmalloc to user errors (such as using an area after freeing it, or freeing it twice) -- Thu Oct 25 17:30:06 CEST 2012 Da SimGrid team SimGrid-java (3.8.1) stable; urgency=low * New module: org.simgrid.trace.Trace (SimGrid trace bindings) Warning: all methods are visible, but only some of them are implemented so far. Check the source (src/jtrace.c) for further information. * New module: org.simgrid.msg.File (SimGrid File management functions) * New Module: org.simgrid.msg.VM (SimGrid interface to mimick IAAS clouds) * Change the meaning of Process.restart: now restart the process from the begining, like MSG_process_restart in C. * Add Process.setAutoRestart: handling of process restart when failed host comes back. * Add Process.getProperty, Host.getProperty, Host.getProperty: allows you to retrieve the properties of the processes/hosts * Deprecate Msg.clean(): you can just forget about it now. * New function Process.getCount(), that only works when compiling with the not yet released version 3.9 of the C library. * New context factory based on Coroutines. It mandates a modified JVM but then, the simulations run about five times faster, and there is no limit to the amount of processes (beside of the available memory). -- 2012-12-04 Da SimGrid team SimGrid (3.7.1) stable; urgency=low MSG: * Restore the prototype of MSG_process_create_with_environment() to the pre-3.7 situation by removing the kill_time argument. * Add a MSG_process_set_kill_time() function instead. SURF: * Fix weird behaviors when dealing with parallel tasks. WINDOWS: * Simgrid is now built as a dll. * Simgrid-java now works on Windows. * Simgrid-Java is now included into Windows package. MacOS: * First pre-build package for MacOSX. Build System: * Fix compilation when using MSG_USE_DEPRECATED. * Fix some compilation issues on Macs and Windows. * Reduce the number of failing tests on exotic systems, like Debian/Hurd. * Environment variables CFLAGS and LDFLAGS are now honored by cmake. We discovered that the Lua console is broken, but we are missing the manpower to fix it right now. The problem existed in 3.7 too, so we are not blocking the release for that. Sorry if you depended on this feature, any help would be really welcome. -- Thu Jun 7 2012 Da SimGrid team SimGrid-java (3.7.1) stable; urgency=low The "Java aint got to be bloated and slow" release Major cleanups: * Various internal cleanups and performance improvement Simulations are expected to run up to twice faster or so * Make Process.kill(process) an instance method, not a static one * User processes are not java.lang.Thread subclasses. This breaks the compatibility (sorry), but previous API was brain-dead, making it impossible to have non-trivial initializations in the process constructor. * Require a full constructor per Process sub-class. Kinda breaks the compatibility (sorry), but this allows a much more efficient way to launch the processes at simulation startup. * Do not embeed our version of semaphores, java 1.5 can be considered as sufficiently prevalent for us to not dupplicate its features. * Lot of bug fixes Extend the API: * Add examples for almost every part of the API We spotted and fixed a lot of bugs in the process * New module: asynchronous communication API * New function: Process.sleep() It takes milliseconds as argument, just as java.lang.Thread.sleep() * New module: org.simgrid.msg.Mutex (SimGrid mutexes) * New module: org.simgrid.msg.RngStream (RngStreams random generators) -- 2012-06-12 Da SimGrid team SimGrid (3.7) stable; urgency=low The "spring cleanups (before the next Big Project kicks in)" release. Models: * We can specify the SMPI latency/bandwidth factor with command line add --cfg=smpi/bw_factor:"threshold0:value0;...;thresholdN:valueN" or add --cfg=smpi/lat_factor:"threshold0:value0;...;thresholdN:valueN" You can also use the "config tag" from platform file by adding this line (see "example/platforms/tag_config.xml" to use "config tag"). Note that the command line supersedes the platform file configuration. * Change the correction factors used in LMM model, according to the latest experiments described in INRIA RR-7821. Accuracy should be improved this way. * Use the partial invalidation optimization by default for the network too. Should produce the exact same results, only faster. * Major cleanup in surf to merge models and split some optimization mechanisms from the core of the models. As a result you can now specify which model to use (e.g., --cfg=network/model:LV08 --cfg=cpu/model:Cas01) and which optimization mode to use (e.g., --cfg=network/optim:lazy --cfg=cpu/optim:TI). Incompatible combinations should err at initialization. See --help-models for the list of all models and optimization modes. * The CLM03 workstation model was dropped for simplicity because it used the deprecated CM02 network model. Use default instead. * Rename the TCP_gamma configuration option to network/TCP_gamma * Rename the coordinates configuration option to network/coordinates, and document it * Use now crosstraffic keyword instead of the terribly misleading fullduplex keyword. It is activated by default now in the current default model, use --cfg=network/crosstraffic:0 to turn it off. * Ongoing refactoring the model parsing to make XML files optional See include/simgrid/platf.h for details (still to be completed) MSG: * Major overhaul of the documentation. Almost instructive now :/ * Deprecate the use of m_channel_t mechanism like MSG_task_{get,put} functions and friends. This interface was considered as deprecated since over 2 years, it's time to inform our users that it is. Switch to MSG_task_{send,recv} instead, or compile SimGrid command line 'cmake -Dcustom_flags="-DMSG_USE_DEPRECATED" .' if you really need to use these (crappy) functions in your code. These functions will be removed soon. Stop using them now. * Deprecate MSG_get_host_{table,number} Implement MSG_hosts_as_dynar() instead. * Implement MSG_processes_as_dynar() (Closes gforge #13642) * Remove the public field msg_host_t->name. Use MSG_host_get_name() Simix: * Stabilize the parallel execution mode of user contexts * Introduce configuration variables to control parallel execution: - contexts/synchro: Synchronization mode to use when running contexts in parallel (either futex, posix or busy_wait) - contexts/parallel_threshold: Minimal number of user contexts that must be part of a scheduling round to switch to parallel execution mode (raw contexts only) * Fix bugs that prevented to use suspend/resume along with synchronization structures. * Fix bugs in process termination that lead to invalid memory access in very specific conditions. SURF: * Introduce a parallel mode for the models (controlled by surf/nthreads configuration item). In our tests, running the models in parallel never lead to any speedups because they are so fast that the gain of computing each model in parallel does not amortizes the synchronization costs, even when ultra fast futexes are used. This is released anyway because YMMV. SimDag: * Performance boost by using a swag internally to compute the set of tasks that are finished and should constitute the return value of SD_simulate. SMPI: * Enable it by default now that it is considered rather stable. TRACE: * Documentation of the tracing functions. * Performance gains when tracing categorized/uncategorized resource utilization by avoiding calls to get route when updating resource variables. LMM constraints are being used instead. * API changed to set task categories. Use MSG_task_set_category instead of TRACE_msg_set_task_category, and SD_task_set_category instead of TRACE_sd_set_task_category. They only work if ENABLE_TRACING is ON. * Bugfix for graphicator, routes not correctly obtained, memory leaks * Examples for link user variables added (see at examples/msg/tracing/) * Deprecated function TRACE_msg_set_process_category completely removed * Trace header updated according to the latest Paje file format * Tracing network lazy updates, no longer obligate users to use full updates * --cfg=tracing/platform:1 also registers power/bandwidth variables * Experimental: let user code declare/set/push/pop application states for hosts * API changed to allow the manual creation of graph configuration files for Triva. See TRACE_get_node_types() and TRACE_get_edge_types(). Lua: * Improve the API of Lua MSG bindings, using the Lua spirit. * Each simulated process now lives in its own Lua world (globals are automatically duplicated). It helps writing simulators. Will allow to run Splay programs within SimGrid in the future. * Add a Chord example in Lua, equivalent to the MSG one. MODEL-CHECKING: * Start the implementation of a solution to express temporal properties, not only local assertions. This is still an experimental work in progress, stay clear from it to be safe. XBT: * Logs: - Add new runtime parameters --help-logs and --help-log-categories to display information about supported logging parameters and categories. - Old deprecated parameters --{gras,surf,msg,simix,xbt}-log=... don't exists anymore. * Mallocators: allow value NULL for the reset function. * Dicts: - New function xbt_dict_new_homogeneous(void(*)(void*)) to create homogeneous dictionaries, where all the elements share the same free function. Non homogeneous dictionaries will be deprecated in the next release. - Dicts of scalar elements (xbt_dicti_*) are deprecated. - Multi-level dictionaries are deprecated. * Dynars: - new function xbt_dynar_search_or_negative() that is useful when you have less than 2 million elements in your dynar and don't want of the extra complexity of catching exceptions when the element is not found. * Portability layer - Make xbt_os_thread module (for thread portability) public. Documentation is still to come, sorry. * mmalloc module: - Cleanups and simplifications to make it maintainable again. - Exotic features (such as memalign and valloc) were removed. - The metadata were extended and improved so that the model-checker becomes able to explore and inspect the heaps. - This may induce a performance drop when enable_model-checking is ON in cmake (even if it's not used in the simulation), but it is necessary at this point to get MC working. Turn model-checking OFF if simulation performance matters to you. Not enabling it at runtime is not enough, disable it in cmake. -- Tue May 15 11:30:19 UTC 2012 Da SimGrid team SimGrid (3.6.2) stable; urgency=low The "Not coding new stuff allows to polish old things" release. General * New bindings to the NS3 packet level simulator (experimental) * Use the raw (efficient) execution contexts instead of the sysv (portable) ones when possible. * libpcre is now mandatory in any cases since not using it led to severe performance loss and possibly other issues * Update the XML platforms: - G5K: include the latest machine in Nancy - GridPP and LCG: new platforms * Documentation was partially updated, at least (more to come) Bug fixes, cosmetics and small improvements * Free terminated processes before the end of the simulation to avoid exhausting the memory of users having very dynamic amount of processes. * Bug fix and cosmetics about canceling non-running tasks * Bug fix about the dot loader's issues when using libcgraph Portability * Create an installer for windows with nsis (amd64 and win32) - Add an hello world project to illustrate simgrid project creation. - Embed libpcre into the Simgrid installer to avoid its compilation burden * The raw execution contexts should work on Apple now * Port to Windows 64 bits - Sysv contexts now have an implementation for this arch - GRAS communication features now support this arch * Drop support for borland compiler on windows - this code was not maintained, and we kinda depend on gcc nowadays * Fix portability issues on kfreebsd/gnu: build error about semaphores * Fix portability issue on unstable ubuntu: linker became picky on argument order -- Wed Oct 5 15:51:01 CEST 2011 Da SimGrid team SimGrid (3.6.1) stable; urgency=low The "Oops, we broke Macs too" release Portability * Fixed contexts detection so that raw ones are used when possible * On Mac, do not use Posix Ucontexts with gcc v4.[1-5] since this leads to a strange error, with user code segfaulting sometimes when the generated code is not perfectly aligned (which is not controllable from the user side, depends on the amount of code) XBT * New macro: CATCH_ANONYMOUS, which is like CATCH(e) but without argument. -- Mon Jun 27 13:59:03 CEST 2011 Da SimGrid team SimGrid-java (3.6) unstable; urgency=low * Initial release. * Split of every thing from simgrid v3.5 into a separate package. -- 2011-10-05 Da SimGrid team SimGrid (3.6) stable; urgency=medium The Summer Release, also known as the "OMG! They Killed Kenny!" version Java and Ruby: * Bindings now constitute their own package, separated from the main one. Rationale: reduce our maintenance nightmare by reducing the module coupling They will soon be released on their own on gforge. * In the meanwhile: svn co svn://scm.gforge.inria.fr/svn/simgrid/contrib/trunk/simgrid-java svn co svn://scm.gforge.inria.fr/svn/simgrid/contrib/trunk/simgrid-ruby GRAS: It is not considered as stable anymore, but experimental. Sorry. * It's not quite deprecated for now because we have no replacement, but it may soon become the case. SMPI * New MPI functions supported: MPI_Comm_disconnect, MPI_Comm_get_name * Fortran: New user-level cache variable to store the rank of the running process. This improves performance by an order of magnitude. * C: New coccinelle script to automatically locate and modify global and local static variables. * Improved SMPI network model with a sender-side gap to account for multiple parallel sends. MSG * New function MSG_comm_get_status(). MSG_comm_test() and MSG_comm_testany() only say if a communication is finished, no matter whether it succeeded or failed. You can call MSG_comm_get_status() to know the status of a finished communication. * New function MSG_task_dsend() to send a task and detach it. When a communication is detached, you are never notified of its success or failure and the memory is released automatically once it is finished. This function is useful when you don't care about the end nor the success of a communication. * Change the prototypes of action replay. Sorry for inconvenience, but this is really more efficient this way (and to adapt your code, you just have to fix the initialization, that shouldn't be too long) * Kill the braindead MSG_task_refcount_dec() function. I guess nobody ever managed to do anything useful with it. * New function MSG_comm_testany(). Similarly to MSG_comm_waitany(), it takes a dynar of communications. It returns immediately and gives the index of a finished communication (if any). * New example: a basic implementation of the Chord P2P algorithm. SURF * New model for multi-core CPUs. You can now use the core attribute to precise the number of cores of a host. This is a basic model. Every process running on the host receives at most the power provided in the DTD (throughput<=power). Total throughput of process cannot exceed power * num_cores. * New peer tag. This peer tag creates a tiny AS comprising a host and a router linked by an up-link and a down-link (possibly asymmetrical). This kind of pattern allows to easily build last-mile model style platforms. Aggregating such patterns in a rule-based AS is thus the technique of choice for modeling large peer-to-peer/volunteer computing/cloud platforms. * New model for Vivaldi routing. We transformed the Vivaldi network model into a Vivaldi routing model (based on the rule-based model). This allows to combine Vivaldi based latencies with last-mile platforms. SIMIX * Added a check for NaN of IEEE754 infinite in the double entries of the smx_user.c file * Introduce a new context factory "raw", highly inspirated from the ucontext factory, but using manually crafted functions in assembly to do the work in an efficient manner. * Allow to change the used context factory at run time, not only at compilation time. Use --cfg=contexts/factory:raw for maximal speed. * Add an option --cfg=contexts/stacksize:N to set the stack size of the user contexts at runtime (only with raw contexts or ucontexts). * Completely rewrote this module to allow parallel execution of user processes. Use --cfg=contexts/nthreads:N to execute user processes with N parallel threads (the default is 1, meaning no parallelism). * Allow to decide dynamically between sequential and parallel modes. When nthreads > 1, you can use --cfg=contexts/threshold:P to run the user processes in parallel only when their number is greater than or equal to P (the default is 2). * Added a check for NaN of IEEE754 infinite in the double entries of the smx_user.c file XBT * New command line option: if you pass --cfg=verbose-exit:0, SimGrid won't output the state of processes when interrupted with Ctrl-C * Add a new function xbt_dynar_to_array that transforms a dynar into a NULL-terminated array. This may solve backward compatibility issues due to the change to return type of SD_simulate. See also: http://lists.gforge.inria.fr/pipermail/simgrid-user/2010-December/002206.html * Add new macros with variable number of arguments. - in xbt/log.h: XBT_DEBUG, XBT_VERB, XBT_INFO, etc. - in xbt/asserts.h: xbt_assert - in xbt/cunit.h: xbt_test_{add,fail,assert,log} - in xbt/ex.h: THROWF and RETHROWF. Define XBT_USE_DEPRECATED if you want to use the old numbered macros like INFO1, INFO2, etc. * Change xbt_die() to accept a format string with arguments, just like printf. * New data structure: xbt_lib_t, like a dict but more general and with better memory handling. INSTR * New configuration options Options triva/categorized and triva/uncategorized can be used to generate graph configuration files for Triva visualization tool. * Configuration option tracing/platform is renamed to tracing/categorized * XBT logging makes tracing error checks easier, new root log hierarchy: instr * New TRACE_user_link_variable interface: User provides the name of the link and the tracing variable to attach to it * the declaration of tracing categories must be done after the environment creation * simpler tracing interface, just one way to declare categories TRACE_category or TRACE_category_with_color, it is up to you * links in the trace file are again identified by their names * trace contains the full platform hierarchy exactly as declared using the ASes * Options tracing/msg/[task|process]:1 groups the process by hosts for both cases, tasks and processes must have names that are unique during the simulation these options generate traces that are suited to gantt-charts, such as the space-time view of Paje * The experimental option tracing/msg/volume is deprecated its functionality may be reincorporated if needed * Buffering The tracing generates a trace file with unordered timestamped events, because of the way the core simulator (surf) works. A script available at the tools directory (fix-paje-trace.sh) can be used to put the events in order. We have changed the tracing so it can generate ordered timestamped events in the final trace, but depending on the simulator (and how much time is simulated) that can lead to a huge memory utilization. It is deactivated by default, but it can be activated using the --cfg=tracing/buffer:1 switch. Build Infrastructure * Define a SIMGRID_VERSION macro in simgrid_config.h. - We are trying hard to keep the API stable, but it may happen that some things change (we're a research project after all, not a nuclear plant operating system). If such things should happen, you could rely on that macro to adapt. - current value: 30600 for 3.06.00, aka 3.6 * Define macro MAKE_SIMGRID_VERSION(major, minor, patch) to help building a number that can be compared with SIMGRID_VERSION. * Add a build option -Denable_debug (set to ON by default): when set to OFF, assertions and verbose/debug logging events are disabled at compile time. -- Tue Jun 21 08:57:43 CEST 2011 Da SimGrid team SimGrid (3.5) stable; urgency=medium Model Checking * New feature to any SimGrid-based simulator: Model-Checking Check SIN#1 for more details. SMPI * New Model SMPI (three-interval linear regression for correction factors) See RR-7426, available at http://hal.inria.fr/inria-00527150 * Ability to use FORTRAN MPI code (through f2c, automatically privatized) * New MPI functions supported: MPI_Get_count(), MPI_Comm_split() * New: RAM folding (see RR-7426 and examples/smpi/NAS/DT-folding) * New: execution sampling (see RR-7426 and examples/smpi/NAS/EP-sampling) * See also src/smpi/README Tracing: Tracing: * Tracing system - Tracing API changes: TRACE_start and TRACE_end should not be called by user-code. They are automatically called by simulators created with SimDAG, MSG and SMPI if the toolkit is compiled with tracing_enabled=ON. Categories declaration and utilization remain the same for MSG and SimDag. - A function was added to the tracing API to declare categories with colors: - TRACE_category_with_color (char *category, char *color) where color must be in the following format "%f %f %f", red, green, blue and red, green, blue are float values in the interval [0, 1] - User can specify NULL as color parameter, or continue calling TRACE_category (cat) On that case, the tracing system will define random colors - The following command-line options are supported: --cfg=tracing/filename:msg.trace --cfg=tracing:1 (activate tracing, needed to use others) --cfg=tracing/platform:1 (categorized resource use) --cfg=tracing/uncategorized:1 (uncategorized resource use) --cfg=tracing/msg/task:1 (task creation) --cfg=tracing/msg/process:1 (process creation, migration) --cfg=tracing/msg/volume:1 (volume of MSG send/recv) --cfg=tracing/smpi:1 (SMPI interface tracing) --cfg=tracing/simdag:1 (allow SimDAG tasks receive categories) - examples of examples/msg/tracing updated * Tracing SimDag - DAXLoader and DOTLoader functions can generate tasks with categories - A new function to attribute a category to SD tasks: TRACE_sd_set_task_category (SD_task_t task, char *category) * Tracing the MPI interface implemented by SMPI - Collective operations are traced with states - Point-to-Point operations are traced with states/links - Tracing activated by a parameter "-trace filename" passed to smpirun during execution (considering that simgrid is compiled with tracing enabled) - To run the simulation with gdb, the simulator accepts --cfg=tracing/smpi:1 to trace SMPI - tesh files to check if smpi tracing is ok - See examples/smpi/NAS/DT-trace * GTNetS tracing re-worked - adaptation to the tracing system of GTNets to cope with modifications regarding the fullduplex mode - new tesh files to check if gtnets tracing is ok MSG * Asynchronous communications through the functions: MSG_task_isend/irecv and MSG_comm_test/wait/waitall * New function: MSG_load_platform_script() to make possible using a lua script instead of XML files to set up platforms * New function: MSG_set_function to associate functions to processes, used when bypassing the parser * New functions: MSG_task_set_name(), MSG_task_set_compute_duration() Platforms: Add some more examples in examples/platforms * Grid'5000: see www.grid5000.fr * *_30000_hosts.xml: various huge files [mainly scalability testing] SURF * Change the XML format. This is a very important modification. SimGrid 3.5 introduces a new hierarchical format based on the notion of Autonomous Systems. Compatibility with old format is ensured through the perl script provided in the install bin directory bin/simgrid_update_xml. It is now possible to build platforms with specific routing mechanism (Full/Dijkstra/DijkstraCache/Floyd) and to easily connect several platforms together. We will try to provide soon set of realistic platforms exploiting these properties (have a look at examples/platforms/ for the moment). * Take the opportunity of the XML format change to be a good XML citizen: rename link:ctn to link_ctn and similar changes (also dealt with by simgrid_update_xml) * Add a new routing scheme (rule-based) using regular expressions. It enables to have an extremely low memory footprint when the underlying routing is simple and can be compactly described. You need to have libpcre4-dev (perl regular expressions) installed if you want to use this routing scheme. * Revive the cluster TAG and allow to easily and efficiently (both in term of memory and speed) connect clusters together. Have a look at teshsuite/simdag/platforms/ to see how this can be done. With this tag, you can create clusters with thousands of tasks at no cost (have a look at examples/platforms/). Note: clusters are implemented as ASes, so there is no need for an enclosing AS tag if you have only one cluster in your platform. * Add new generic functions in the public interface that allows the user to call SURF 'create_resource' methods from your code (same functionality as the XML bypass mechanism but with a much lighter burden). * Add a new model (enabled through command line --cfg=network/model:SMPI) that uses a piecewise linear approximation to produce better results when exchanging small messages. * Add a new parameter to handle correctly full duplex link and account for interferences between uplink and downlink communications (activate with --cfg=fullduplex:1). SIMDAG * Rename the SD_READY (all dependencies are satisfied and task is scheduled) state in SD_RUNNABLE and define a new SD_SCHEDULABLE (all dependencies are satisfied) state. This prevents a confusion between the notion of "ready to schedule" (SD_SCHEDULABLE) used in DAG scheduling and that of "ready to be simulated" (SD_RUNNABLE) used by the simulation kernel. * Change the way a task is considered as ready. Instead of removing dependencies when a task is done, a counter is decreased. This way, it is always possible to reach ancestors thanks to the SD_taks_get_parents function (even after the end of the simulation.) * Change the return type of SD_Simulate from (SD_task_t*) into xbt_dynar_t. This function was in handling a dynar internally and converted it into a NULL terminated array for historical reasons. * New function SD_dotload(char*) to load a DAG described in dot format. This loader and the corresponding examples require the installation of the graphviz library. * Fix a bug in the management of tasks of size 0 in the surf network models. This problem was only visible with SIMDAG and you should thus disregard results produced with earlier versions if you relied on this feature (some tasks were blocked because of this). * Fix a bunch of stuff that prevented to use classical models with SIMDAG even though your applications were doing only point-to-point communications and sequential computations. Now you can really use any model you want (of course, if you create real parallel tasks, which are not implemented in most models beside ptaskL07, this will abort). * Add an example that schedules a DAX on an heterogeneous platform using a Min-Min strategy. * New function SD_workstation_get_current_task() that returns the kind of task currently running on a workstation in the sequential access mode. * Raise some warnings when unexecuted tasks remains at the end of the simulation. This is usually caused by cycles in the DAG. SIMIX * New function: SIMIX_process_set_function() called by MSG_set_function * Change the underlying waiting queue in semaphores so that a process can wait on several of them simultaneously (as in waitany). * Fix the way to handle tokens in semaphores so that all access patterns work: {acquire, acquire_timeout, waitany} / {release, release_forever}. * kill the dirty pimple SIMIX_message_sizes_output() Please use (proper) visualization instead XBT * New data container: setset (set of sets of elements) * New module: mmalloc (mapped malloc, allowing to have several independent segments of malloc) * New function: xbt_dict_cursor_set_data() * New functions: xbt_dynar_sort(), xbt_dynar_compare() * New function: xbt_dynar_is_empty() * New function: xbt_fifo_get_last_item() * Fix xbt_dynar_shrink(): use the right element size. * Fix xbt_dynar_set*(): allow index larger than current size and memset 0 uninitialized areas during expand. * Fix semaphores: previous implementation was severely broken. * Use library init/fini functions for our initialization. - you can use logs and other feature as soon as you want in your code (even before the xbt_init / MSG_init) - xbt_exit is now a no-op and produce a warning when used. GRAS: * Port GRAS to new SIMIX mechanisms. This allows gras users to benefit from the latest improvement to the simulation kernel. * Kill measurement sockets for now. If you rely on them, sorry. This release is not for you. This feature will be reintroduced in the future, but we cannot delay the release any further. * New function: gras_msgtype_get_name(). * Implement gras_agent_spawn in RL too (the prototype changed a bit) * Fix (at last) the pmm example: it should not randomly fail anymore. Build chain: bug fixes and overall polishing * Cmake is now stable enough. Hence, we killed the autotools. * Port to windows ( TM :) * Fix the 'make install' target. No need to use 'make install-simgrid' anymore * Introduce a 'make dist' target compiling a *source* archive 'make package' compiles a binary archive * Compile java files only on need * Add --cd and --setenv command line options to tesh * Out of source builds are not fully supported yet, but we are close * Enable supernovae and optimization flags by default for our users LUA Bindings * Add layer to set up environment directly from lua, without XML. * The effect of gras_stub_generator can be achieved through lua too (check examples/gras/console/ping_generator.lua) -- Wed, 01 Dec 2010 22:09:23 +0100 Da SimGrid team SimGrid (3.4.1) stable; urgency=low The "Polishing easter eggs is probably a good idea" release. This is a bug fixes release only. Java Bindings * Fix a bug preventing the tasks from begin garbage collected. MSG * Fix a bug occuring when a host involved in a communication fails. This was not detected properly by the other peer involved in the communication. Now, it's reported as a network error. SimDag * Warn the user about loop dependencies in data flow of DAX files * Obey the control-flow dependencies of DAX files Cmake * Add option "enable_smpi" allowing to not compile SMPI. Probably useful for the (Mac) users experiencing a build error here * Improve the detection of lua5.1 and ruby1.8 -- Da SimGrid team Tus, 04 May 2010 28 16:11:16 +0100 SimGrid (3.4) stable; urgency=low The "Easter in Cargese" release. Also known as (major changes): * the "se habla Java, Ruby 話せます, fala-se Lua (and deaf-friendly)" ~> bindings were greatly improved ~> new tracing infrastructure for better visualization introduced * the "Welcome to configury modernity" release. ~> we switched from autotools to cmake, and improved our cdash A more detailled list of changes follow (full detail in svn log). Java Bindings: Various Cleanups * (install java-gcj-compat-dev on debian-like to use them) * Remove put/get: no need to export deprecated interface in Java Use send/receive instead. * Cleanup the examples and add a README per directory * Remove example autoDestination (that's the only way to go now) * Remove example explicitDestination (was a plain copy of basic) * Make JniException a runtime exception, so that there is no need to declare the fact that you may encounter such a beast. I guess that nobody will ever want to survive such error. * Create specific errors for each MSG case of failure: host failure, transfer failure, timeout, task cancelled * Cleanup the exceptions that may get thrown by each function * Other internal cleanups in Java bindings. Performance still bad :/ Ruby and Lua Bindings: create them * (install ruby1.8-dev/liblua5.1-0-dev on debian-like to use them) * That's new and great, you should try them out. Same functionalities than Java bindings, only even less polished SimDag: * Kill the useless "rate" argument of SD_task_get_execution_time() Everyone used to provide -1 as a value, it was not used, and the semantic of a possible use wasn't even clear. * SD_SCHED_NO_COST: Constant to use as cost in SD_task_schedule() either as comm costs or compute costs to mean that there is no such thing for that specific task. * Add a SD_task_set_name() function * Fix SD_task_unschedule() on typed tasks * Fix SD_task_get_execution_time() to return seconds, not flop*sec * In DAX loader, accept useless 'level' attributes to since LIGO DAGs have them (seem to be to ease graphical representation). MSG: * Add an example masterslave_mailbox.c using send/receive and not the deprecated put/get interface. * Kill the MSG_paje_output() function. It's a noop since 2 years. * Kill MSG_WARNING and MSG_FATAL return codes: they were not used anywere in source. * Rename MSG_TIMEOUT_FAILURE into MSG_TIMEOUT for sake of logic (declare MSG_USE_DEPRECATED to still have the old name) * Add a MSG_task_set_data() function * About trace replay (see examples/msg/actions): - implement barrier - Allow to work with splitted trace files for each process Give the specific trace file as argument of each process, and call MSG_action_trace_run(NULL) You can still have one merged file for all processes. - Fix implementation of collective operations * Allow task_execute() on 0-sized tasks (closes #10063) SMPI: * This is the first release of SimGrid where SMPI is not considered beta anymore (even if some corners should still be improved) * Port over the new SIMIX_network submodule (internal refactoring) * Basic support to log events as with SMPE (use --cfg=SMPE:1) * Implement more missing elements of the standard: - MPI_COMM_SELF - MPI_MAXLOC MPI_MINLOC + all associated datatype MPI_DOUBLE_INT, MPI_FLOAT_INT, etc. - MPI_Address() MPI_Get_count() MPI_Type_free() MPI_Type_extent() MPI_Scan() MPI_Get_processor_name() - Added implementation of missing case for Alltoall (warning: it's *not* the bruck variant from OpenMPI; based on Alltoallv instead) - SMPI_MPI_Gather() SMPI_MPI_Gatherv() SMPI_MPI_Scatterv() SMPI_MPI_Reduce_scatter() SMPI_MPI_Allgather() SMPI_MPI_Allgatherv() * Bug fixes include: - MPI_Waitsome() was broken - Allow relative includes in smpicc - Command line cfg argument 'reference_speed' was ignored... - Some functions did not properly lead to auto-benching of user code - smpicc passes -O2 by default (just like openmpi one) SIMIX: * add SIMIX_action_suspend() and SIMIX_action_resume() functions * Bug fixes about timeouts during communications * add SIMIX_message_sizes_output() as a pimple to write to file the amount of messages per size. Use gnuplot to get histogram. Pimple because that's the only user-visible function of simix, defined directly in xbt.h (irk, sorry) * About semaphores: - Add a SIMIX_sem_get_capacity() function - Fix interactions with processe resume/suspende - release_forever() was stupidly broken - Fix SIMIX_display_process_status() for processes in a semaphore - Make SIMIX_sem_block_onto() user-visible * Refactoring context stuff: - Use pseudo-OOP for better modularity - reimplement SIMIX_process_kill() without process_schedule() so that the latter can take as invariant that it is called from maestro. - Merge context_start into context_new for sake of simplicity SURF: * Add a Vivaldi network model, coded live during SUD'10 ;) * Rename configuration variables to start a hierarchy: o cpu_model -> cpu/model o network_model -> network/model o workstation_model -> workstation/model * New configuration variables: o network/bandwidth_factor: correction to bandwith o network/latency_factor: correction to latency o netwotk/weight_S: correction to the weight of competing streams * Add a long description to the models, that users can see with such argument on the command line: --cfg=cpu/model:help * --help-models display the long description of all known models XBT: * config: add the ability to set a default value after registration Does not override any previously set value (e.g. from cmd line) * dict: allow to have integer key and data. When so, you need to use the following functions void xbt_dicti_set(xbt_dict_t dict, uintptr_t key, uintptr_t data); uintptr_t xbt_dicti_get(xbt_dict_t dict, uintptr_t key); void xbt_dicti_remove(xbt_dict_t dict, uintptr_t key); In contrary to regular dicts, the key is not malloced before copy. Mixing scalar and regular elements in the same dict is not tested (but may work). * Allow to use xbt_dynar_shrink() to expend the dynar instead Tracing for Visualization: * SimGrid is now instrumented in order to generate a trace file for visualization analysis: to use it, need to compile SimGrid with the "tracing" option enabled, and instrument the program using SimGrid with TRACE_start, TRACE_category, TRACE_msg_set_task_category and TRACE_end (among other functions). * The instrumentation only traces the platform utilization for now * Documentation to use the tracing functions and how to analyze the traces with the Triva tool is written. * More information about: SimGrid FAQ (in the section Tracing Simulations for Visualization) Build system: * We moved to cmake as default build system. Autotools support will be dropped soon. Check the FAQ for more info about how to use it. * Greatly improved our cdash/ctest interactions Check http://cdash.inria.fr/CDash/index.php?project=Simgrid * Added memory checking tests with valgrind; lot of memleak fixing. This may be the first release of simgrid with so few memory issues * Added code coverage tests. Our coverage is still improvable, but at least we see it on cdash. -- Da SimGrid team Wed, 28 Apr 2010 28 17:11:16 +0100 SimGrid (3.3.4) stable; urgency=low The "Desktop Grid needs love too" release (also called Xmas release). Models improvements: * Major speedup in the maxmin system solving by using lazy evaluation Instead of solving completely the maxmin system at each iteration, only invalidate (and recompute) the modified parts. This new feature is enabled in default models but you can try to turn it on with "--cfg:maxmin-selective-update=1" for other models. * Cas01 IMproved as default CPU model This CPU model is the same Cas01 model, but it uses the maxmin-selective-update flag and a heap structure to manage actions on SURF kernel. It reduces the complexity to find the next action to finish and, consequently, it's faster than the old Cas01. This is the new default CPU model (Cas01). * Rename the old Cas01 model to Cas01_fullupdate Keep the old cpu model Cas01 with the new name of Cas01_fullupdate. Use "--cfg=cpu_model:Cas01_fullupdate" to use the old default CPU model. * CpuTI (CPU Trace Integration) A new CPU model whose objective is simulate faster when using availability trace files. Instead of using a full featured, over engineered maxmin system for CPU modeling, this model does the pre-integration of traces files to calculate the amount of CPU power available, and so, executes faster than the old CPU models. Use "--cfg=cpu_model:CpuTI" to change to this CPU model. * Use LV08 as default network model since it gives better accuracy for small messages and shouldn't change things for big ones. Use --cfg=network_model:CM02 to get the previous behavior. ****************************************** *DO NOT MIX 3.3.4 RESULTS WITH OLDER ONES* ****************************************** * The new CPU model may changes simulations! The point is that events occurring at the exact same timestamp are not scheduled in the same order with the old and new version. This may be enough to completely change the execution of simulations in some cases. * The new network model will change simulations! This new model is more realistic than the previous one, so you should consider redoing your old experiments with this model. Sorry for the inconvenience. Build System: * Introduce the supernovae compilation mode When compiled that way, the whole SimGrid (or almost) is put in a single compilation unit and compiled in one shoot. This is to help gcc which has difficulties to inline stuff from one file into another. The speedup seem to be above 15%, althrough more tests are needed on amd64 to confirm that gain. MSG: * Port of MSG's mailbox on top of SIMIX network The put/get mechanism was greatly simplified on the way. SIMIX: * New SIMIX network module. Provides: - Mailbox: rendez-vous mechanism to find with who you want to speak - Synchronous send/recv: easier and hopefully faster since the logic is handled in the maestro process directly now - Asynchronous send/recv: you dreamt of it? It's here now Too bad that nobody cared enough to propagate the change to MSG. * Add semaphores as SIMIX synchronization mechanism. SimDag: * new function SD_daxload(char*) to load a DAX file (see http://vtcpc.isi.edu/pegasus/index.php/WorkflowGenerator) * Introduce typed tasks. Specify its kind and cost at creation. At scheduling, just give where it should be placed, and the cost for each involved resource is automatically computed. Existing constructors so far (more to come of course): - SD_task_create_comm_e2e() for end-to-end communication - SD_task_create_comp_seq() for sequential computation Use SD_task_schedulev() / SD_task_schedulel() to schedule them. * new function SD_task_dump() for debuging display * new function SD_task_dotty(task,FILE*) writing to file the info about the task in dotty format * SD_task_dependency_exists() can now cope with having one of its arguments NULL. If so, it tests whether the other argument has any dependency. * Add getters on list of preceding/following tasks: SD_task_get_parents(task) and SD_task_get_children(task) * Add getters on amount of workstations and list: SD_task_get_workstation_count(t) and SD_task_get_workstation_list(t) * Add getter on task kind: SD_task_get_kind(task) * Update the start_time and finish_time of tasks on completion/failure * Bugfix: Remove task from state swags when destroyed GRAS: * New function: void gras_cpu_burn(double flops) -- a simple CPU burner XBT: * New function: xbt_dynar_dopar(dynar,fun) to map a function over the dynar with one separate thread per value of the dynar. * Change the prototype of xbt_thread_create(), sorry. Added a boolean parameter indicating whether we want to join this thread (used in SG only for now) * Implement xbt_thread_join and xbt_thread_yield in SG also. Bug fixes: * GTNetS wrappers should now be usable again (and betterly tested too) * Fix a major regression from 3.2 where the timeout provided to MSG_task_put_with_timeout() was used as absolute time before which the comm should be done. * Start to fix the tag. - Internal links should be good now (beside of the loopback, which use the private link instead) - paths to the external world is still rather broken - the tag is just broken. Actually that's brain-dead. We need sth like to make it less stupid ** Check your platform with teshsuite/simdag/platforms/flatifier ** * Fix a source-level compatibility glitch from 3.2: after defining MSG_USE_DEPRECATED, you can use the old name MSG_task_put_with_time_out() for MSG_task_put_with_timeout() * Allow to compile from the SVN with automake 1.11 * Fix some problems when using the "start_time" tag in deployment XMLs. * Fix #8569: XBT/synchro.h has redundant declarations * Fix #8563: MSG return values and exceptions Introduce a MSG_TIMEOUT_FAILURE return code and use it consistently. * Integrate patch #8636: Obey DESTDIR when installing documentation. Thanks to Robson Peixoto. * Fix a vicious bug in dictionaries inducing that some elements were not freed on xbt_dict_free() Portability report of this version: * Main portability targets: - linux (ubuntu (804/810/910) /debian (4/5/testing) /fedora (core11)) on (amd64/i386/ia64) - mac leopard on i386 Known problems: http://cdash.inria.fr/CDash/index.php?project=Simgrid but nothing critical. * Other platforms: windows, AIX and others were not tested for this release Timing report of this version: * Lazy evaluation brings arbitrary speedup (ie, speedup depending on scenario parameters). From 8h to a few seconds in desktop grid settings. * Supernovae brings about 25% speedup on i386. -- Da SimGrid team Thu, 24 Dec 2009 19:07:39 +0100 SimGrid (3.3.3) stable; urgency=low The "Need for Speed" release. The timings done to validate the 3.3.2 were faulty. Instead of being 5% faster, it was 15% slower (compared to 3.3.1). The problem was a conversion from a manually handled vector to xbt_dynar_t on the critical path. xbt_dynar_foreach calls functions, inducing stack management crap. We inlined these functions and xbt_dynar_foreach is now breath taking. We also inlined xbt_swag_belong on the way. Here are some approximate speedup measurements (on master/slaves simulations lasting between 10s and 20s each): 3.3.1 -> 3.3.2: about same performance 3.3.2 -> 3.3.3: 40% speedup 3.3.1 -> 3.3.3: 40% speedup 3.3.1 with inline patch -> 3.3.3: 30% speedup Our reading is that the refactoring which occurred in 3.3.2 made us suffer much more from the xbt_dynar_foreach low performance, but once we solved this, this refactoring proved to be very performance effective. From the 40% speedup, somehow, 10% are due to the inlining and 30% to the refactoring. That's a pitty that gcc cannot inline functions placed in other files alone. We have to choose between: - break the encapsulation (by putting private data structures and accessors in headers files to help gcc) - live with low performance - switch to a decent compiler such as icc (not quite possible). -- Da SimGrid team Thu, 20 Aug 2009 21:21:33 +0200 SimGrid (3.3.2) stable; urgency=low The "Simplicity does not preceed complexity, but follows it" release. The main contributors of this release were (lexical order): Silas De Munck, Stéphane Genaud, Martin Quinson, Cristian Rosa. SURF: * Extract the routing logic into its own object. (was dupplicated in network.c and workstation_LV07.c; Allows to implement other ways of storing that info) => kill now useless network_card concept - Use dynar to represent routes (instead of void** + int*) - kill link_set (use surf_network_model->resource_set instead) - Add a command-line option to choose the routing schema to use - Add three new models: * Floyd (shortest path computed at initialization) * Dijikstra (shortest path recomputed all the time) * Cached Dijikstra (shortest path computed on need) All these models where contributed by Silas De Munck, and are described in his ICCS09 paper. * Simplify model declaration (less redirections, less function to write when defining a model) - Factorize stuff between models: - model_init/exit - Set of resources: surf_model_resource_set(model) surf_model_resource_by_name(model, name) - Unify the types of models in s_surf_model_t (using an union) - Embeed fields of common_public directly into s_surf_model_t - Rename model methods: action_free ~> action_unref action_change_state ~> action_state_set action_get_state ~> action_state_get - Change model methods into functions : (model)->common_public->action_use ~> surf_action_ref * Implement a generic resource; use it as ancestor to specific ones (allows to kill duplicated code in models) Drawback: timer command don't need no name nor properties; workstation_CLM03 don't need no properties (but I guess we can live with those few bytes wasted) * Improve the action object model - implement a constructor avoiding dupplicated code about field initialization in generic_action part. * Kill the SDP model: it has an external dependency, is deprecated in flavor of modern lmm models, and didn't compile since a while SIMIX: * Relocation of the context module from XBT to SIMIX. (the context were decoupled from the simix processes, duplicating a lot of code) => a lot of code was factorized - less overhead is introduced during scheduling - simpler API for the context factory - the logic for process creation,destruction and manipulation was simplified * Simplification of the s_smx_process_t data structure. => accesing the simix level data associated to a process is faster now, and the code is a lot more readable. SMPI: * Implement some more MPI primitives: MPI_Bcast, MPI_Waitany, MPI_Waitall, MPI_Reduce, MPI_Allreduce, MPI_Scatter, MPI_Sendrecv, MPI_Alltoall -implementation: Bcast: flat or 2-ary tree (default), Barrier: 4-ary tree, Reduce: flat tree Allreduce: Reduce then Bcast Alltoall: "basic_linear" if data per proc < 3Kb, "otherwise pairwise". Not yet implemented: "Bruck" for data per proc < 200b and comm size > 12 Alltoallv: flat tree, like ompi Scatter: flat tree * Add support for optimized collectives (Bcast is now binomial by default) * Port smpirun and smpicc to OS X SimDag: * Kill SD_link_get_properties: hard to maintain and makes very little sense Shout out if you used it. GRAS: * Display the list of still queued messages in SG mode when existing the process. XBT: * Add xbt_set_get_by_name_or_null() [Silas De Munck] * Add xbt_graph_node_get_outedges() [Silas De Munck] * Add xbt_str_from_file(FILE*) * Add xbt_dict_get_key achieving a linear reverse search * Remove the context module Portability report of this version: * Main portability targets: - Linux(debian)/x86/context - Linux(debian)/x86/pthreads - Linux(debian)/amd64/context - Linux(debian)/amd64/pthreads On these, we still have the eratic breakages of gras/pmm and amok/saturate_sg reported in previous version. We still think that the tests are the cause of the fault, not the tested code. - Mac OSX Leopard/x86/context Still false negative in tesh autotesting. Smpi still fails, but this time because readlink does not accept -f Everything seems to work properly beside of that. * Exotic platforms: - AIX version 5.3 (only tested contexts this time) Smpi still fails there because mktemp is not installed. Everything seems to work properly beside of that. - OpenSolaris 11 I managed to compile it for the first time, but several breakages. Won't delay the release for this exotic platform. * Windows: it's still lagging behind. If you want to help, please stand up. Timing report of this version: This version seem to be more than 5% faster than 3.3.1 (on linux 64bits with contextes). The gain is less than expected, we are investigating this for next release. -- Da SimGrid team Wed, 19 Aug 2009 17:07:12 +0200 SimGrid (3.3.1) stable; urgency=low OVERALL CHANGES: * Implement a --cfg-help to show existing configuration variables * Build chain do not require doxygen in maintainer mode GRAS: * fix a bug on struct sizeof computation, which prevented the exchange of arrays of structs in some conditions - added a regression test about this in datadesc_usage * Allow the exchange of 0-long dynamic vectors. - for that, use -1 as indicator of dynamic size instead of 0 - This implied to change any size from unsigned long to long, reducing a bit communication abilities, but I guess that with 64bits being quite common, this is more than enough. - This also induce a protocol change, thus bumping network protocol version from 0 to 1 (if we have external users, we have to get clean on that point too ;) - added two regression tests about this in datadesc_usage * Be more verbose when propagating local exceptions This helps debugging. * Display the status of simulated processes when receiving SIGINT in simulation mode MSG: * Allow to control the simulation from a trace file. New functions MSG_action_register() and MSG_action_trace_run() The first one allows to associate a function execution to each kind of action while the second one parses a trace file and triggers the corresponding actions within the system. For now, only a toy example is provided in examples/msg/actions * Add an exemple of process migration in examples/msg/migration * Fix a bug in task exchange which broke MSG_task_get_sender() Add a teshsuite regression test for that. [Bug: if MSG_task_get_sender() is called after sender exit, bad things happen] * Fix a bug which prevented suspend/resume to work properly * Display the status of simulated processes when receiving SIGINT This fixes a regression of v3.3. due to the introduction of SIMIX * Bug fixing in failure management: - trace could not start by a failure at time 0 - failure during communications were not working SIMIX: * Add SIMIX_process_set_name() to change the name of the current process in the log messages. * Store smx_hosts in a dict since we only retrieve them by name * Move the configuration infrastructure to surf SIMDAG: * Move the configuration infrastructure to surf SMPI: * Massive internal cleanups: - Store internal structures on processes instead of hosts (allows to have more than one process per host, in addition of being more logical) - Cleanup the initialization/finalization process - Kill a whole bunch of unneeded synchronization: processes run in exclusive manner within the simulator - Move queues from global tables to process data fields * Improve smpirun: - now accept -platform and -hostfile arguments - Pass the right rank value to processes according to the hostfile * Compile the examples by default, and use them as regression tests * Implement MPI_Wtime() * Change the reference speed to a command line option SURF: * TCP_gamma can now be specified as command line option using --cfg=TCP_gamma:10000000.0 * Change the --surf-path cmd line option into --cfg=path: XBT: * Also include strbuff from xbt.h public header * xbt_ex_display(): do not free the exception after displaying This allows to do more with the given exception afterward. Users should call xbt_ex_free() themselves. Portability report of this version: * Main portability targets: - Linux(debian)/x86/context - Linux(debian)/x86/pthreads - Linux(debian)/amd64/context - Linux(debian)/amd64/pthreads These targets fail about 1/10 of times on gras/pmm, but we believe that this is because of the test, not because of simgrid. amok/saturate_sg fails even more rarely, and the test may not be the problem. - Mac OSX Leopard/x86/context The test suite still spits tons of errors because some obscure force prevents us from removing the temporary directories arguing that they still contain some metadata I've never heard of. Smpi fails because seq is not installed. Everything seems to work properly beside of that. * Exotic platforms: - AIX version 5.3 (both contexts and pthread) Smpi still fails there because mktemp is not installed. XML inclusions seems rosty on AIX. * Windows: it's still lagging behind. If you want to help, please stand up. -- Da SimGrid team Sat, 27 Jun 2009 00:14:30 +0200 SimGrid (3.3) stable; urgency=high OVERALL CHANGES: * JAVA BINDINGS for MSG (you dreamt of them? We made them) [Malek Cherier & Mt] * Introduce the SIMIX module: factorize code between MSG and GRAS. [Bruno Donassolo] Until now, GRAS were using MSG as an interface to SURF. It was quite difficult because both interface have several differences (MSG channels vs GRAS sockets were the most notable point). This also opens the gate to SMPI (which should occur soon) and speed up simulations by to 40% (even if it were not the main goal). ************************************** *DO NOT MIX 3.2 RESULTS WITH 3.3 ONES* Simix may changes simulations! ************************************** The point is that events occuring at the exact same timestamp are not scheduled in the same order with the old and new version. This may be enough to completely change the execution of simulations in some cases. Sorry for the inconvenience. * Cleanup and upgrade the XML format to push further scalability issues (check http://hal.inria.fr/inria-00256883/ for more info) * Improve the testing infrastructure with tesh. Now a very large part of the code is tested not only by being run but also by checking that the output match an expected output [Mt]. * Move on to FleXML v1.7 for the embeeded XML parsers. This version is really less memory-demanding, which should allow you to use larger files in SimGrid [AL]. * Inform valgrind about our contextes, so that it becomes usable with the default (and more effecient) version of SimGrid [contributed by Sékou Diakite, many thanks] GRAS: * Introduce a listener thread in charge of receiving incoming messages from the network. It allows to overlap communication and computation but most notably, it removes some stupid deadlocks due to the fact that so far, a process could not send and receive at the same time. This made most non trivial communication schema impossible. * Convert the PIDs from long int to int to match the MSG ones (and linux ones too) [Mt] * New function: gras_agent_spawn() to launch a new process on current host. Only working in simulation for now. [Mt] * New function: gras_os_hostport() returning a constant form (ie, not needing to be freed) of "gras_os_hostname():gras_os_myport()" XBT: * Make the backtrace of exceptions more human readable [Mt] * New module: xbt/str [Mt] a ton of string utility functions (split, join, printf to a newly allocated buffer, trim, etc) * New module: xbt/hash [Mt] SHA1 hashing algorithm (more to come if needed) * New module: xbt/synchro [Mt] synchronization tools (mutex and conditions) working the same way in simulation and in real life (mainly useful for GRAS, but not only). * New module: xbt/queue [Mt] classical producer/consumer synchronization scheme * xbt_dynar_new_sync() creates a synchronized dynar. All access (using the classical functions will get serialized) [Mt] * Make dictionary internal table dynamic. No need to specify its size anymore; functions xbt_dict_new_ext() and xbt_dict_hashsize_set() thus dropped. [Mt]. * Make sure the log channels are organized as a tree under windows (because of ANSI C compatibility issue, any channel were child of root directly) [Mt]. SURF: * Cleaned many thing in surf and fixed a few bugs [AL]. * Add a nice command line configuration mechanism to compose models [AL]. * Add a new model for parallel tasks (ptask_L07) that is less buggy than the previous one (KCCFLN05). It relies on something that looks like a max-min sharing mechanism but cannot be written as such. A new solver was thus designed [AL]. * Add a new solver to lmm. Based on Lagrange optimization and gradient-based descent, it enables to efficiently maximise systems s.a sum f_i(x_i) s.t Ax<= b with A_{i,j}>=0 and f_i a concave function. This solver enables to propose two new network models for TCP Reno and TCP Vegas based on Low's work. These models still need to be fully tested though [Pedro Velho]. SIMDAG [AL]: * Bug fix in SD_simulate. Now the time bound given as argument is used. * Use the new parallel task model (ptask_L07) as default. * Use the SURF command line configuration mechanism. * 0-size tasks (for synchronization) should now work. -- Da SimGrid team Sun Apr 12 05:20:36 CEST 2009 SimGrid (3.2) stable; urgency=high OVERALL CHANGES: * Port to windows. We still experience issues on this platform, but we believe that at least MSG is usable. GRAS API BREAKAGE (for simplification purpose, sorry): * the gras_msgtype_by_name is not used anymore. Instead of gras_msg_send(toserver, gras_msgtype_by_name("request"), &request); you can write (and must) gras_msg_send(toserver, "request", &request); - If you still want to pass a gras_msgtype_t to the function (to cache the type and avoid the lookup time), use the gras_msg_send_() variant. - Impacted functions: gras_cb_register, gras_cb_unregister, gras_msg_send, gras_msg_wait, gras_msg_rpccall, gras_msg_rpc_async_call, gras_msg_wait_ext * The callbacks are now expected to return 0 when everything went well (just like the main() function) GRAS new features and improvements: * New module mechanism where user code can use per process globals [Mt] This is similar to gras_userdata_*() functions, but for libraries. It factorize some code developped over and over in the examples and AMOK. It has still to be documented and used (only amok/peermanagement is converted for now). * Fix a vicious bug in the TCP buffering mechanism which leaded to message loss when they were small enough to fit into the buffer and sent quickly enough so that they can all get received in one shoot. * gras_datadesc_by_name and gras_msgtype_by_name: now raise an exception if not found. Use the *_or_null() variant for the old semantic. * In gras_msg_handle, do not discard messages without callback. They are probably messages to be explicitly awaited later (ie, proofs of mis-synchronization in userland since they are sent before being awaited) No big deal usually. * gras_socket_meas_send/recv: semantic changed! The numerical arguments used to be (1) the total amount of data to send and (2) msg_size. This was changed to (1) msg_size and (2) amount of messages. This was need for the fool willing to send more than MAXINT bytes on quite fat pipes. AMOK: * Do really rename the hostmanagement module to peermanagement. [Mt] Ie, rename functions from amok_hm_* to amok_pm_*. This breaks the API, but this is rather new and this was documented in the module documentation (poor excuses, I admit) * Bandwidth measurement semantic changed! This follows the changes to gras_socket_meas_send/recv explained above. SIMDAG: * A sequential mode has been added to the workstations. When a workstation is in sequential mode, it can execute only one task, and the other tasks are waiting in a FIFO. [Christophe Thiery] SURF: * The KCCFLN05 workstation model now handles parallel tasks. It is the model for SIMDAG. [Christophe Thiery] * Bug fix in the maxmin solver: Some values were close to 0 instead of equal to 0, which caused some bad behaviors in saturated_constraint_set_update. I now use a threshold mechanism like in surf. [AL] XBT: * When running manually src/testall, you select specific units [Mt] testall is the result of our cunit mechanism, and should replace all the scripty thingy around since bash don't run easily on billware. * A mallocator system has been added. [Christophe Thiery] Mallocators allow you to recycle your unused objects instead of freeing them and allocating new ones. Documentation update: * FAQ reworking + New FAQs: - "Valgrind spits tons of errors!" [Mt] - "How to repport bugs" [Mt] - "Cross-compiling a Windows DLL of SimGrid from Linux" [Mt] - "What is the difference between MSG, SimDag, and GRAS?" [Mt] - Communication time measurement within MSG [AL] - I experience weird communication times when I change the latency [AL] * GRAS tutorial [Mt] It contains: - an introduction to the framework and to the used communication model - an initiatic tour introducing the most proheminent features: o Part 1: Bases . Lesson 0: Installing GRAS . Lesson 1: Setting up your own project o Part 2: Message passing . Lesson 2: Exchanging simple messages . Lesson 3: Passing arguments to the processes (in SG) . Lesson 4: Attaching callbacks to messages . Lesson 5: Using globals in processes . Lesson 6: Logging information properly . Lesson 7: Using internal timers . Lesson 8: Handling errors through exceptions . Lesson 9: Exchanging simple data . Lesson 10: Remote Procedure Calling (RPC) . Lesson 11: Explicitely waiting for messages . Recapping of message passing features in GRAS - A HOWTO section containing: o HOWTO design a GRAS application More are due, of course. They will come latter. In the meanwhile, you can check the examples which are still here. -- Da SimGrid team Fri Mar 16 21:11:46 CET 2007 SimGrid (3.1) stable; urgency=high General: * Port to gcc 4.x There was a stack corruption somewhere, visible only when optimizing with these versions. [Vince] SIMDAG: * This is a NEW module! SimDAG (SD for short) is a revival of the old SG module that enabled to play with Directed Acyclic Graphs. It is built directly on top of SURF and provides an API rather close to the old SG. Some old codes using SG are currently under rewrite to check that all needful functions are provided. [Christophe Thiery] SURF: * Complete rewrite of the KCCFLN05 workstation model. It is now an extension of the classical CLM03 model that gracefully handles failures. This is now the default model for MSG and GRAS. It doesn't handle parallel tasks yet though. [AL] * Bug fix: Weights were not correctly set in the network part. WARNING: This may have resulted in incorrect results with simulations where there are more than one flow on a given link. [AL] SURF, MSG, GRAS: * After a (long ?) discussion on simgrid-devel, we have decided that the convention we had on units was stupid. That is why it has been decided to move from (MBits, MFlops, seconds) to (Bits, Flops, seconds). WARNING : This means that all previous platform files will not work as such with this version! A warning is issued to ask users to update their files. [AL] A conversion script can be found in the contrib module of the CVS, under the name contrib/platform_generation/surfxml_update.pl [MQ] MSG,GRAS: * Bug fix: Processes were started in reverse order, wrt deployment file. WARNING: if your code relies on this bug, please fix it. [AL] * Bug fix: Add a test in MSG_task_execute to stop whenever a task is being executed on two different locations. [AL] * Bug fix: Failures are now better supported thanks to Derrick's tests (there was many failure situations I hadn't thought of and that weren't correctly handled). [AL] * New function: MSG_host_is_avail indicates you whether a given m_host_t is up or down. [AL] GRAS: * New! a real RPC mechanism, as it ought to be since too long. [MQ] Exception occurring on server-side are propagated back to client (!). API CHANGE: the callback changed their prototype. Change: int my_handler(gras_socket_t expeditor, void *payload_data) { to: int my_handler(gras_msg_cb_ctx_t ctx , void *payload_data) { gras_socket_t expeditor=gras_msg_cb_ctx_from(ctx); and you're set. * New! function: gras_msg_handleall to deal with all messages arriving within a given period. * New! function: gras_socket_server_range to get a server socket in a range of port numbers (ease to avoid port number conflicts) [MQ] * New! gras processes display their backtrace when they get a SIGUSR1 or when Ctrl-C is pressed. Use Ctrl-C Ctrl-C to exit. Sweet to debug RL processes [MQ] AMOK: * Bandwidth module: - Do not force experiment sizes to be expressed in kb, or it becomes impossible to measure the latency this way (needs one byte-long tests) WARNING: this changes the amok_bw_* function semantic. [MQ] - Implements the link saturation stuff. [MQ] * Peer management module: New! module factorizing code that we wrote over and over [MQ]. XBT: * New module: cunit (my jUnit implementation in ansi C) [MQ] - Test units are placed directly into the library code, they get extracted automatically and placed into the src/testall binary. - Convert most of the XBT tests to this system. * New functions: xbt_dynar_getfirst_as() and xbt_dynar_getlast_as() [MQ] * XML parsing: rewrote parts of flexml to enable multiple xml parsers to live in the same C code. This required to change a little bit the API of surfxml parsing but shouldn't be an issue for end-users. [AL] * New module: sparse graph structure with basic algorithms (this is work in progress and the API is not considered to be frozen yet). [AL] * Display more information on backtraces: source line & function names are now displayed just like valgrind does (rely on addr2line tool) [MQ] * New function: xbt_backtrace_display(). Sweet while debuging [MQ] * Reworked a little bit some #include statements to load only required headers. Some user code that relied on SimGrid to include stdlib or stdio may need to include it by themselves. [AL] * Fixed xbt/log.h. A missing SG_BEGIN_DECL prevented compilation with g++. [AL] * Renamed xbt_host_t into xbt_peer_t since it betterly describes what I meant. This breaks the API of AMOK and of xbt/config. Sorry about this, but I guess that almost nobody used those parts. [MQ] -- Da SimGrid team Fri, 14 Jul 2006 01:32:27 +0200 SimGrid (3.0.1) stable; urgency=low XBT: * Unfortunately, I had missed 5 misnamed functions: xbt_fifo_item_t xbt_fifo_newitem(void); void xbt_fifo_freeitem(xbt_fifo_item_t); xbt_fifo_item_t xbt_fifo_getFirstItem(xbt_fifo_t l); xbt_fifo_item_t xbt_fifo_getNextItem(xbt_fifo_item_t i); xbt_fifo_item_t xbt_fifo_getPrevItem(xbt_fifo_item_t i); They're now deprecated. Please use their new versions: xbt_fifo_item_t xbt_fifo_new_item(void); void xbt_fifo_free_item(xbt_fifo_item_t); xbt_fifo_item_t xbt_fifo_get_first_item(xbt_fifo_t l); xbt_fifo_item_t xbt_fifo_get_next_item(xbt_fifo_item_t i); xbt_fifo_item_t xbt_fifo_get_prev_item(xbt_fifo_item_t i); [AL] * Bugfix: really disconnect fifo items which are remove_item()ed [AL] * Documentation: xbt_log module unmercifully reworked [MQ] * Bugfix: there was a problem with the ending of contexts with the pthread backend. It caused some weird deadlock or behavior depending on the pthread implementation. [AL] * Bugfix: get the exceptions raised in the simulator repport where and why they come from when they are not catched in time [AL, MQ] SURF: * Bugfix: Do repport the error when two non-connected hosts try to exchange data (Thanks to Flavien for stumbling into this one) [AL] SURF: * Add additionnal checkings on communications. Assert that two communicating hosts are connected by a set of links... [AL] MSG: * Add additionnal checkings on channel values in communication [AL] * New: MSG_task_get_source to see on which host a task was generated [HC] * New: int MSG_task_probe_from_host(int channel, m_host_t host): returns the number of tasks waiting to be received on channel and sent by host. [AL] * New: MSG_error_t MSG_task_get_from_host(m_task_t * task, int channel, m_host_t host); waits for the first task coming from a given host.. [AL] GRAS new functionnalities: [MQ] * Enhance the parsing macro to allow the size of multidimentional objects to be given thru annotations. * New example (and documentation): Matrix Multiplication a la RPC (as when I was young!) and fix a bunch of bugs found on the way. GRAS performance improvements: [MQ] [DataDesc] * Reduce the amount of cbps creation/destruction by making it static to datadesc_send/recv() and using a (newly created) cbps_reset (based on dynar_reset ()) [Virtu] * Change libdata to a set so that we can search for stuff by ID (and thus reduce the insane amount of dict lookups) [Transport] * Actually implement gras_datadesc_copy() so that we don't have to mimick RL communication on top of SG since it's so uneffective. It may also be used for inter-thread communication in RL, one day. * Use gras_datadesc_copy() to exchange messages on top of SG Allows to: - improve message exchange performance on top of SG - deprecate transport_plugin_sg.c:gras_trp_sg_chunk_send() & recv() * Don't exchange on the network the size of the used part of buffer, instead, specify the possible buffer size to read(). Advantages: - reduces the amount of read/write calls (one pair per exchange) - reduces the amount of exchanged data (the size) - allows to retrieve all arrived data on receiver side, if we don't need it right now (subsequent read will peek the buffer) - allows the receiver to proceed with the begining of the stream before everything is arrived - make it possible to build an iov transport (using readv/writev) Extra difficulty: - take care of the data with non-stable storage (like stacked data), and bufferize them. * If possible, TCP send uses vector I/O (when writev() is here) - Don't use it for receive since we send data sizes and data on the same stream, so we wouldn't be able to chain large amount of chunks before having to flush the stuff to read the size. * Rework the transport plugin mechanism to simplify it and reduce the amount of pointer dereferencement when searching for the right function to use. * I guess that now, we do almost as few system calls as possible while doing as few data copy as possible. To improve it further, we could try to send all the sizes first and then all the data (to use iov on receiving size), but it's only a partial solution: when you have 2 dimensional data, the sizes of the second dimension is data of the first dimension, so you need 3 streams. I'm not sure the potential performance gains justify the coding burden. -- Da SimGrid team Fri, 21 Oct 2005 14:42:20 +0200 SimGrid (3.00) stable; urgency=high SURF: * New! Give the possibility to hijack the surf parser and thus bypass MSG_create_environment and MSG_launch_application. Have a look at examples/msg/msg_test_surfxml_bypassed.c to see how it can be done. -- Arnaud Legrand Sat, 20 Aug 2005 23:25:25 -0700 SimGrid (2.96) unstable; urgency=low AKA SimGrid 3 rc 2. XBT: * New! Exception handling with setjmp or such (code from OSSP ex) [MQ] This deprecates the xbt_error_t mechanisms. It modifies (simplifies) all XBT and GRAS API. MSG API keeps unchanged (exceptions raised by XBT are catched from within MSG and masked with existing error handling facilities) SURF: * New! Add a FATPIPE model. [AL] * New! Add a parallel task model. [AL] * New! Add automatically a loopback interface (in the default network model) if none was precised. MSG * Bugfix: MSG_process_resume now works with the current running process. [AL] * New! Add MSG_parallel_task_create and MSG_parallel_task_execute. [AL] * Modification of MSG_task_get_compute_duration. Once a task has been processed, the value returned by this function is now equal to 0. [AL] * New! Add double MSG_task_get_remaining_computation(m_task_t task) and MSG_error_t MSG_task_cancel(m_task_t task). Add a state (MSG_TASK_CANCELLED) to MSG_error_t corresponding to the cancelation of a m_task. For now, MSG_task_cancel only works with computation tasks. [AL] * New! Add double MSG_get_host_speed(m_host_t h) that returns the speed of the processor (in Mflop/s) regardless of the current load on the machine. [AL] * API Change: use proper naming convention for MSG_getClock and MSG_process_isSuspended: MSG_get_clock and MSG_process_is_suspended. [AL] * New! Add void MSG_task_set_priority(m_task_t task, double priority). This function changes the priority of a computation task. This priority doesn't affect the transfer rate. A priority of 2 will make a task receive two times more cpu power than the other ones. This function has been added to suit the needs of Nguyen The Loc and hasn't been that much tested yet. So if it fails, please report it and send me your code. [AL] * API Change: removed all functions and types that were marked "deprecated" since many months. Renamed MSG_global_init_args to MSG_global_init. -- Da SimGrid team Mon, 8 Aug 2005 17:58:47 -0700 SimGrid (2.95) unstable; urgency=low XBT * Steal some nice code to GNU pth to fix context detection and usage [AL] * Cleanup in the xbt_config API; add configuration callbacks. [MQ] * Cleanup in the initialization API: the unused "defaultlog" is dead. [MQ] SURF * Bugfix: Allow absolute paths for platform description files [MQ] * Bugfix: do free the variables after use. Leads to drastic performance improvement [AL] * Implement max_duration (ie, timeouts) on resources [AL] MSG * Implement MSG_config to configure MSG at runtime. xbt_cfg test on a real case ;) [MQ] * Implement MSG_channel_select_from() to help GRAS now that SURF provide the needed support (timeouts) [AL] GRAS (new features) * Implement measurement sockets. You can now get the bandwidth between two hosts thanks to AMOK (see below). [MQ] * gras_datadesc_dynar() builds a dynar type descriptor, allowing to send dynar over the network (yeah) [MQ] * Real (even if simplistic) implementation of gras_os_myname() on RL [MQ] * simple/static token-ring example. [Alexandre Colucci and MQ] * Use MSG_channel_select_from() and remove the *slow* hack we had to put in place before [MQ] GRAS (bug fixes) * Differentiate the types "char[22]" and "unsigned char[22]" in automatic type parsing. "short" and "long" modifiers were also ignored; other modifier (such as reference level) are still ignored. [MQ] * Embeed the buffer size within the buffer itself on SG. [MQ] That way, send() are atomic and cannot get intermixed anymore (at least the ones which are less than 100k; bigger messages still have the issue) * Array size pushed by the field, not by the field type (or each and every long int will push stuff to the cbps) [MQ] * use select() to sleep since it allows to portably sleep less than one second. [MQ] GRAS (minor cleanups) * .Makefile.local (generated from gras_stub_generator) |MQ]: - Do clean .o files - Compile with -g * Type Callbacks now receive the gras_datadesc_type_t they work on as argument. * type category 'ignored' killed as it was never used and were difficult to transmit. * whether a type can cycle or not is now a flag, leaving room for more flags (as "ignored", if we feel the need one day ;) * Rename raw sockets to measurement sockets since "raw" has another meaning in networking community. AMOK * Advanced Metacomputing Overlay Kit introduction. It is based over GRAS and offers features not belonging to GRAS but that most applications need. One day, it may be a set of plugins loadable at runtime. * New module: bandwidth bandwidth measurement between arbitrary nodes running this module. [MQ] -- Da SimGrid team Thu, 30 Jun 2005 16:29:20 -0700 SimGrid (2.94) unstable; urgency=low The first beta release of SimGrid 3 ! >>>Arnaud<<< (documentation) * Update the main page and the FAQ. Adding references to gforge. (gras) * Add a gras_os_getpid function. (msg) * Add MSG_task_get_compute_duration() and MSG_task_get_data_size() * Extend the logs so that they also print PID, hostname, date, ... if available. * Convert the MSG example to the use of xbt_logs instead of PRINT_MESSAGE, and kill the old version which were in testsuite/ * Rewrite tools/MSG_visualization/colorize.pl for using with logs instead of PRINT_MESSAGE (xbt) * Add xbt_os_time(). As the rest of xbt/portability, this is not public for users. Instead, each programming environment (GRAS, MSG,...) use it when needed to provide such a feature to users. Don't shortcut the mechanism or you will also shortcut the virtualization you need on the simulator. >>>Martin<<< (infrastructure) * Cleanups in configury with regard to compile optimization/warning flags. Also add -fno-loop-optimize to any powerpc since it's the optimization killing gcc (< 3.4.0). * Doxygen cleanups: move MSG examples, kill the second Doxygen phase needed by MSG examples complications * Borrow configury beautifications from PHP (xbt) * Bugfix: XBT_LOG_NEW_DEFAULT_CATEGORY now compiles without compiler warning (thanks loris for stumbling into this one). * Bugfix: stop loading private headers (gras_config.h) from the public ones (xbt/swag.h). (gras) * Change SIMGRID_INSTALL_PATH to GRAS_ROOT in Makefiles generated for user. * Rename gras_get_my_fqdn to gras_os_myname and implement it in the simulator RL would imply a DNS resolver, which is *hard* to do in a portable way (and therefore delayed). * Implement a real timer mechanism and use it in timing macros. This allows to avoid rounding errors and get a 0.000005 sec precision in timing macros. While I was at it, various cleanups: - allow to declare more than one timed section per file (fix a stupid bug) - move some private declaration to the right place - merge conditional execution and timing macros into emulation module - document the module - make sure the module cleanups its mess on gras_exit * Documentation improvements: - (new) how to compile applications using GRAS - (new) emulation support (timing macros) -- Da SimGrid team Fri, 13 May 2005 10:49:31 +0200 SimGrid (2.93) unstable; urgency=low Alpha 4 on the path to SimGrid 3 (aka the "neuf-trois" version) [Arnaud] - Use Paje properly where used. Still to be sanitized properly. - Portability fix: Add an implementation of the contexts using pthread [Martin] (misc) - Add xbt_procname(): returns the name of the current process. Use it to show the current process's name in all logging. (infrastructure) - fix detection of older flex version and the reaction, since we do depend on modern ones (we use lex_destroy) - Better separation of SG and RL in the libs: remove all simulation code from libgras. As a result, this lib is now only 200k when stripped. Some of the xbt modules may also be duplicated (two sets and such) and should be cleaned/killed before SG3. - Insist on using xlC on AIX because of weird problems involving gcc there. - Cleanup the make remote stuff. This is now done by scripts tools/graspe-{master,slave} (GRAS Platform Expender). This is still mainly for our private use, but we're working on changing them to user tools, too. (gras) - Bugfix: flush the socket on close only if there is some *output*. - Bugfix: flush idempotent when there's nothing to send (don't send size=0) (msg) - Add MSG_task_get_name. The task names are mainly for debugging purpose, but anyway. -- SimGrid team Fri, 4 Mar 2005 14:32:37 -0800 SimGrid (2.92) unstable; urgency=low Alpha 3 on the path to SimGrid 3 [Arnaud] (gras) - New! First try of benchmarking macros. - New! First try so that gras_stub_generator generate deployment and remote compilation helpers. (msg) - Bugfix: Initialization fix in msg_test. [Martin] (surf) - Bugfix: applied patch to lexer so that it doesn't need a huge heap. (xbt) - Bugfix: let dicts work with NULL content (_foreach didn't) and cleanups (gras) - API Change: gras_os_sleep to take the amount of seconds as a double. Accepting an int was error prone since it was the only location where seconds were coded as such. It leaded to damn rounding errors. - Bugfix: Hard to belive that timers ever worked before this. -- SimGrid team Wed, 23 Feb 2005 22:09:21 +0100 SimGrid (2.91) unstable; urgency=low Alpha 2 on the path to SimGrid 3 [Arnaud] (surf) - Bug fix in the lmm_solver. (msg) - New! Interface to Paje (see http://www-id.imag.fr/Logiciels/paje/) through the function MSG_paje_output. - New! Introducing two new functions MSG_process_kill() and MSG_process_killall(). - It is possible to bound the rate of a communication in MSG with MSG_task_put_bounded() (was already in the previous version but I had forgotten to write it in the changelog). - Bug fix to let GRAS run on top of MSG until we move it directly on top of the SURF. [Martin] (infrastructure) - Various cleanups to the autotools stuff - Begin to move Gras examples to examples/gras/ - Let make distcheck work again (yeah!) (documentation) - documentation overhauled using doxygen. gtk-doc-tools is dead in SimGrid now. - Automatically extract all existing logging categories, and add the list to the documentation (long standing one, to say the less) (gras) - Cleanup the known architecture table. Reorder the entries to group what should be, and use a more consistent naming scheme. (some of the test dataset are still to be regenerated) - New! Allow library to register globals on each process just as userdata does. This is implemented using a xbt_dict and not a xbt_set, so we loose the lookup time (for now). Use it in msg and trp. This cleans a lot the internals and helps enforcing privacy of the headers between the gras components. - New! Add a timer mechanism, not unlike cron(8) and at(1). - Bugfix: gras_os_time was delirious in RL. - Bugfix: gras_trp_select/RL don't run into the wall when asked to select onto 0 sockets. - Reenable GRAS now that it works. -- Arnaud Legrand Mon, 14 Feb 2005 14:02:13 -0800 SimGrid (2.90) unstable; urgency=low Alpha 1 on the path to SimGrid 3 * It is a long time since the last release of SimGrid. I'm sorry about that but as I had told you, I was rewriting a lot of things. I apologize to those who had been reporting bugs to me and that I had not answered. If your bug is still in the new version, please tell me. Here is a summary of the main changes. * REVOLUTION 1: The SimGrid project has merged with the GRAS project lead by Martin Quinson. As a consequence SimGrid gains a lot in portability, speed, and a lot more but you'll figure it out later. SimGrid now comprises 3 different projects : MSG, GRAS and SMPI. I wanted to release the new MSG as soon as possible and I have broken GRAS, which is the reason why, for now, only MSG is fully functional. A laconic description of these projects is available in the documentation. * REVOLUTION 2: I have removed SG and I am now using a new simulation kernel optimized for our needs (called SURF but only the developers should use it). Hence, MSG is now roughly 30 times faster and I think that by rewriting a little bit MSG, I could event speed it up a little bit more. Beside the gain in speed, it is also much easier to encode a new platform model with SURF than it was with SG. More to come... * REVOLUTION 3: I have tried to change a little as possible the API of MSG but a few things really had to disappear. The main differences with the previous version are : 1) no more m_links_t and the corresponding functions. Platforms are directly read from a XML description and cannot be hard-coded anymore. The same format is used for application deployment description. The new format is described in the documentation. Have a look in tools/platform_generation. There is a tiny script that converts from the old platform format to the new one. Concerning the application deployment format, parsing the old one is tricky. I think most of you should however be able to convert your files. If it is really an issue, I can write a C code that does the conversion. Let me know. 2) the toolbox tbx does not exist anymore. We now have a library with much more data-structures but without the hash-tables (we have dictionaries that are much faster). -- Arnaud Legrand Mon, 31 Jan 2005 10:45:53 -0800 ***************************************************************************** * Follows the old GRAS changelog. It does not follow the same syntax, but I * * don't feel like converting the oldies. (Mt) * ***************************************************************************** 2005-01-31 Arnaud Version 2.90: "the long awaited one" - Finished rewriting and debugging MSG. Rewrote the documentation. - disable GRAS for now since it needs to be ported to the newest SG 2004-12-16 Martin - Finish the port to windows (using mingw32 for cross-compile) 2004-11-28 Arnaud - Main loop and datastructures of SURF. A cpu resource object is functional. Surf can thus be used to create cpu's with variable performance on which you can execute some actions. 2004-11-15 Martin Quinson - Port to ARM. Simply added the alignment and size descriptions. Should work, but the ARM machines are so slow that I didn't had the opportunity to 'make check' over there yet. 2004-11-15 Arnaud Legrand - Trace manager now written. It uses a heap structure and is therefore expected to be efficient. It may however be speeded up (particularly when many events occur at the same date) by using red and black trees. One day maybe... - Max-min linear system solver written. It uses a sparse matrix structure taking advantage of its expected use. Most operations are O(1) and free/calloc are called as few as possible. The computation of the minimum could however be improved by using a red and black tree (again ! ;). 2004-11-03 Arnaud Legrand - Rename every gras_* function that was in xbt/ to its xbt_ counterpart. - Add a heap and a doubly-linked list to xbt - Added a dichotomy to the dictionaries. make check works as well before so I assume that the patch is correct. I do not know however if things run effectively faster than before now. :) Inclusion of the SimGrid tree in the GRAS one. The archive is renamed to SimGrid, and the version number is bumped to 2.x 2004-10-29 Martin Quinson - Introduction of the remote errors. They are the result of a RMI/RPC on the remote machine. ErrCodes being scalar values, you can't get the host on which those errors did happen. Extending the error mechanism as in Gnome is possible. No idea yet whether it is a good idea. 2004-10-28 Martin Quinson - Interface revolution: the Starred Structure Eradication. I used to do typedef struct {} toto_t; and then handle *toto_t. Arnaud (and Oli) didn't like it, and I surrendered. Now, you have: - ???_t is a valid type (builded with typedef) - s_toto_t is a structure (access to fields with .) - s_toto is a structure needing 'struct' keyword to be used - e_toto_t is an enum - toto_t is an 'object' (struct*) Exemple: typedef struct s_toto {} s_toto_t, *toto_t; typedef enum {} e_toto_t; Moreover, only toto_t (and e_toto_t) are public. The rest (mainly s_toto_t) is private. - While I was at it, all gras__free() functions want a gras__t* so that it can set the variable to NULL. It was so for dicts and sets, it changed for dynars. - Fix a bunch of memleaks in dict_remove - Fix a bug in sg/server_socket opening: it failed all the time. 2004-10-07 Martin Quinson - Speed up dynar lookup operation a bit. gras_dynar_get is dead. Now, you can choose between gras_dynar_get_cpy (the old gras_dynar_get but should be avoided for efficiency reasons) and gras_dynar_get_ptr (which gives you the address of the stored data). gras_dynar_get_as is an helpful macro which allows you to retrieve a copy of the data using an affectation to do the job and not a memcpy. int toto = gras_dynar_get_as(dyn,0,int); rewrites itself to int toto = *(int*)gras_dynar_get_ptr(dyn,0); It does not really speedup the dynar test because they are setting elements all the time (and look them seldom). But the dict does far more lookup than setting. So, this brings the dict_crash test from ~33s to ~25s (200000 elms). 2004-10-05 Martin Quinson - Allow to (en/dis)able the cycle detection at run time. Whether we should check for cycle or not is now a property of each datatype. When you think there may be some cycle, use datadesc_cycle_set. datadesc_cycle_unset allow to remove this property when previously set. Note that the cycle detection is off by default since it impacts the performance. Watch the data you feed GRAS with ;) This property is hereditary. Any element embedded in a structure having it set have it set for the time of this data exchange. You should set it both on sender and receiver side. If you don't set it on sender side, it will enter an endless loop. If you forget on receiver side, the cycles won't be recreated after communication. - Header reorganization. Kill gras_private.h, each submodule must load the headers it needs. 2004-10-04 Martin Quinson - Interface revolution: do not try to survive to malloc failure. Now, gras_malloc and friends call gras_abort() on failure. As a conclusion, malloc_error is not a valid error anymore, and all functions for which it was the only gras_error_t return value are changed. They now return void, or there result directly. This simplify the API a lot. 2004-09-29 Martin Quinson - Re-enable raw sockets. Created by gras_socket_{client,server}_ext; Used with gras_raw_{send,recv} No select possible. It should allow to kill the last bits of gras first version soon. This is not completely satisfactory yet (duplicate code with chunk_{send,recv}; a bit out of the plugin mechanism), but it should work. - Simplify transport plugin (internal) interface by not passing any argument to _server and _client, but embedding them in the socket struct directly. 2004-09-28 Martin Quinson - Finish the port to AIX. autoconf was my problem (segfault within the malloc replacement function. No idea why) 2004-09-16 Martin Quinson - Fix some size_t madness on 64bit architectures. 2004-09-08 Martin Quinson - Reduce the number of system headers loaded, overload some more system calls (such as malloc to cast the result of the system one, and work properly on AIX) - Fix and reintroduce the config support 2004-09-07 Martin Quinson - Source code reorganization to allow Arnaud to surf all over there. - Allow to document the logging categories. - Remove all uppercase from logging categories and useless cleanup in names. 2004-08-18 Martin Quinson Version 0.6.2 (protocol not changed; API changed) - Interface cleanup: gras_msgtype_by_name returns the type (instead of a gras_error_t), and NULL when not found. Functions expecting a msgtype as argument (msg_wait; msg_send) deal with NULL argument by providing a hopefully usefull message. - Portability to prehistoric sparcs again 2004-08-17 Martin Quinson Version 0.6.1 (protocol not changed; ABI not changed) - prealloc some buffers to speed things up 2004-08-11 Martin Quinson Version 0.6 (protocol not changed; ABI expended) - The parsing macro can deal with the references, provided that you add the relevant annotations (using GRAS_ANNOTE(size,field_name)) 2004-08-09 Martin Quinson Version 0.5 (protocol not changed; ABI changed) - Allow to off turn the cycle detection code in data exchange at compilation time. It should be at run time, but I'm short of time (and the config stuff is still broken). That way, we keep dict out of the critical path, which is good because the performance is poor: - search not dichotomial yet - dynar give no way to access their content and memcpy everytime - In composed data description (struct, ref and so on), stop foolness of keeping the subtype's ID, but store the type itself. This keeps sets out of the critical path, which is good since they rely on dynar and dictionnaries. The only loose of that is that we cannot detect the redeclaration of a structure/union with another content (but I'm not sure the code detected well this error before anyway). We still can detect the redefinition discrepancy for the other types. - Use a whole bunch of optimisation flags (plus -fno-strict-aliasing since it breaks the code because of type-punning used all over the place). This breaks on all non-gcc architectures (for now). All those changes (plus the buffer of last time) allow me to gain 2 order of magnitude on cruel tests consisting of 800000 array of integers on two level of a hierarchical structure (200 secondes -> 4 secondes) API change: - the selector of reference must now return the type it points to, not the ID of this type. 2004-08-06 Martin Quinson Version 0.4 (protocol changed; ABI not changed) - Allow to pass --gras-log argument to processes in simulation mode. Really. - New debugging level: trace (under debug) to see effect of GRAS_IN/OUT - Implement a buffer transport, and use it by default (it relies on tcp in real life and on sg in simulation). That's a bit hackish since I had a new field to the structure to store its data without interfering with the subtype ones. Inheritance is tricky in C. And that's a kind of reverse inheritance with one class derivating two classes. Or maybe a game with java interfaces. Anyway, that's damn hard in C (at least). Moreover, I got tired while trying to ensure plugin separation and genericity in SG mode. MSG wants me to do weird things, so let's go for cruel hacks (temporarily of course ;). See comment in transport_private.h:71 - do not include all the _interface headers in private but in the files which really need them (to cut the compilation time when they are modified) 2004-07-26 Martin Quinson Version 0.3 (protocol not changed; ABI changed) - Major overhault of the datadesc interface to simplify it: - shorted the function names: s/gras_datadesc_declare_struct/gras_datadesc_struct/ and so on - add a trivial way to push/pop integers into the cbps without malloc. This allows to make really generic sub_type description, which simply pop their size of the stack. - add a function gras_datadesc_ref_pop_arr() which does what users want most of the time: Declare a dynamic array (which pops its size of the stack) and declare a reference to it. Poor name, but anyway. - kill the post-send callback, add a post-receive one 2004-07-23 Martin Quinson Version 0.2 (protocol changed; ABI changed) - add some testing for cpbs in the test cases, and fix some more bugs. This invalidate again the little64 data file, since I cannot regenerate it myself. - remove an awfull optimization in the logging stuff, allowing me to: - understand it again - learn gcc how to check that the argument match the provided format - fix all errors revealed by gcc after that - internal keys of dict are not \0 terminated. Deal with it properly in loggings instead of segfaulting when the user want to see the logs :-/ 2004-07-22 Martin Quinson - Fix some stupid bug preventing cbps (callback postit) from working 2004-07-21 Martin Quinson - Some documentation cleanups - remove the useless last argument of msgtype_declare - rename the Virtu functions to fit into the 'os' namespace - move headers src/include -> src/include/gras/ and stop fooling with gras -> . symbolic link - make distcheck is now successful 2004-07-19 Martin Quinson Version 0.1.1 - Build shared library also - Install html doc to the right location - stop removing maintainer files in make clean - build tests only on make check 2004-07-13 Martin Quinson version 0.1 - No major issue in previous version => change versioning schema - Re-enable little64 convertion test now that Abdou kindly regenerated the corresponding dataset. 2004-07-11 Martin Quinson version 0.0.4 - Get it working with any kind of structure (we can compute the padding bytes remotely for all the architectures I have access to) - Implement the structure parsing macro (still not quite robust/complete) - Improvement to the remote testing toysuite 2004-07-10 Martin Quinson [autoconf mechanism] - get ride of a bunch of deprecated macros - actually run the test for two-compliment, not only compile it :-/ - test whether the structures get packed (and bail out if yes. Damn. Alignment is a serious matter) - test whether the structures get compacted (but respecting the alignment constraints of each types) - test whether the array fields of structures can straddle alignment boundaries [base] - Damnit, double are bigger than float (typo in creation of 'double' datadesc) (took me 2 hours to find that bug, looking at the wrong place) - Add gras_datadesc_declare_{union,struct}_close(). They must be used before sending/receiving and are used to compute the offsets of fields - Given that padding size depend even on compiler options, keep track of alignment and aligned_size only for the current architecture. Not a big deal since we send structure fields one after the other (seems reasonable). - Add the datastructure used for IEEE paper by the PBIO guys to the test program, let it work on linux/gcc/little32. portability todo. 2004-07-08 Martin Quinson - import and improve remote compilation support from FAST - make sure make check works on half a dozen of machines out there 2004-07-07 Martin Quinson Let's say it's version 0.0.3 ;) - Implement conversions (yuhu!) - Let it work on solaris (beside conversion, of course) - Stupid me, using rand() to generate the conversion datatests in not wise. 2004-07-06 Martin Quinson - Let make dist work, since I'm gonna need it to compile on remote hosts - Let Tests/datadesc_usage write the architecture on which the file was generated as first byte. - Add PowerPC (being also IRIX64), SPARC (also power4) and ALPHA architecture descriptions. - Add datadesc_usage.{i386,ppc,sparc} files being the result of execution on those architectures. - Optimization: send/recv array of scalar in one shoot 2004-07-05 Martin Quinson - YEAH! GRAS/SG and GRAS/RL are both able to run the ping example ! - Plug a whole bunch of memleaks - each process now have to call gras_{init,exit}. One day, their log settings will be separated - Continue the code factorisation between SG, RL and common in Transport. 2004-07-04 Martin Quinson [Transport] - Redistribution between SG and RL. We wanna have to accept in SG, so move accepted related parts of RL in the common part. (more precisely, the dynar of all known sockets is no more a static in transport.c, but part of the process_data) [Core/module.c] [gras_stub_generator] - Bug fix: Do call gras_process_init from gras_init (wasnt called in RL). 2004-07-03 Martin Quinson - Create a new log channel tbx containing dict, set, log, dynar (to shut them all up in one shot) [DataDesc] - Fix the ugly case of reference to dynamic array. - New (semi-public) function gras_datadesc_size to allow the messaging layer to malloc the needed space for the buffer. [Transport] - gras_socket_close now expect the socket to close (and not its address to put NULL in it after it). This is because the socket passed to handlers is one of their argument (=> not writable). [Messaging] - propagate the interface cleanup from last week in datadesc, ie remove a superfluous level of indirection. User pass adress of variable containing data (both when sending and receiving), and not of a variable being a pointer to the data. Let's say that I like it better ;) The price for that is constructs like "int msg=*(int*)payload" in handlers, but it's a fine price, IMHO. [examples/ping] - Let it work in RL (yuhu) 2004-06-21 Martin Quinson [Transport] - porting SG plugin and SG select to new standards (works almost). - plug memleaks and fix bugs around. [DataDesc] - cleanup the prototype of data recv and force users to specify when they want to handle references to objects. Test case working even for cycles. - plug memleaks. Valgrind is perfectly ok with this. 2004-06-12 Martin Quinson [Transport] - cleanup the separation between plugin and main code in plugin creation 2004-06-11 Martin Quinson [Transport] - Reput hook for raw sockets, needed for BW experiments - kill a few lines of dead code [Data description] Interface cleanup - gras_datadesc_by_name returns the searched type or NULL. That way, no variable is needed to use a type desc once, which makes the code clearer. - gras_datadesc_declare_[struct|union]_append_name is removed. The last two parameters were strings (field name, type name), leading to common errors. [Dicos] Interface cleanup - gras_dico_retrieve -> gras_dico_get ; gras_dico_insert -> gras_dico_set This is consistant with the dynar API. 2004-04-21 Martin Quinson [Messaging] - Porting to new standards. [Data description] - interface cleanup. There is no bag anymore, no need to take extra provision to mask the pointers behind "ID". Better splitup of functions between files create/exchange/convert. This is still a bit artificial since convert and receive are so interleaved, but anyway. [Virtu(process)] - add a queued message list to procdata (the ones not matching criteria in msg_wait) - factorize some more code between SG and RL wrt procdata [Tests] - use gras_exit in example to track memleaks - get rid of gs_example now that GS is properly integrated into gras - update run_test to integrate the latest tests (datadesc) [Logging] - rename WARNINGn macros to WARNn since it prooved error-prone 2004-04-19 Martin Quinson [Data description] - register init/exit functions within gras module mechanism - send/receive function. Convertion is not implemented, but short-cutted if not needed. struct/array elements are sent one by one (instead of block-wise), but nobody really cares (yet). Get a prototype before optimizing. - tests (using a file socket) for DD send/receive on: - base types: int, float - array: fixed size, string (ie ref to dynamic string) - structure: homogeneous, heterogeneous - chained list, graph with cycle Believe it or not, valgrind is not too unhappy with the results. The cycle happily segfaults, but the others are ok. And I'm sick of pointers for now. [Transport] [File plugin] - Bugfix when using a filename explicitely (instead of '-') 2004-04-09 Martin Quinson [Transport plugins] - factorize more code between RL and SG in socket creation - Complete the implementation and tests of: o TCP o file (only in RL, and mainly for debugging) I lost 3 days to design a portable address resolver, and then decided that the prototype mainly have to run on my box. Addressing portability too early may be like optimizing too early :-/ [Tests] - use gras_init in the Tests instead of the crappy parse_log_opt (the latter function is removed) [Conditional execution] - New functions: gras_if_RL/gras_if_SG (basic support for this) [Code reorganisation] - Get rid of libgrasutils.a since it makes more trouble than it solves. Build examples against the RL library, since there is no way to disable its creation for now. For information, the beginning of coding on GRAS was back in june 2003. I guess that every line has been rewritten at least twice since then. SimGrid-3.18/tools/0000755000175000017500000000000013217757343014463 5ustar mquinsonmquinsonSimGrid-3.18/tools/doxygen/0000755000175000017500000000000013217757325016140 5ustar mquinsonmquinsonSimGrid-3.18/tools/doxygen/xbt_log_extract_hierarchy.pl0000755000175000017500000000734313217757325023735 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright (c) 2008, 2010, 2012-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; my $debug = 0; print "/* Generated file, do not edit */\n"; print "/** \\addtogroup XBT_log_cats\n"; print " \@{\n"; # Search for calls to macros defining new channels, and prepare the tree representation my %ancestor; my %desc; # $ancestor{"toto"} is the ancestor of the toto channel as declared by XBT_LOG_NEW_SUBCATEGORY and # XBT_LOG_NEW_DEFAULT_SUBCATEGORY ie, when the channel toto is initialized (does not work under windows) # $desc{"toto"} is its description my %connected; # $connected{"toto"} is defined if XBT_LOG_CONNECT("toto") is used sub cleanup_ctn { my $ctn = shift; # cleanup the content of a macro call $ctn =~ s/^\s*(.*)\s*$/$1/gs; my @elms; print "ctn=$ctn\n" if $debug > 1; if ($ctn =~ m/^(\w+)\s*,\s*(\w+)\s*,\s*"?([^"]*)"?$/s) { # Perfect, we got 0->name; 1->anc; 2->desc $elms[0] = $1; $elms[1] = $2; $elms[2] = $3; } elsif ($ctn =~ m/^(\w+)\s*,\s*"?([^"]*)"?$/s) { # Mmm. got no ancestor. Add the default one. $elms[0] = $1; $elms[1] = "XBT_LOG_ROOT_CAT"; $elms[2] = $2; } else { die "Unparsable content: $ctn\n"; } $elms[2] =~ s/\\\\/\\/gs; return @elms; } sub parse_file { my $filename = shift; my $data = ""; print "Parse $filename\n" if $debug; open IN, "$filename" || die "Cannot read $filename: $!\n"; while () { $data .= $_; } close IN; # Purge $data from C comments $data =~ s|/\*.*?\*/||sg; # C++ comments are forbiden in SG for portability reasons, but deal with it anyway $data =~ s|//.*$||mg; my $connect_data = $data; # save a copy for second parsing phase while ($data =~ s/^.*?XBT_LOG_NEW(_DEFAULT)?_(SUB)?CATEGORY\(//s) { $data =~ s/([^"]*"[^"]*")\)//s || die "unparsable macro: $data"; my ($name,$anc,$desc) = cleanup_ctn($1); # build the tree, checking for name conflict die "ERROR: Category name conflict: $name used several times (in $ancestor{$name} and $anc, last time in $filename)\n" if defined ($ancestor{$name}) && $ancestor{$name} ne $anc && defined ($desc{$name}) && $desc{$name} ne $desc; $ancestor{$name}=$anc; $desc{$name}=$desc; print " $name -> $anc\n" if $debug; } # Now, look for XBT_LOG_CONNECT calls $data = $connect_data; while ($data =~ s/^.*?XBT_LOG_CONNECT\(//s) { $data =~ s/\s*(\w+)\s*\)//s || die "unparsable macro: $data"; $connected{$1} = 1; } } # Retrieve all the file names, and add their content to $data my $data; open FILES, "find ../src/ ../tools/ ../include/ -name '*.c' -o -name '*.cpp' |" || die "Cannot search for the source file names: $!\n"; while (my $file=) { chomp $file; parse_file($file); } parse_file("../include/xbt/sysdep.h"); close FILES; # Display the tree, looking for disconnected elems my %used; sub display_subtree { my $name=shift; my $indent=shift; $used{$name} = 1; unless ($name eq "XBT_LOG_ROOT_CAT") { # do not display the root print "$indent - $name: ".($desc{$name}|| "(undocumented)")."\n"; } foreach my $cat (grep {$ancestor{$_} eq $name} sort keys %ancestor) { display_subtree($cat,"$indent "); } } display_subtree("XBT_LOG_ROOT_CAT",""); map { warn "Category $_ does not seem to be connected. Use XBT_LOG_CONNECT($_).\n"; } grep {!defined $connected{$_}} sort keys %ancestor; map { warn "Category $_ does not seem to be connected to the root (anc=$ancestor{$_})\n"; } grep {!defined $used{$_}} sort keys %ancestor; print "@}*/\n"; SimGrid-3.18/tools/doxygen/fig2dev_postprocessor.pl0000755000175000017500000000100213217757325023024 0ustar mquinsonmquinson#! /usr/bin/env perl # Copyright (c) 2010, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; my $print = 0; my $line; while($line=) { chomp $line; if($line=~/IMG/) {$print=1;} if($print) {print $line."\n";} if($line=~/\/MAP/) {$print=0;} } #perl -pe 's/imagemap/simgrid_modules/g'| perl -pe 's/ header DEFEVENTS=$(grep Define < header | awk '{ print $3 }') GREP="" GREP2="" for i in $DEFEVENTS do GREP="/^$i /d; $GREP" GREP2="-e '^$i ' $GREP2" done GREP="/^%\ /d; /^% /d; /^%E/d; $GREP" grep $GREP2 < $TRACE > types /bin/sed -e "$GREP" $TRACE > events sort -n -k 2 -s < events > events.sorted cat header types events.sorted > $OUTPUT rm types events events.sorted header echo "output: $OUTPUT" SimGrid-3.18/tools/CMakeLists.txt0000644000175000017500000000203313217757323017217 0ustar mquinsonmquinsonset(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/fix-paje-trace.sh ${CMAKE_CURRENT_SOURCE_DIR}/generate-dwarf-functions ${CMAKE_CURRENT_SOURCE_DIR}/normalize-pointers.py ${CMAKE_CURRENT_SOURCE_DIR}/sg_unit_extractor.pl ${CMAKE_CURRENT_SOURCE_DIR}/sg_xml_unit_converter.py ${CMAKE_CURRENT_SOURCE_DIR}/simgrid_update_xml.pl ${CMAKE_CURRENT_SOURCE_DIR}/doxygen/fig2dev_postprocessor.pl ${CMAKE_CURRENT_SOURCE_DIR}/doxygen/xbt_log_extract_hierarchy.pl ${CMAKE_CURRENT_SOURCE_DIR}/MSG_visualization/colorize.pl ${CMAKE_CURRENT_SOURCE_DIR}/MSG_visualization/trace2fig.pl PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/simgrid.supp ${CMAKE_CURRENT_SOURCE_DIR}/simgrid2vite.sed PARENT_SCOPE) SimGrid-3.18/tools/smpi/0000755000175000017500000000000013217757322015430 5ustar mquinsonmquinsonSimGrid-3.18/tools/smpi/generate_smpi_defines.pl0000755000175000017500000000403613217757322022312 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright 2016 Vincent Danjean # # Call this script like this: # C/C++ : ./generate_smpi_defines.pl ../../include/smpi/smpi.h # FORTRAN: ./generate_smpi_defines.pl -f ../../include/smpi/smpi.h # # It will generate macros that are useful for adding file and line numbers to # traces without obtaining a backtrace (as this would be very slow and make # the simulation/trace unreliable when compared to a "real" trace as obtained # with MPI+TAU). use strict; use warnings; use Getopt::Std; my %options=(); getopts("fc", \%options); # $incall denotes whether we are currently parsing a macro or not... my $incall=0; my $commentChar="//"; if (defined $options{f}) { $commentChar="!" } print "$commentChar This file has been automatically generated by the script\n"; print "$commentChar in tools/smpi/generate_smpi_defines.pl\n"; print "$commentChar DO NOT EDIT MANUALLY. ALL CHANGES WILL BE OVERWRITTEN!\n"; # Formatting of the output sub output_macro { my $line = shift; my @parts = split (/\s*,\s*/, $line); my $id = $parts[1]; # This is a GCC extension. The last statement is the value of the expression # in parentheses. if (defined $options{f}) { print "#define ". uc($id) ." smpi_trace_set_call_location(__FILE__,__LINE__); call ". lc $id ."\n"; } else { if ($id eq "MPI_Init") { print "#define MPI_Init(...) ({ smpi_process_init(__VA_ARGS__); smpi_trace_set_call_location(__FILE__,__LINE__); MPI_Init(NULL, NULL); })\n"; } else { print "#define $id(...) ({ smpi_trace_set_call_location(__FILE__,__LINE__); $id(__VA_ARGS__); })\n"; } } } my $wholemacro; my $line; while (defined($line = <>)) { chomp($line); if ($line =~ /^MPI_CALL/) { if ($incall) { output_macro($wholemacro); } $incall=1; $wholemacro = $line; } elsif ($incall && $line =~ /^\s+\S/) { # Did we already start parsing an MPI_CALL macro? If so, just concatenate $wholemacro .= ' '.$line; } elsif ($incall) { output_macro($wholemacro); $incall=0; } } SimGrid-3.18/tools/normalize-pointers.py0000755000175000017500000000154513217757325020706 0ustar mquinsonmquinson#!/usr/bin/env python # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. """ Tool for normalizing pointers such as two runs have the same 'addresses' first address encountered will be replaced by 0X0000001, second by 0X0000002, ... """ import sys import re if len(sys.argv) != 2: print "Usage ./normalize-pointers.py " sys.exit(1) f = open(sys.argv[1]) t = f.read() f.close() r = re.compile(r"0x[0-9a-f]+") s = r.search(t) offset = 0 pointers = {} while (s): if s.group() not in pointers: pointers[s.group()] = "0X%07d" % len(pointers) print t[offset:s.start()], print pointers[s.group()], offset = s.end() s = r.search(t, offset) print t[offset:] SimGrid-3.18/tools/simgrid.supp0000644000175000017500000001004013217757343017025 0ustar mquinsonmquinson# Valgrind suppressions for stuff that we cannot control # Memory leaks in standard tools (e.g. dash, tail, or sort) { Memory leak in /bin tools Memcheck:Leak ... obj:/bin/* } { Memory leak in /usr/bin tools Memcheck:Leak ... obj:/usr/bin/* } { Memory leak in cmake Memcheck:Leak match-leak-kinds:reachable ... fun:_Znwm fun:_ZN4Json5Value13nullSingletonEv obj:*/libjsoncpp.so* ... fun:_dl_init } # There's a constant leak of 56 bytes in the depths of libc which # manifests, for example, when using backtrace() { Memory leak in libc/dlopen with -pthread Memcheck:Leak fun:malloc fun:_dl_map_object_deps fun:dl_open_worker fun:_dl_catch_error fun:_dl_open fun:do_dlopen fun:_dl_catch_error fun:dlerror_run fun:__libc_dlopen_mode } # Another problem in glibc, where makecontext does not reset the EBP register, # and backtrace goes too far when walking up the stack frames { Invalid read in backtrace, called after makecontext Memcheck:Addr4 fun:backtrace ... fun:makecontext } #There seems to be an issue with libc using an uninitialized value somewhere in dlopen { Invalid read in dl_start Memcheck:Cond fun:index fun:expand_dynamic_string_token ... fun:_dl_start } # 72704 bytes leak from GCC >5.1 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64535 { Memory leak in dl_init Memcheck:Leak match-leak-kinds:reachable fun:malloc obj:/usr/lib/*/libstdc++.so.* fun:call_init.part.0 ... fun:_dl_init } #Ignore leaks in SMPI sample codes { Leaks in SMPI sample codes Memcheck:Leak match-leak-kinds: all fun:malloc fun:smpi_simulated_main_ } #SMPI leaks the dlopen handle used to load the program { dlopen handle leaks (1/3) Memcheck:Leak match-leak-kinds:reachable fun:malloc ... fun:dlopen@@GLIBC_* } { dlopen handle leaks (2/3) Memcheck:Leak match-leak-kinds:reachable fun:calloc ... fun:dlopen@@GLIBC_* } { dlopen handle leaks (3/3) Memcheck:Leak match-leak-kinds:reachable fun:realloc ... fun:dlopen@@GLIBC_* } # Memory leaks appearing to be in libcgraph. They can be seen with the # following simple program: # ,---- # | #include # | #include # | int main(int argc, char *argv[]) # | { # | if (argc == 1) { # | printf("Usage: %s \n", argv[0]); # | return 1; # | } # | Agraph_t *g; # | FILE *inf = fopen(argv[1], "r"); # | g = agread(inf, 0); # | fclose(inf); # | agclose(g); # | return 0; # | } # `---- { Memory leak in libcgraph (1/2) Memcheck:Leak fun:malloc ... obj:/usr/lib/libcgraph.so* fun:aaglex fun:aagparse fun:agconcat } { Memory leak in libcgraph (1/2) Memcheck:Leak fun:calloc ... obj:/usr/lib/libcgraph.so* fun:aagparse fun:agconcat } { Memory leak in libcgraph (2/2) Memcheck:Leak fun:malloc ... fun:agnode obj:/usr/lib/libcgraph.so* fun:aagparse fun:agconcat } # We're not interested by memory leaks in the Lua interpreter { Memory leak in lua Memcheck:Leak ... fun:luaD_precall } # libunwind seems to be using msync poorly, thus triggering these # https://github.com/JuliaLang/julia/issues/4533 { msync unwind Memcheck:Param msync(start) ... obj:*/libpthread*.so ... } { ignore unwind cruft Memcheck:Param rt_sigprocmask(set) ... obj:/usr/lib/x86_64-linux-gnu/libunwind.so.* ... } { ignore unwind cruft Memcheck:Param msync(start) ... obj:/usr/lib/x86_64-linux-gnu/libunwind.so.* ... } { ignore unwind invalid reads Memcheck:Addr8 fun:_Ux86_64_setcontext } # Java cruft { JavaCruft 1 Memcheck:Addr4 ... fun:_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread fun:JVM_DoPrivileged ... } { JavaCruft 2 Memcheck:Cond ... fun:_ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask ... } { Somewhere within the Java conditions and monitors Memcheck:Cond fun:MarsagliaXORV ... } SimGrid-3.18/tools/stack-cleaner/0000755000175000017500000000000013217757322017174 5ustar mquinsonmquinsonSimGrid-3.18/tools/stack-cleaner/cc0000755000175000017500000000010513217757322017503 0ustar mquinsonmquinson#!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper cc "$@" SimGrid-3.18/tools/stack-cleaner/compiler-wrapper0000755000175000017500000000165313217757322022417 0ustar mquinsonmquinson#!/usr/bin/env python # Compiler wrapper with stack-cleaner support (enabled by default). # Usage: ./compiler-wrapper target-compiler args [-f[no-]stack-cleaner] import sys import os import re compiler = sys.argv[1] enabled = True args = [] for arg in sys.argv[2:]: if arg == "--help": sys.stderr.write( "Compiler wrapper with stack-cleaner.\n" "Usage: {} compiler [-f[no-]stack-cleaner] [options]\n".format(sys.argv[0])) sys.exit(0) elif arg == "-fno-stack-cleaner": enabled = False elif arg == "-fstack-cleaner": enabled = True else: args.append(arg) if enabled: if re.match("^clang", os.path.basename(compiler)): args.insert(0, "-no-integrated-as") args.insert(0, os.path.dirname(sys.argv[0])) args.insert(0, "-B") args.insert(0, compiler) os.execvp(args[0], args) sys.stderr.write("compiler-wrapper: Could not exec\n") os.exit(1) SimGrid-3.18/tools/stack-cleaner/fortran0000755000175000017500000000011313217757322020570 0ustar mquinsonmquinson#!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper gfortran "$@" SimGrid-3.18/tools/stack-cleaner/clean-stack-filter0000755000175000017500000000327413217757322022600 0ustar mquinsonmquinson#!/usr/bin/env perl # Transform assembly in order to clean each stack frame for X86_64. use strict; use warnings; $SIG{__WARN__} = sub { die @_ }; # Whether we are still scanning the content of a function: our $scanproc = 0; # Save lines of the function: our $lines = ""; # Size of the stack for this function: our $size = 0; # Counter for assigning unique ids to labels: our $id=0; sub emit_code { my $qsize = $size / 8; my $offset = - $size - 8; if($size != 0) { # This is a crude hack to disable the stack cleaning on the main # stack. It relies on the fact that the main stack is high in # the address space and the other stacks are in the heap (lower). print("\tmovq \$0x7fff00000000, %r11\n"); print("\tcmpq %r11, %rsp\n"); print("\tjae .Lstack_cleaner_done$id\n"); # Loop over the stack frame quadwords and zero it: print("\tmovabsq \$$qsize, %r11\n"); print(".Lstack_cleaner_loop$id:\n"); print("\tmovq \$0, $offset(%rsp,%r11,8)\n"); print("\tsubq \$1, %r11\n"); print("\tjne .Lstack_cleaner_loop$id\n"); print(".Lstack_cleaner_done$id:\n"); } print $lines; $id = $id + 1; $size = 0; $lines = ""; $scanproc = 0; } while (<>) { if ($scanproc) { $lines = $lines . $_; if (m/^[ \t]*.cfi_endproc$/) { emit_code(); } elsif (m/^[ \t]*pushq/) { $size += 8; } elsif (m/^[ \t]*subq[\t *]\$([0-9]*),[ \t]*%rsp$/) { my $val = $1; $val = oct($val) if $val =~ /^0/; $size += $val; emit_code(); } } elsif (m/^[ \t]*.cfi_startproc$/) { print $_; $scanproc = 1; } else { print $_; } } SimGrid-3.18/tools/stack-cleaner/README0000644000175000017500000000120413217757322020051 0ustar mquinsonmquinsonProvides stack-cleaning compilers for x86_64: * as * cc * c++ * fortran Each of them modify the generated/given X86_64 assembly by prepending stack-cleanup code to each function: movq $0x7ffff7ff8000, %r11 cmpq %r11, %rsp jbe .Lstack_cleaner_done0 movq $QSIZE, %r11 .Lstack_cleaner_loop0: movq $0, OFFSET(%rsp,%r11,8) subq $1, %r11 jne .Lstack_cleaner_loop0: .Lstack_cleaner_done0: The modification of the assembly is done by the clean-stack-filter program. If the underlying compiler is clang, it might be necessary to add the -no-integrated-as flag in order to force the usage of an external assembler. SimGrid-3.18/tools/stack-cleaner/c++0000755000175000017500000000010613217757322017467 0ustar mquinsonmquinson#!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper c++ "$@" SimGrid-3.18/tools/stack-cleaner/as0000755000175000017500000000307013217757322017525 0ustar mquinsonmquinson#!/usr/bin/env python # Wrapper around the real `as` which adds filtering capabilities. import os import re import subprocess import sys import tempfile # Process the arguments: # * build the argument array for calling the real assembler; # * get the input file name. args = [] input_filename = None args.append('as') i = 1 while i < len(sys.argv): if sys.argv[i] == '-o' or sys.argv[i] == '-I': args.append(sys.argv[i]) if i + 1 >= len(sys.argv): sys.stderr.write("Missing argument\n") sys.exit(1) args.append(sys.argv[i + 1]) i = i + 1 elif re.match('^-', sys.argv[i][0]): args.append(sys.argv[i]) elif input_filename: sys.stdout.write("Too many input files\n") sys.exit(1) else: input_filename = sys.argv[i] i = i + 1 if input_filename == None: sys.stderr.write("Missing input file\n") sys.exit(1) temp_file, temp_filename = tempfile.mkstemp(suffix=".s", prefix="as_wrapper") try: # Generate temporary file with modified assembly code: script_file = os.path.join( os.path.dirname(sys.argv[0]), "clean-stack-filter") input_file = os.open(input_filename, os.O_RDONLY) status = subprocess.call([script_file], stdin=input_file, stdout=temp_file) os.close(input_file) if status != 0: sys.stderr.write("Filtering the assembly code failed.\n") sys.exit(status) # Call the real assembler on this modified assembly code: args.append(temp_filename) sys.exit(subprocess.call(args)) finally: os.remove(temp_filename) SimGrid-3.18/tools/sg_unit_extractor.pl0000755000175000017500000002346713217757325020602 0ustar mquinsonmquinson#! /usr/bin/env perl # Copyright (c) 2005-2012, 2014-2017. The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use Getopt::Long qw(GetOptions); my $progname="sg_unit_extractor"; # Get the args sub usage($) { my $ret; print "USAGE: $progname [--root=part/to/cut] [--outdir=where/to/generate/files] infile [infile+]\n\n"; print "This program is in charge of extracting the unit tests out of the SimGrid source code.\n"; print "See http://simgrid.gforge.inria.fr/doc/latest/inside_tests.html for more details.\n"; exit $ret; } my $outdir=undef; my $root; my $help; Getopt::Long::config('permute','no_getopt_compat', 'no_auto_abbrev'); GetOptions( 'help|h' => sub {usage(0)}, 'root=s' =>\$root, 'outdir=s' =>\$outdir) or usage(1); usage(1) if (scalar @ARGV == 0); map {process_one($_)} @ARGV; sub process_one($) { my $infile = shift; my $outfile; $infile =~ s|src/|| unless (-e $infile); $outfile = $infile; $outfile =~ s/\.c$/_unit.c/; $outfile =~ s/\.cpp$/_unit.cpp/; $outfile =~ s|.*/([^/]*)$|$1| if $outfile =~ m|/|; $outfile = "$outdir$outfile"; print "$progname: processing $infile (generating $outfile)...\n"; # Get the unit data my ($unit_source,$suite_name,$suite_title)=("","",""); my (%tests); # to detect multiple definition my (@tests); # actual content open IN, "$infile" || die "$progname: Cannot open input file '$infile': $!\n"; $infile =~ s|$root|| if defined($root); my $takeit=0; my $line=0; my $beginline=0; while () { $line++; if (m/ifdef +SIMGRID_TEST/) { $beginline = $line; $takeit = 1; next; } if (m/endif.*SIMGRID_TEST/) { $takeit = 0; next } if (m/XBT_TEST_SUITE\(\w*"([^"]*)"\w*, *(.*?)\);/) { #" { die "$progname: Multiple suites in the same file ($infile) are not supported yet\n" if length($suite_name); ($suite_name,$suite_title)=($1,$2); die "$progname: Empty suite name in $infile" unless length($suite_name); die "$progname: Empty suite title in $infile" unless length($suite_title); next; } elsif (m/XBT_TEST_SUITE/) { die "$progname: Parse error: This line seem to be a test suite declaration, but failed to parse it\n$_\n"; } if (m/XBT_TEST_UNIT\(\w*"([^"]*)"\w*,([^,]*),(.*?)\)/) { #"{ die "$progname: multiply defined unit in file $infile: $1\n" if (defined($tests{$1})); my @t=($1,$2,$3); push @tests,\@t; $tests{$1} = 1; } elsif (m/XBT_TEST_UNIT/) { die "$progname: Parse error: This line seem to be a test unit, but failed to parse it\n$_\n"; } $unit_source .= $_ if $takeit; } close IN || die "$progname: cannot close input file '$infile': $!\n"; if ($takeit) { die "$progname: end of file reached in SIMGRID_TEST block.\n". "You should end each of the with a line matching: /endif.*SIMGRID_TEST/\n". "Example:\n". "#endif /* SIMGRID_TEST */\n" } die "$progname: no suite defined in $infile\n" unless (length($suite_name)); # Write the test my ($GENERATED)=("/*******************************/\n". "/* GENERATED FILE, DO NOT EDIT */\n". "/*******************************/\n\n"); $beginline+=2; open OUT,">$outfile" || die "$progname: Cannot open output file '$outfile': $!\n"; print OUT $GENERATED; print OUT "#include \n"; print OUT "#include \"xbt.h\"\n"; print OUT $GENERATED; print OUT "#line $beginline \"$infile\" \n"; print OUT "$unit_source"; print OUT $GENERATED; close OUT || die "$progname: Cannot close output file '$outfile': $!\n"; # write the main skeleton if needed if (! -e "${outdir}simgrid_units_main.c") { open OUT,">${outdir}simgrid_units_main.c" || die "$progname: Cannot open main file '${outdir}simgrid_units_main.c': $!\n"; print OUT $GENERATED; print OUT "#include \n\n"; print OUT "#include \"xbt.h\"\n\n"; print OUT "extern xbt_test_unit_t _xbt_current_unit;\n\n"; print OUT "#define STRLEN 1024\n"; print OUT "/* SGU: BEGIN PROTOTYPES */\n"; print OUT "/* SGU: END PROTOTYPES */\n\n"; print OUT $GENERATED; # print OUT "# 93 \"sg_unit_extractor.pl\"\n"; print OUT <) { $newmain .= $_; # print "Look for proto: $_"; last if /SGU: BEGIN PROTOTYPES/; } # search my prototype while () { # print "Seek protos: $_"; last if (/SGU: END PROTOTYPES/ || /SGU: BEGIN FILE $infile/); $newmain .= $_; } if (/SGU: BEGIN FILE $infile/) { # found an old section for this file. Kill it while () { last if /SGU: END FILE/; } $_ = ; # pass extra blank line chomp; die "this line should be blank ($_). Did you edit the file?" if /\W/; } my ($old_)=($_); # add my section $newmain .= " /* SGU: BEGIN FILE $infile */\n"; map { my ($name,$func,$title) = @{$_}; $newmain .= " void $func(void);\n" } @tests; $newmain .= " /* SGU: END FILE */\n\n"; if ($old_ =~ /SGU: BEGIN FILE/ || $old_ =~ /SGU: END PROTOTYPES/) { $newmain .= $old_; } # pass remaining prototypes, search declarations while () { $newmain .= $_ unless /SGU: END PROTOTYPES/; last if /SGU: BEGIN SUITES DECLARATION/; } ### Done with prototypes. And now, the actual code # search my prototype while () { last if (/SGU: END SUITES DECLARATION/ || /SGU: BEGIN FILE $infile/); $newmain .= $_; } if (/SGU: BEGIN FILE $infile/) { # found an old section for this file. Kill it while () { last if /SGU: END FILE/; } $_ = ; # pass extra blank line chomp; die "this line should be blank ($_). Did you edit the file?" if /\W/; } my ($old_)=($_); # add my section $newmain .= " /* SGU: BEGIN FILE $infile */\n"; $newmain .= " suite = xbt_test_suite_by_name(\"$suite_name\",$suite_title);\n"; map { my ($name,$func,$title) = @{$_}; $newmain .= " xbt_test_suite_push(suite, \"$name\", $func, $title);\n"; } @tests; $newmain .= " /* SGU: END FILE */\n\n"; if ($old_ =~ /SGU: BEGIN FILE/ || $old_ =~ /SGU: END SUITES DECLARATION/) { $newmain .= $old_; } # pass the remaining while () { $newmain .= $_; } close IN || die "$progname: Cannot close main file '${outdir}simgrid_units_main.c': $!\n"; # write it back to main open OUT,">${outdir}simgrid_units_main.c" || die "$progname: Cannot open main file '${outdir}simgrid_units_main.c': $!\n"; print OUT $newmain; close OUT || die "$progname: Cannot close main file '${outdir}simgrid_units_main.c': $!\n"; } # end if process_one($) 0; SimGrid-3.18/tools/sg_xml_unit_converter.py0000644000175000017500000000615713217757325021465 0ustar mquinsonmquinson#!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. # grep -ohrI 'bw=".*"' . | sort | uniq import sys import fnmatch import os from decimal import Decimal, DecimalException import re def to_str(dec): return re.sub(r"(\.\d*?)0*$", r"\1", dec.to_eng_string()).rstrip(".") def convert(xml, formats, attrib): res = [] m = re.search(r'%s="(.*?)"' % attrib, xml) while m: b, e = m.span(1) res.append(xml[:b]) val = xml[b:e] xml = xml[e:] try: power = Decimal(val) tmp = to_str(power) for p, f in formats: d = power / p if d >= 1.0: tmp = "%s%s" % (to_str(d), f) break res.append(tmp) except DecimalException: print "Error with:", val res.append(val) m = re.search(r'%s="(.*?)"' % attrib, xml) res.append(xml) return "".join(res) def formats(ll): return sorted(((Decimal(i), j) for i, j in ll), key=lambda x: x[0], reverse=True) for root, dirnames, filenames in os.walk(sys.argv[1]): for filename in fnmatch.filter(filenames, '*.xml'): print root, dirnames, filename path = os.path.join(root, filename) xml = open(path).read() power_formats = formats([("1E0", "f"), ("1E3", "kf"), ("1E6", "Mf"), ("1E9", "Gf"), ("1E12", "Tf"), ("1E15", "Pt"), ("1E18", "Ef"), ("1E21", "Zf")]) xml = convert(xml, power_formats, "power") bandwidth_formats = formats([("1E0", "Bps"), ("1E3", "kBps"), ("1E6", "MBps"), ("1E9", "GBps"), ("1E12", "TBps")]) xml = convert(xml, bandwidth_formats, "bandwidth") xml = convert(xml, bandwidth_formats, "bw") xml = convert(xml, bandwidth_formats, "bb_bw") xml = convert(xml, bandwidth_formats, "bw_in") xml = convert(xml, bandwidth_formats, "bw_out") time_formats = formats([("1E-12", "ps"), ("1E-9", "ns"), ("1E-6", "us"), ("1E-3", "ms"), ("1E0", "s"), ("60E0", "m"), ("3600E0", "h"), ("86400E0", "d"), ("604800E0", "w")]) xml = convert(xml, time_formats, "latency") xml = convert(xml, time_formats, "lat") xml = convert(xml, time_formats, "bb_lat") # print xml outfile = open(path, "w") outfile.write(xml) outfile.close() SimGrid-3.18/tools/tesh/0000755000175000017500000000000013217757337015431 5ustar mquinsonmquinsonSimGrid-3.18/tools/tesh/IO-orders.tesh0000644000175000017500000000157313217757337020127 0ustar mquinsonmquinson p This tests that TESH accepts any order for the input/output p Order: in, out, cmd < < TOTO < > TOTO < $ cat > Test suite from stdin > [(stdin):3] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: out, in, cmd < > TOTO < < TOTO < $ cat > Test suite from stdin > [(stdin):3] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: out, cmd, in < > TOTO < $ cat < < TOTO > Test suite from stdin > [(stdin):2] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: in, cmd, out < < TOTO < $ cat < > TOTO > Test suite from stdin > [(stdin):2] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: cmd, out, in < $ cat < > TOTO < < TOTO > Test suite from stdin > [(stdin):1] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: cmd, in, out < $ cat < < TOTO < > TOTO > Test suite from stdin > [(stdin):1] cat > Test suite from stdin OK $ ${bindir:=.}/tesh SimGrid-3.18/tools/tesh/catch-wrong-output.tesh0000644000175000017500000000047313217757337022074 0ustar mquinsonmquinson#! ./tesh p This tests whether TESH detects wrong outputs ! expect return 2 < > TOTO < < TUTU < $ cat $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):3] cat > Output of <(stdin):3> mismatch: >--- expected >+++ obtained >@@ -1 +1 @@ > -TOTO > +TUTU > Test suite `(stdin)': NOK (<(stdin):3> output mismatch) SimGrid-3.18/tools/tesh/CMakeLists.txt0000644000175000017500000000151613217757323020167 0ustar mquinsonmquinsonconfigure_file("${CMAKE_HOME_DIRECTORY}/tools/tesh/tesh.py" "${CMAKE_BINARY_DIR}/bin/tesh" @ONLY IMMEDIATE) foreach(x setenv set-output-ignore set-output-sort set-return set-timeout background basic bg-basic bg-set-signal catch-return catch-signal catch-timeout catch-wrong-output cd color ignore-regexp IO-bigsize IO-broken-pipe IO-orders) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}.tesh) if(NOT enable_memcheck AND NOT WIN32) ADD_TESH(tesh-self-${x} --setenv bindir=${CMAKE_BINARY_DIR}/bin ${CMAKE_HOME_DIRECTORY}/tools/tesh/${x}.tesh) endif() endforeach() set(tesh_files ${tesh_files} PARENT_SCOPE) set(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/tesh.py ${CMAKE_CURRENT_SOURCE_DIR}/generate_tesh PARENT_SCOPE) SimGrid-3.18/tools/tesh/IO-broken-pipe.tesh0000644000175000017500000000515313217757337021042 0ustar mquinsonmquinson # TESH autotest: check that the father is protected from broken pipes # # If not, it breaks sometimes (when the child is scheduled before the # father), so we do the test a bunch of times. # ! output ignore < blablablablablablablablablablablablablablablablablablablabla 01 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 02 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 03 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 04 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 05 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 06 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 07 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 08 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 09 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 10 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 11 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 12 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 13 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 14 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 15 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 16 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 17 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 18 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 19 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 20 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 21 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 22 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 23 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 24 $ cmake -E echo ! output ignore < blablablablablablablablablablablablablablablablablablablabla 25 $ cmake -E echo SimGrid-3.18/tools/tesh/ignore-regexp.tesh0000644000175000017500000000015213217757337021067 0ustar mquinsonmquinson ! ignore .*0x[0-9A-F]+\. $ printf 'word\nMemory address: 0x42AA42.\nanotherword\n' > word > anotherword SimGrid-3.18/tools/tesh/IO-bigsize.tesh0000644000175000017500000157323613217757337020300 0ustar mquinsonmquinson#! ./tesh # This suite contains two tests: # The first one uses a very big input (150k) to check whether trucated input do work. # The second one uses both a big input and a big output (150k each). # # This checks whether the non-blocking I/O mess is functionnal. # p First, a write test < 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA $ sed -n '$=' > 1000 p And now, a read/write test < 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA $ cat > 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA SimGrid-3.18/tools/tesh/catch-return.tesh0000644000175000017500000000044313217757337020716 0ustar mquinsonmquinson#! ./tesh # This suite builds and uses a program returning 1. # tesh must detect this condition and report the issue. ! expect return 2 < $ perl -e "exit 1" $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):1] perl -e "exit 1" > Test suite `(stdin)': NOK (<(stdin):1> returned code 1) SimGrid-3.18/tools/tesh/set-timeout.tesh0000644000175000017500000000023713217757337020577 0ustar mquinsonmquinson#! ./tesh # This suite must be functional because we changed the timeout value to 10 # before sleeping 6 secs. ! timeout 10 $ sleep 6 ! timeout no $ sleep 1SimGrid-3.18/tools/tesh/basic.tesh0000644000175000017500000000274513217757337017407 0ustar mquinsonmquinson#! ./tesh p This is a basic test < TOTO \ TUTU $ mkfile tmp_fich $ cat tmp_fich > TOTO TUTU ! expect return 2 ! output ignore < $ cat tmp_fich < > TUTU TOTO $ ${bindir:=.}/tesh $ rm tmp_fich p And now, some multilines examples p Really basic multiline < a < b < c < d $ mkfile tmp_fich $ cat tmp_fich > a > b > c > d $ sed -n '$=' tmp_fich > 4 $ rm tmp_fich p Now, check for spaces in input p * leading and trailing spaces... < a < b < c $ sed 's/ /_/g' > __a > _b_ > c__ p * empty lines... < a < < c $ sed '2s/^/b/' > a > b > c < < b < c $ sed '1s/^/a/' > a > b > c p Now that input should be good, check for spaces in output p * leading spaces... < _x $ sed 's/_/ /' > x ! expect return 2 ! output ignore < < _x < $ sed 's/_/ /' < > x $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < x < $ cat < > x $ ${bindir:=.}/tesh p * trailing spaces... < x_ $ sed 's/_/ /' > x ! expect return 2 ! output ignore < < x_ < $ sed 's/_/ /' < > x $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < x < $ cat < > x $ ${bindir:=.}/tesh p * empty lines... < a < < c $ cat > a > > c ! expect return 2 ! output ignore < < a < < < < c < $ cat < > a < > c $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < a < < c < $ cat < > a < > < > c $ ${bindir:=.}/tesh < < b < c $ cat > > b > c ! expect return 2 ! output ignore < < < < b < < c < $ cat < > b < > c $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < b < < c < $ cat < > < > b < > c $ ${bindir:=.}/tesh SimGrid-3.18/tools/tesh/bg-set-signal.tesh0000644000175000017500000000071513217757337020755 0ustar mquinsonmquinson#! ./tesh # This suite builds and uses a program raising a segfault, ie a program dying # of SIGSEV. tesh must detect this condition and report the issue. ! output ignore $ cmake -E remove_directory temp_testdir-bg-set-signal $ mkdir temp_testdir-bg-set-signal $ cd temp_testdir-bg-set-signal < kill 'SEGV', $$; $ mkfile segfault.pl ! expect signal SIGSEGV $ perl segfault.pl $ cd .. ! output ignore $ cmake -E remove_directory temp_testdir-bg-set-signal SimGrid-3.18/tools/tesh/setenv.tesh0000644000175000017500000000070613217757337017625 0ustar mquinsonmquinson#! ./tesh # This suite builds and uses a program returning 1. # tesh is instructed of this return code and must not whine. ! output ignore $ cmake -E remove_directory temp_testdir_setenv $ mkdir temp_testdir_setenv $ cd temp_testdir_setenv < print "tesh_test_toto=$ENV{tesh_test_toto}"; $ mkfile testenv.pl ! setenv tesh_test_toto=blah $ perl testenv.pl > tesh_test_toto=blah $ cd .. ! output ignore $ cmake -E remove_directory temp_testdir_setenv SimGrid-3.18/tools/tesh/set-return.tesh0000644000175000017500000000031113217757337020421 0ustar mquinsonmquinson#! ./tesh # This suite builds and uses a program returning 1. # tesh is instructed of this return code and must not whine. ! expect return 1 $ perl -e "exit 1" ! expect return 42 $ perl -e "exit 42" SimGrid-3.18/tools/tesh/set-output-sort.tesh0000644000175000017500000000525713217757337021445 0ustar mquinsonmquinson#! ./tesh p This tests whether TESH correctly sorts command output < p Test sorting and filtering of output < < $ true < < ! output sort < $ true < < $ printf 'profiling: foo\n' < < $ printf 'profiling: foo' < < ! output sort < $ printf 'profiling: foo\n' < < ! output sort < $ printf 'profiling: foo' < < $ printf 'a\nb\nc\nd\n' < > a < > b < > c < > d < < $ printf 'a\nb\nc\nd' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nd\nb\na\n' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nd\nb\na' < > a < > b < > c < > d < < $ printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz\n' < > a < > b < > c < > d < < $ printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz\n' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz' < > a < > b < > c < > d $ ${bindir:=.}/tesh --ignore-jenkins > Ignore all cruft seen on SimGrid's continous integration servers > Test suite from stdin > [(stdin):1] Test sorting and filtering of output > [(stdin):3] true > [(stdin):6] true > [(stdin):8] printf 'profiling: foo\n' > [(stdin):10] printf 'profiling: foo' > [(stdin):13] printf 'profiling: foo\n' > [(stdin):16] printf 'profiling: foo' > [(stdin):18] printf 'a\nb\nc\nd\n' > [(stdin):24] printf 'a\nb\nc\nd' > [(stdin):31] printf 'c\nd\nb\na\n' > [(stdin):38] printf 'c\nd\nb\na' > [(stdin):44] printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz\n' > [(stdin):50] printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz' > [(stdin):57] printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz\n' > [(stdin):64] printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz' > Test suite from stdin OK p Check the Right Prefix Length (19) for "output sort" ! output sort 19 < 12345678901234567 B line < 12345678901234567 A line $ cat > 12345678901234567 A line > 12345678901234567 B line ! output sort 19 < 123456789012345678 B line < 123456789012345678 A line $ cat > 123456789012345678 B line > 123456789012345678 A line ! expect return 2 ! output ignore < ! output sort 19 < < 123456789012345678 B line < < 123456789012345678 A line < $ cat < > 123456789012345678 A line < > 123456789012345678 B line $ ${bindir:=.}/tesh p Check user-defined prefix length for "output sort" ! output sort 5 < 000 B line < 000 A line $ cat > 000 A line > 000 B line ! output sort 4 < 000 B line < 000 A line $ cat > 000 B line > 000 A line ! expect return 2 ! output ignore < ! output sort 4 < < 000 B line < < 000 A line < $ cat < > 000 A line < > 000 B line $ ${bindir:=.}/tesh SimGrid-3.18/tools/tesh/color.tesh0000644000175000017500000000050413217757337017433 0ustar mquinsonmquinson ! output display $ printf "I \033[0;31mlove\033[0m tesh\n" > I love tesh ! output display $ echo -e "\e[1;4mBold and Underlined\e[0m" > Bold and Underlined ! output display $ echo -e "Default \e[94mLight blue\e[0m" > Default Light blue ! output display $ echo -e "\e[38;5;82mHello \e[38;5;198mWorld\e[0m" > Hello World SimGrid-3.18/tools/tesh/cd.tesh0000644000175000017500000000126413217757337016707 0ustar mquinsonmquinson#! ./tesh # This example uses the cd command ! output ignore $ cmake -E remove_directory testdir_temp-cd $ mkdir testdir_temp-cd $ cd testdir_temp-cd # Check that there is nothing in the current dir (which must be testdir_temp-cd) $ ls # Check that tesh detects properly cd to non-existing directories ! expect return 4 < $ cd toto > Test suite from stdin > Chdir to toto failed: No such file or directory > Test suite `(stdin)': NOK (system error) $ ${bindir:=.}/tesh # The next command checks that there is a testdir_temp-cd in the upper directory, # ie that mkdir and cd both worked. $ test -e ../testdir_temp-cd $ cd .. ! output ignore $ cmake -E remove_directory testdir_temp-cd SimGrid-3.18/tools/tesh/catch-signal.tesh0000644000175000017500000000133713217757337020657 0ustar mquinsonmquinson#! ./tesh # This suite builds and uses a program raising a segfault, ie a program dying # of SIGSEV. tesh must detect this condition and report the issue. ! output ignore $ cmake -E remove_directory temp_testdir-catch-signal $ mkdir temp_testdir-catch-signal $ cd temp_testdir-catch-signal < kill 'SEGV', $$; $ mkfile segfault.pl p Check that we notice when SEGV is raised ! expect signal SIGSEGV $ perl segfault.pl p Check that we return the expected return value on SEGV ! expect return 11 < $ perl segfault.pl $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):1] perl segfault.pl > Test suite `(stdin)': NOK (<(stdin):1> got signal SIGSEGV) $ cd .. ! output ignore $ cmake -E remove_directory temp_testdir-catch-signal SimGrid-3.18/tools/tesh/tesh.py0000755000175000017500000004410713217757325016754 0ustar mquinsonmquinson#! @PYTHON_EXECUTABLE@ # -*- coding: utf-8 -*- """ tesh -- testing shell ======================== Copyright (c) 2012-2017. The SimGrid Team. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the license (GNU LGPL) which comes with this package. #TODO: child of child of child that printfs. Does it work? #TODO: a child dies after its parent. What happen? #TODO: regular expression in output #ex: >> Time taken: [0-9]+s #TODO: linked regular expression in output #ex: # >> Bytes sent: ([0-9]+) # >> Bytes recv: \1 # then, even better: # ! expect (\1 > 500) """ import sys, os import shlex import re import difflib import signal import argparse if sys.version_info[0] == 3: import subprocess import _thread else: raise "This program is expected to run with Python3 only" ############## # # Utilities # # # Singleton metaclass that works in Python 2 & 3 # http://stackoverflow.com/questions/6760685/creating-a-singleton-in-python class _Singleton(type): """ A metaclass that creates a Singleton base class when called. """ _instances = {} def __call__(cls, *args, **kwargs): if cls not in cls._instances: cls._instances[cls] = super(_Singleton, cls).__call__(*args, **kwargs) return cls._instances[cls] class Singleton(_Singleton('SingletonMeta', (object,), {})): pass SIGNALS_TO_NAMES_DICT = dict((getattr(signal, n), n) \ for n in dir(signal) if n.startswith('SIG') and '_' not in n ) #exit correctly def tesh_exit(errcode): #If you do not flush some prints are skipped sys.stdout.flush() #os._exit exit even when executed within a thread os._exit(errcode) def fatal_error(msg): print("[Tesh/CRITICAL] "+str(msg)) tesh_exit(1) #Set an environment variable. # arg must be a string with the format "variable=value" def setenv(arg): print("[Tesh/INFO] setenv "+arg) t = arg.split("=") os.environ[t[0]] = t[1] #os.putenv(t[0], t[1]) does not work #see http://stackoverflow.com/questions/17705419/python-os-environ-os-putenv-usr-bin-env #http://stackoverflow.com/questions/30734967/how-to-expand-environment-variables-in-python-as-bash-does def expandvars2(path): return re.sub(r'(? 1 and line[-2] == "\\": txt = txt[0:-1] line = next(self.f) self.linenumber += 1 txt += line[0:-1] return txt #keep the state of tesh (mostly configuration values) class TeshState(Singleton): def __init__(self): self.threads = [] self.args_suffix = "" self.ignore_regexps_common = [] self.jenkins = False # not a Jenkins run by default self.timeout = 10 # default value: 10 sec self.wrapper = None self.keep = False def add_thread(self, thread): self.threads.append(thread) def join_all_threads(self): for t in self.threads: t.acquire() t.release() #Command line object class Cmd(object): def __init__(self): self.input_pipe = [] self.output_pipe_stdout = [] self.output_pipe_stderr = [] self.timeout = TeshState().timeout self.args = None self.linenumber = -1 self.background = False self.cwd = None self.ignore_output = False self.expect_return = 0 self.output_display = False self.sort = -1 self.ignore_regexps = TeshState().ignore_regexps_common def add_input_pipe(self, l): self.input_pipe.append(l) def add_output_pipe_stdout(self, l): self.output_pipe_stdout.append(l) def add_output_pipe_stderr(self, l): self.output_pipe_stderr.append(l) def set_cmd(self, args, linenumber): self.args = args self.linenumber = linenumber def add_ignore(self, txt): self.ignore_regexps.append(re.compile(txt)) def remove_ignored_lines(self, lines): for ign in self.ignore_regexps: lines = [l for l in lines if not ign.match(l)] return lines def _cmd_mkfile(self, argline): filename = argline[len("mkfile "):] file = open(filename, "w") if file is None: fatal_error("Unable to create file "+filename) file.write("\n".join(self.input_pipe)) file.write("\n") file.close() def _cmd_cd(self, argline): args = shlex.split(argline) if len(args) != 2: fatal_error("Too many arguments to cd") try: os.chdir(args[1]) print("[Tesh/INFO] change directory to "+args[1]) except FileNotFoundError: print("Chdir to "+args[1]+" failed: No such file or directory") print("Test suite `"+FileReader().filename+"': NOK (system error)") tesh_exit(4) #Run the Cmd if possible. # Return False if nothing has been ran. def run_if_possible(self): if self.can_run(): if self.background: #Python threads loose the cwd self.cwd = os.getcwd() lock = _thread.allocate_lock() lock.acquire() TeshState().add_thread(lock) _thread.start_new_thread( Cmd._run, (self, lock) ) else: self._run() return True else: return False def _run(self, lock=None): #Python threads loose the cwd if self.cwd is not None: os.chdir(self.cwd) self.cwd = None #retrocompatibility: support ${aaa:=.} variable format def replace_perl_variables(m): vname = m.group(1) vdefault = m.group(2) if vname in os.environ: return "$"+vname else: return vdefault self.args = re.sub(r"\${(\w+):=([^}]*)}", replace_perl_variables, self.args) #replace bash environment variables ($THINGS) to their values self.args = expandvars2(self.args) if re.match("^mkfile ", self.args) is not None: self._cmd_mkfile(self.args) if lock is not None: lock.release() return if re.match("^cd ", self.args) is not None: self._cmd_cd(self.args) if lock is not None: lock.release() return if TeshState().wrapper is not None: self.timeout *= 20 self.args = TeshState().wrapper + self.args elif re.match(".*smpirun.*", self.args) is not None: self.args = "sh " + self.args if TeshState().jenkins and self.timeout != None: self.timeout *= 10 self.args += TeshState().args_suffix print("["+FileReader().filename+":"+str(self.linenumber)+"] "+self.args) args = shlex.split(self.args) #print (args) try: proc = subprocess.Popen(args, bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) except FileNotFoundError: print("["+FileReader().filename+":"+str(self.linenumber)+"] Cannot start '"+args[0]+"': File not found") tesh_exit(3) except OSError as osE: if osE.errno == 8: osE.strerror += "\nOSError: [Errno 8] Executed scripts should start with shebang line (like #!/bin/sh)" raise osE cmdName = FileReader().filename+":"+str(self.linenumber) try: (stdout_data, stderr_data) = proc.communicate("\n".join(self.input_pipe), self.timeout) except subprocess.TimeoutExpired: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> timeout after "+str(self.timeout)+" sec)") proc.kill() tesh_exit(3) if self.output_display: print(stdout_data) #remove text colors ansi_escape = re.compile(r'\x1b[^m]*m') stdout_data = ansi_escape.sub('', stdout_data) #print ((stdout_data, stderr_data)) if self.ignore_output: print("(ignoring the output of <"+cmdName+"> as requested)") else: stdouta = stdout_data.split("\n") while len(stdouta) > 0 and stdouta[-1] == "": del stdouta[-1] stdouta = self.remove_ignored_lines(stdouta) stdcpy = stdouta[:] # Mimic the "sort" bash command, which is case unsensitive. if self.sort == 0: stdouta.sort(key=lambda x: x.lower()) self.output_pipe_stdout.sort(key=lambda x: x.lower()) elif self.sort > 0: stdouta.sort(key=lambda x: x[:self.sort].lower()) self.output_pipe_stdout.sort(key=lambda x: x[:self.sort].lower()) diff = list(difflib.unified_diff(self.output_pipe_stdout, stdouta,lineterm="",fromfile='expected', tofile='obtained')) if len(diff) > 0: print("Output of <"+cmdName+"> mismatch:") if self.sort >= 0: # If sorted, truncate the diff output and show the unsorted version difflen = 0; for line in diff: if difflen<50: print(line) difflen += 1 if difflen > 50: print("(diff truncated after 50 lines)") print("Unsorted observed output:\n") for line in stdcpy: print(line) else: # If not sorted, just display the diff for line in diff: print(line) print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> output mismatch)") if lock is not None: lock.release() if TeshState().keep: f = open('obtained','w') obtained = stdout_data.split("\n") while len(obtained) > 0 and obtained[-1] == "": del obtained[-1] obtained = self.remove_ignored_lines(obtained) for line in obtained: f.write("> "+line+"\n") f.close() print("Obtained output kept as requested: "+os.path.abspath("obtained")) tesh_exit(2) #print ((proc.returncode, self.expect_return)) if proc.returncode != self.expect_return: if proc.returncode >= 0: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> returned code "+str(proc.returncode)+")") if lock is not None: lock.release() tesh_exit(2) else: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> got signal "+SIGNALS_TO_NAMES_DICT[-proc.returncode]+")") if lock is not None: lock.release() tesh_exit(-proc.returncode) if lock is not None: lock.release() def can_run(self): return self.args is not None ############## # # Main # # if __name__ == '__main__': parser = argparse.ArgumentParser(description='tesh -- testing shell', add_help=True) group1 = parser.add_argument_group('Options') group1.add_argument('teshfile', nargs='?', help='Name of teshfile, stdin if omitted') group1.add_argument('--cd', metavar='some/directory', help='ask tesh to switch the working directory before launching the tests') group1.add_argument('--setenv', metavar='var=value', action='append', help='set a specific environment variable') group1.add_argument('--cfg', metavar='arg', action='append', help='add parameter --cfg=arg to each command line') group1.add_argument('--log', metavar='arg', action='append', help='add parameter --log=arg to each command line') group1.add_argument('--ignore-jenkins', action='store_true', help='ignore all cruft generated on SimGrid continous integration servers') group1.add_argument('--wrapper', metavar='arg', help='Run each command in the provided wrapper (eg valgrind)') group1.add_argument('--keep', action='store_true', help='Keep the obtained output when it does not match the expected one') try: options = parser.parse_args() except SystemExit: tesh_exit(1) if options.cd is not None: print("[Tesh/INFO] change directory to " + options.cd) os.chdir(options.cd) if options.ignore_jenkins: print("Ignore all cruft seen on SimGrid's continous integration servers") # Note: regexps should match at the beginning of lines TeshState().ignore_regexps_common = [ re.compile("profiling:"), re.compile("Unable to clean temporary file C:"), re.compile(".*Configuration change: Set \'contexts/"), re.compile("Picked up JAVA_TOOL_OPTIONS: "), re.compile("Picked up _JAVA_OPTIONS: "), re.compile("==[0-9]+== ?WARNING: ASan doesn\'t fully support"), re.compile("==[0-9]+== ?WARNING: ASan is ignoring requested __asan_handle_no_return: stack top:"), re.compile("False positive error reports may follow"), re.compile("For details see http://code.google.com/p/address-sanitizer/issues/detail\\?id=189"), re.compile("For details see https://github.com/google/sanitizers/issues/189"), re.compile("Python runtime initialized with LC_CTYPE=C .*"), re.compile("cmake: /usr/local/lib/libcurl.so.4: no version information available (required by cmake)"), # Seen on CircleCI ] TeshState().jenkins = True # This is a Jenkins build if options.teshfile is None: f = FileReader(None) print("Test suite from stdin") else: if not os.path.isfile(options.teshfile): print("Cannot open teshfile '"+options.teshfile+"': File not found") tesh_exit(3) f = FileReader(options.teshfile) print("Test suite '"+f.abspath+"'") if options.setenv is not None: for e in options.setenv: setenv(e) if options.cfg is not None: for c in options.cfg: TeshState().args_suffix += " --cfg=" + c if options.log is not None: for l in options.log: TeshState().args_suffix += " --log=" + l if options.wrapper is not None: TeshState().wrapper = options.wrapper if options.keep: TeshState().keep = True #cmd holds the current command line # tech commands will add some parameters to it # when ready, we execute it. cmd = Cmd() line = f.readfullline() while line is not None: #print(">>============="+line+"==<<") if len(line) == 0: #print ("END CMD block") if cmd.run_if_possible(): cmd = Cmd() elif line[0] == "#": pass elif line[0:2] == "p ": print("["+str(FileReader())+"] "+line[2:]) elif line[0:2] == "< ": cmd.add_input_pipe(line[2:]) elif line[0:1] == "<": cmd.add_input_pipe(line[1:]) elif line[0:2] == "> ": cmd.add_output_pipe_stdout(line[2:]) elif line[0:1] == ">": cmd.add_output_pipe_stdout(line[1:]) elif line[0:2] == "$ ": if cmd.run_if_possible(): cmd = Cmd() cmd.set_cmd(line[2:], f.linenumber) elif line[0:2] == "& ": if cmd.run_if_possible(): cmd = Cmd() cmd.set_cmd(line[2:], f.linenumber) cmd.background = True elif line[0:15] == "! output ignore": cmd.ignore_output = True #print("cmd.ignore_output = True") elif line[0:16] == "! output display": cmd.output_display = True cmd.ignore_output = True elif line[0:15] == "! expect return": cmd.expect_return = int(line[16:]) #print("expect return "+str(int(line[16:]))) elif line[0:15] == "! expect signal": sig = line[16:] #get the signal integer value from the signal module if sig not in signal.__dict__: fatal_error("unrecognized signal '"+sig+"'") sig = int(signal.__dict__[sig]) #popen return -signal when a process ends with a signal cmd.expect_return = -sig elif line[0:len("! timeout ")] == "! timeout ": if "no" in line[len("! timeout "):]: cmd.timeout = None else: cmd.timeout = int(line[len("! timeout "):]) elif line[0:len("! output sort")] == "! output sort": if len(line) >= len("! output sort "): sort = int(line[len("! output sort "):]) else: sort = 0 cmd.sort = sort elif line[0:len("! setenv ")] == "! setenv ": setenv(line[len("! setenv "):]) elif line[0:len("! ignore ")] == "! ignore ": cmd.add_ignore(line[len("! ignore "):]) else: fatal_error("UNRECOGNIZED OPTION") line = f.readfullline() cmd.run_if_possible() TeshState().join_all_threads() if f.filename == "(stdin)": print("Test suite from stdin OK") else: print("Test suite `"+f.filename+"' OK") SimGrid-3.18/tools/tesh/background.tesh0000644000175000017500000000054713217757337020443 0ustar mquinsonmquinson! output ignore $ cmake -E remove bgtesh.pl bgtesh.data < use strict; < sleep(1); < open (FILE, ") { < print; < } < close(FILE); < exit 0; < $ mkfile bgtesh.pl & perl bgtesh.pl > TOTO < TOTO $ mkfile bgtesh.data $ sleep 2 ! output ignore $ cmake -E remove bgtesh.pl bgtesh.data SimGrid-3.18/tools/tesh/generate_tesh0000755000175000017500000000037613217757325020177 0ustar mquinsonmquinson#!/bin/sh # Generate a tesh file from a given command (simple case). command="$1" output=$(sh -c "$command" 2>&1) return=$? tesh="#! ./tesh ! expect return $return $ $command $(echo "$output" | sed 's/^/> /' ) " echo "$tesh" echo "$tesh" | tesh 1>&2 SimGrid-3.18/tools/tesh/catch-timeout.tesh0000644000175000017500000000043513217757337021066 0ustar mquinsonmquinson#! ./tesh # This suite must be functional because we changed the timeout value to 10 # before sleeping 6 secs. ! expect return 3 < ! timeout 1 < $ sleep 6 > Test suite from stdin > [(stdin):2] sleep 6 > Test suite `(stdin)': NOK (<(stdin):2> timeout after 1 sec) $ ${bindir:=.}/tesh SimGrid-3.18/tools/tesh/set-output-ignore.tesh0000644000175000017500000000040013217757337021722 0ustar mquinsonmquinson#! ./tesh p This tests whether TESH accepts to ignore command output < ! output ignore < > TOTO < < TUTU < $ cat $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):4] cat > (ignoring the output of <(stdin):4> as requested) > Test suite from stdin OK SimGrid-3.18/tools/tesh/bg-basic.tesh0000644000175000017500000000021613217757337017764 0ustar mquinsonmquinson#! ./tesh p This is a basic test < TOTO \ TUTU & cat > TOTO TUTU p And now, some multilines examples < a < b < c < d & cat > a > b > c > dSimGrid-3.18/tools/simgrid2vite.sed0000644000175000017500000000025513217757343017572 0ustar mquinsonmquinson/0 1 0 MPI/i \0 ROOT 0 ROOT s/0 1 0 MPI/0 1 ROOT MPI/ s/4 3 0 1 1 MPI_LINK/4 3 ROOT 1 1 MPI_LINK/ s/1 0 \"/1 root \"/g /6 0 1 1/i \6 0 root ROOT 0 "root" s/0 PTP/root PTP/g SimGrid-3.18/tools/lualib.patch0000644000175000017500000000343413217757322016755 0ustar mquinsonmquinsonThis patch is to be applied to the Lua 5.3 source file to get a shared library. This is because the authors of Lua don't bother distributing a working build system for their software, so we have to make one... As unfortunate as it may be, there is nothing else we can do. -- Da SimGrid team. diff --git a/Makefile b/Makefile index 5ee5601..93830a3 100644 --- a/Makefile +++ b/Makefile @@ -52,7 +52,7 @@ R= $V.1 all: $(PLAT) $(PLATS) clean: - cd src && $(MAKE) $@ + cd src && $(MAKE) $@ V=$(V) R=$(R) test: dummy src/lua -v diff --git a/src/Makefile b/src/Makefile index d71c75c..64bda79 100644 --- a/src/Makefile +++ b/src/Makefile @@ -7,7 +7,7 @@ PLAT= none CC= gcc -std=gnu99 -CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS) +CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS) -fPIC LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS) LIBS= -lm $(SYSLIBS) $(MYLIBS) @@ -29,6 +29,7 @@ MYOBJS= PLATS= aix bsd c89 freebsd generic linux macosx mingw posix solaris LUA_A= liblua.a +LUA_SO= liblua.so CORE_O= lapi.o lcode.o lctype.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o \ lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o \ ltm.o lundump.o lvm.o lzio.o @@ -43,7 +44,7 @@ LUAC_T= luac LUAC_O= luac.o ALL_O= $(BASE_O) $(LUA_O) $(LUAC_O) -ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T) +ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T) $(LUA_SO) ALL_A= $(LUA_A) # Targets start here. @@ -59,6 +60,11 @@ $(LUA_A): $(BASE_O) $(AR) $@ $(BASE_O) $(RANLIB) $@ +$(LUA_SO): $(CORE_O) $(LIB_O) + $(CC) -shared -ldl -Wl,-soname,$(LUA_SO).$(V) -o $@.$(R) $? -lm $(MYLDFLAGS) + ln -sf $(LUA_SO).$(R) $(LUA_SO).$(V) + ln -sf $(LUA_SO).$(R) $(LUA_SO) + $(LUA_T): $(LUA_O) $(LUA_A) $(CC) -o $@ $(LDFLAGS) $(LUA_O) $(LUA_A) $(LIBS) SimGrid-3.18/tools/generate-dwarf-functions0000755000175000017500000000330313217757325021311 0ustar mquinsonmquinson#!/bin/sh # Generate files from a given dwarf.h # Usage: tools/generate-dwarf-functions /usr/include/dwarf.h HEADER="\ /* Copyright (c) 2014-$(date +%Y). The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include #include \"src/mc/mc_dwarf.hpp\"" cat - > src/mc/mc_dwarf_tagnames.cpp < tagname_map = { {0x00, "DW_TAG_invalid"}, $(cat "$1" | grep DW_TAG_ | sed 's/.*\(DW_TAG_[^ ]*\) = \(0x[0-9a-f]*\).*/ {\2, "\1"},/') }; } namespace simgrid { namespace dwarf { /** \brief Get the name of a dwarf tag (DW_TAG_*) from its code * * \param tag tag code (see the DWARF specification) * \return name of the tag */ XBT_PRIVATE const char *tagname(int tag) { auto name = tagname_map.find(tag); return name == tagname_map.end() ? "DW_TAG_unknown" : name->second; } } } EOF cat - > src/mc/mc_dwarf_attrnames.cpp << EOF $HEADER namespace { const std::unordered_map attrname_map = { $(cat "$1" | grep DW_AT_ | sed 's/.*\(DW_AT_[^ ]*\) = \(0x[0-9a-f]*\).*/ {\2, "\1"},/') }; } namespace simgrid { namespace dwarf { /** \brief Get the name of an attribute (DW_AT_*) from its code * * \param attr attribute code (see the DWARF specification) * \return name of the attribute */ XBT_PRIVATE const char *attrname(int attr) { auto name = attrname_map.find(attr); return name == attrname_map.end() ? "DW_AT_unknown" : name->second; } } } EOF SimGrid-3.18/tools/graphicator/0000755000175000017500000000000013217757340016763 5ustar mquinsonmquinsonSimGrid-3.18/tools/graphicator/CMakeLists.txt0000644000175000017500000000141113217757323021521 0ustar mquinsonmquinsonadd_executable (graphicator graphicator.c) target_link_libraries(graphicator simgrid) set_target_properties(graphicator PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) ADD_TESH(graphicator --setenv srcdir=${CMAKE_HOME_DIRECTORY} --setenv bindir=${CMAKE_BINARY_DIR}/bin --cd ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/graphicator.tesh) ## Clean generated files get_directory_property(extra_clean_files ADDITIONAL_MAKE_CLEAN_FILES) set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${extra_clean_files};${CMAKE_CURRENT_BINARY_DIR}/simgrid.trace;") set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/graphicator.tesh PARENT_SCOPE) set(tools_src ${tools_src} ${CMAKE_CURRENT_SOURCE_DIR}/graphicator.c PARENT_SCOPE) SimGrid-3.18/tools/graphicator/graphicator.c0000644000175000017500000000143013217757340021430 0ustar mquinsonmquinson/* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/msg.h" #include "xbt/graph.h" XBT_LOG_NEW_DEFAULT_CATEGORY(graphicator, "Graphicator Logging System"); int main(int argc, char **argv) { XBT_LOG_CONNECT(graphicator); MSG_init(&argc, argv); xbt_assert(argc == 3, "Usage: %s ", argv[0]); MSG_create_environment(argv[1]); int status = TRACE_platform_graph_export_graphviz (argv[2]); xbt_assert(status != 0, "%s expects --cfg=tracing:yes --cfg=tracing/platform:yes", argv[0]); return 0; } SimGrid-3.18/tools/graphicator/graphicator.tesh0000644000175000017500000000047613217757337022170 0ustar mquinsonmquinson#! ./tesh $ ${bindir:=.}/graphicator ${srcdir:=.}/teshsuite/simdag/platforms/one_cluster.xml --cfg=tracing:yes --cfg=tracing/platform:yes test.dot > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/platform' to 'yes' $ rm test.dot SimGrid-3.18/tools/MSG_visualization/0000755000175000017500000000000013217757325020072 5ustar mquinsonmquinsonSimGrid-3.18/tools/MSG_visualization/colorize.pl0000755000175000017500000000623113217757325022262 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright (c) 2005, 2007, 2010, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. $col_white = "\033[00m"; $col_black = "\033[30m"; $col_red = "\033[31m"; $col_green = "\033[32m"; $col_yellow = "\033[33m"; $col_blue = "\033[34m"; $col_purple = "\033[35m"; $col_cyan = "\033[36m"; $col_ltgray = "\033[37m"; $col_darkgray = "\033[30m"; $col_norm = $col_white; $col_background = "\033[07m"; $col_brighten = "\033[01m"; $col_underline = "\033[04m"; $col_blink = "\033[05m"; # Customize colors here... $col_default = $col_ltgray; my (@coltab) = ( $col_green, $col_yellow, $col_purple, $col_cyan, $col_red, $col_blue, $col_background . $col_green, $col_background . $col_yellow, $col_background . $col_purple, $col_background . $col_cyan, $col_background . $col_red, $col_background . $col_blue, $col_background . $col_magenta, ); my %pid; # Get options while (($_ = $ARGV[0]) =~ /^-/) { shift; if (/-location/i) { $opt_print_location = 1; shift; } elsif (/-h(elp)?$|-u(sage)?$/i) { print< where is a text file of values or '-' for STDIN Options: () denote short version -location Print the location in the code of the message. EOH ; exit; } } sub pidcolor { my $pid = shift; unless (defined($pid{$pid})) { # first time we see this pid. Affect it a color $pid{$pid}=(scalar keys %pid) % (scalar @coltab); } return $coltab[$pid{$pid}]; } sub print_line { my($host,$procname,$pid,$date,$location,$xbt_channel,$message)=@_; print $col_norm; printf "[% 10.6f]",$date; print pidcolor($pid); if(defined($location)) { printf "[%10s:%-10s] %s ",$host,$procname,$location; } else { printf "[%10s:%-10s]",$host,$procname; } print " $message"; print $col_norm."\n"; } # Read the messages and do the job while (<>) { $orgline = $thisline = $_; # Typical line [Gatien:slave:(9) 11.243148] msg/gos.c:137: [msg_gos/DEBUG] Action terminated if ($thisline =~ /^\[(.+):([^:]+):\((\d+)\) ([\d\.]*)\] ([^\[]*) \[([^\[]*)\] (.*)$/) { $host=$1; $procname=$2; $pid=$3; $date=$4; if($opt_print_location) { $location=$5; $location =~ s/:$//; } else { $location = undef; } $xbt_channel=$6; $message=$7; print_line($host,$procname,$pid,$date,$location,$xbt_channel,$message); # Typical line [Boivin:slave:(2) 9.269357] [pmm/INFO] ROW: step(2)<>myrow(0). Receive data from TeX } elsif ($thisline =~ /^\[([^:]+):([^:]+):\((\d+)\) ([\d\.]*)\] \[([^\[]*)\] (.*)$/) { $host=$1; $procname=$2; $pid=$3; $date=$4; $xbt_channel=$5; $message=$6; print_line($host,$procname,$pid,$date,undef,$xbt_channel,$message); } elsif ( $thisline =~ /^==(\d+)== (.*)$/) { # take care of valgrind outputs print pidcolor($1)."$2\n"; } else { print $col_default. $orgline; } } print $col_norm; SimGrid-3.18/tools/MSG_visualization/trace2fig.pl0000755000175000017500000002641713217757325022312 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright (c) 2006-2007, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; #use Data::Dumper; use XFig; use POSIX; my($grid_Y_size)=225; my($grid_X_size)=100550; # Can be changed to improve readability in function of the total execution time my($color_suspended)=1; my($color_compute)=2; my($color_wait_for_recpt)=3; my($color_communicate)=4; # Determine the order of the colors in the legend my(@color_list)=($color_compute,$color_communicate,$color_wait_for_recpt,$color_suspended); sub read_cat { my(%Cat); my($filename)=@_; my($line); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^7\s/) { my($event,$date,$id,$type,$father,@name) = split(/\s+/,$line); $Cat{$id}{name}="@name "; $Cat{$id}{name}=~s/\"//g; $Cat{$id}{father}=$father; $Cat{$id}{type}=$type; $Cat{$id}{date}=$date; } } close INPUT; return \%Cat; } sub read_event { my($filename,$Cat)=@_; my($line); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^11\s/) { my($event,$date,$type,$id,$state) = split(/\s+/,$line); push @{$$Cat{$id}{state}}, [$date,$state]; } if($line =~ /^12\s/) { my($event,$date,$type,$id) = split(/\s+/,$line); push @{$$Cat{$id}{state}}, [$date]; } } close INPUT; } sub read_link { my($filename)=@_; my($line); my(%link); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^16\s/) { my($event,$date,$type,$father,$channel,$src,$key,$trash) = split(/\t+/,$line); my($numkey)=hex "$key"; while (defined($link{$numkey})) {$numkey++;} $link{$numkey}{src}=$src; $link{$numkey}{src_date}=$date; } if($line =~ /^17\s/) { my($event,$date,$type,$father,$channel,$dst,$key,$trash) = split(/\t+/,$line); my($numkey)=hex "$key"; while (defined($link{$numkey}{dst})) {$numkey++;} $link{$numkey}{dst}=$dst; $link{$numkey}{dst_date}=$date; } } close INPUT; return \%link; } sub build_cat_tree { my($root,$Cat)=@_; my(@childs)=(); my($cat); foreach $cat (keys %$Cat) { if($$Cat{$cat}{father} eq $root) { push @childs, build_cat_tree($cat,$Cat); } # print "$$Cat{$cat}{name}\t\t $Cat{$cat}{father}\n"; } return [$root,@childs]; } sub build_cat_list { my($tree,$cat_list)=@_; my($root) = shift @$tree; my($u); push @$cat_list,$root; foreach $u (@$tree) { build_cat_list($u,$cat_list); } unshift @$tree, $root; } sub cat_sorting_function { my($cat1,$cat2,$Cat)=@_; if (!defined($$Cat{$cat1}{state})) { if (!defined($$Cat{$cat2}{state})) { return 0; } else { return 1; } } if (!defined($$Cat{$cat2}{state})) { return -1; } my($comp) = $$Cat{$$Cat{$cat1}{'father'}}{'name'} cmp $$Cat{$$Cat{$cat2}{'father'}}{'name'}; if ($comp == 0) { return $$Cat{$cat1}{'name'} cmp $$Cat{$cat2}{'name'}; } else { return $comp; } } sub update_host_Y { my($host,$i) = @_; if (!defined($$host{'Y_min_host'})) { $$host{'Y_min_host'} = $i; } else { if ($$host{'Y_min_host'} > $i) { $$host{'Y_min_host'} = $i; } } if (!defined($$host{'Y_max_host'})) { $$host{'Y_max_host'} = $i+1; } else { if ($$host{'Y_max_host'} < $i+1) { $$host{'Y_max_host'} = $i+1; } } } sub set_cat_position { my($Cat,$cat_list)=@_; my($i)=0; my($cat); foreach $cat (sort {cat_sorting_function($a,$b,$Cat)} @$cat_list) { if(defined($$Cat{$cat}{state})) { update_host_Y($$Cat{$$Cat{$cat}{'father'}},$i); $$Cat{$cat}{Y_min} = $i; $$Cat{$cat}{Y_max} = $i+1; $i++; } } } sub create_fig { my($filename)=shift; my($fig)=new XFig; $fig->{object} = 'compound'; # Compound $fig->{elements} = []; $fig->{version} = 3.2; $fig->{orientation} = 'Landscape'; $fig->{justification} = 'Center'; $fig->{units} = 'Metric'; $fig->{papersize} = 'A4'; $fig->{magnification} = '100.00'; $fig->{multiplepage} = 'Single'; $fig->{transparent} = '-2'; $fig->{resolution} = '1200'; $fig->{coordsystem} = '2'; $fig->{filename} = $filename; return $fig; } sub draw_cat { my($fig,$Cat,$Link)=@_; my($cat,$e,$link); my($max_string_length)=0; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min}) && defined($$Cat{$cat}{Y_max})); my($text) = new XFig ('text'); # $text->{'text'} = "$$Cat{$$Cat{$cat}{father}}{name}"."$$Cat{$cat}{name}"; my($printed_name)= $$Cat{$cat}{name}; $printed_name =~ s/\d+ \(0\)\s*$//; if (length($printed_name) > $max_string_length) { $max_string_length = length($printed_name); } $text->{'text'} = "$printed_name"; $text->{'y'} = ($$Cat{$cat}{Y_min}+$$Cat{$cat}{Y_max})/2*$grid_Y_size+68; $text->{'x'} = -100; $text->{'subtype'} = 2; $fig->add ($text); } my($max_date)=0; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min}) && defined($$Cat{$cat}{Y_max})); my(@states)=(); my($e); foreach $e (@{$$Cat{$cat}{state}}) { my($new_date,$state) = ($$e[0],$$e[1]); if ($new_date > $max_date) { $max_date = $new_date; } if(defined($state)) { push @states, $e; } else { my($old_event) = pop @states; my($old_date) = $$old_event[0]; $state = $$old_event[1]; # LM: I added the next line because of "undefined values"... # normally, I think that this should not happen, but this part of code is a bit too cryptic to me next unless (defined($state)); my($line) = new XFig ('polyline'); $line->{'depth'} = 50; # line $line->{'subtype'} = 1; # line $line->{'points'} = [ [$old_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size], [$new_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size], [$new_date*$grid_X_size, $$Cat{$cat}{Y_max}*$grid_Y_size], [$old_date*$grid_X_size, $$Cat{$cat}{Y_max}*$grid_Y_size], [$old_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size] ]; $line->{'areafill'} = 20; if($state eq "S") { $line->{'fillcolor'} = $color_suspended; } elsif ($state eq "E") { $line->{'fillcolor'} = $color_compute; } elsif ($state eq "B") { $line->{'fillcolor'} = $color_wait_for_recpt; } elsif ($state eq "C") { $line->{'fillcolor'} = $color_communicate; } $fig->add ($line); } } } foreach $link (keys %$Link) { my($line) = new XFig ('polyline'); my($src_date)=$$Link{$link}{src_date}; my($src)=$$Link{$link}{src}; my($dst_date)=$$Link{$link}{dst_date}; my($dst)=$$Link{$link}{dst}; $line->{'subtype'} = 1; # line print STDERR "$link: $src ($src_date) -> $dst ($dst_date)\n"; print STDERR "$$Cat{$src}{name} -> $$Cat{$dst}{name}\n"; $line->{'points'} = [ [$src_date*$grid_X_size, ($$Cat{$src}{Y_min}+$$Cat{$src}{Y_max})/2*$grid_Y_size], [$dst_date*$grid_X_size, ($$Cat{$dst}{Y_min}+$$Cat{$dst}{Y_max})/2*$grid_Y_size] ]; $line->{'forwardarrow'} = ['1', '1', '1.00', '60.00', '120.00']; $fig->add ($line); } # Host visualization my($max_Y)= 0; my($index_fill)=0; my($width_of_one_letter)=80; my($min_x_for_host)=-400 - $max_string_length*$width_of_one_letter; my($host_text_x)= $min_x_for_host + 200; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min_host}) && defined($$Cat{$cat}{Y_max_host})); my($line) = new XFig ('polyline'); $line->{'depth'} = 150; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$min_x_for_host, $$Cat{$cat}{Y_min_host}*$grid_Y_size], [$max_date*$grid_X_size+150, $$Cat{$cat}{Y_min_host}*$grid_Y_size], [$max_date*$grid_X_size+150, $$Cat{$cat}{Y_max_host}*$grid_Y_size], [$min_x_for_host, $$Cat{$cat}{Y_max_host}*$grid_Y_size] ]; $line->{'areafill'} = 4+3*($index_fill % 2); $line->{'fillcolor'} = 0; $line->{'thickness'} = 0; $index_fill++; $fig->add ($line); my($text) = new XFig ('text'); $text->{'text'} = "$$Cat{$cat}{name}"; $text->{'angle'} = 3.14159265/2; $text->{'x'} = $host_text_x; $text->{'y'} = ($$Cat{$cat}{Y_min_host}+$$Cat{$cat}{Y_max_host})/2*$grid_Y_size; $text->{'subtype'} = 1; $text->{'font_size'} = 30; $fig->add ($text); if ($max_Y<$$Cat{$cat}{Y_max_host}) { $max_Y = $$Cat{$cat}{Y_max_host}; } } # Legend: my($i)=1; my($color); foreach $color (@color_list) { my($min_x)=0; my($min_Y)=($max_Y+1)*$grid_Y_size; my($width)=1700; my($height)=$grid_Y_size; my($line) = new XFig ('polyline'); $line->{'depth'} = 50; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$min_x,$min_Y + ($i-1)*$height ], [$min_x + $width,$min_Y + ($i-1)*$height], [$min_x + $width,$min_Y+$height + ($i-1)*$height], [$min_x,$min_Y+$height + ($i-1)*$height], [$min_x,$min_Y+ ($i-1)*$height]]; $line->{'areafill'} = 20; $line->{'fillcolor'} = $color; $fig->add ($line); my($text) = new XFig ('text'); if ($color==$color_suspended) { $text->{'text'} = "Suspended"; } if ($color==$color_compute) { $text->{'text'} = "Computing"; } if ($color==$color_wait_for_recpt) { $text->{'text'} = "Waiting for reception"; } if ($color==$color_communicate) { $text->{'text'} = "Communicating"; } $text->{'y'} = $min_Y + ($i-0.5)*$height +68; $text->{'x'} = 50; $text->{'subtype'} = 0; $fig->add ($text); $i++; } # Time axis my($line) = new XFig ('polyline'); $line->{'depth'} = 0; $line->{'subtype'} = 1; # line $line->{'points'} = [ [0,0],[$max_date * $grid_X_size+150,0] ]; $line->{'forwardarrow'} = ['1', '1', '1.00', '60.00', '120.00']; $fig->add ($line); my($digits)=POSIX::floor(log($max_date)/log(10)); my($exponent) = 10**$digits; my($mantissa)= $max_date / $exponent; my($incr); if ($mantissa<2) { $incr = $exponent/10; } elsif ($mantissa<5) { $incr = $exponent/4; } else { $incr = $exponent/2; } print "$max_date $digits $exponent $mantissa $incr\n"; my($x); for($x=0; $x < $max_date; $x += $incr) { print "$x\n"; $line = new XFig ('polyline'); $line->{'depth'} = 0; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$x * $grid_X_size,0],[$x * $grid_X_size, -100] ]; $line->{'forwardarrow'} = 0; $fig->add ($line); my($text) = new XFig ('text'); $text->{'text'} = "$x"; $text->{'y'} = -200; $text->{'x'} = $x * $grid_X_size; $text->{'subtype'} = 1; $fig->add ($text); } # Empty line so that the text of the time axis can be seen on the pdf $line = new XFig ('polyline'); $line->{'depth'} = 999; $line->{'subtype'} = 1; # line $line->{'thickness'} = 0; $line->{'points'} = [ [0,0],[0, -400] ]; $fig->add ($line); } sub main { my($Cat) = read_cat($ARGV[0]); my($cat_tree)=build_cat_tree("0",$Cat); read_event($ARGV[0],$Cat); my($Link)=read_link($ARGV[0]); # print Dumper($cat_tree); # print Dumper($Cat); my($cat_list)=[]; build_cat_list($cat_tree,$cat_list); shift @$cat_list; shift @$cat_list; # print "@$cat_list \n"; set_cat_position($Cat,$cat_list); my($fig)=create_fig("toto.fig"); draw_cat($fig,$Cat,$Link); $fig->writefile (); system "fig2dev -L pdf toto.fig toto.pdf"; } main; SimGrid-3.18/tools/cmake/0000755000175000017500000000000013217757322015540 5ustar mquinsonmquinsonSimGrid-3.18/tools/cmake/MakeLibWin.cmake0000644000175000017500000000126713217757322020532 0ustar mquinsonmquinson### Make Libs add_library(simgrid SHARED ${simgrid_sources}) if(MSVC) set_target_properties(simgrid PROPERTIES COMPILE_FLAGS "/DDLL_EXPORT" VERSION ${libsimgrid_version} ) else() set_target_properties(simgrid PROPERTIES COMPILE_FLAGS "-DDLL_EXPORT" LINK_FLAGS "-shared" VERSION ${libsimgrid_version} PREFIX "lib" SUFFIX ".dll" IMPORT_PREFIX "lib" IMPORT_SUFFIX ".dll") set(SIMGRID_DEP "-lm") if (HAVE_PTHREAD) set(SIMGRID_DEP "${SIMGRID_DEP} -lpthread") endif() if (HAVE_BOOST_CONTEXTS) set(SIMGRID_DEP "${SIMGRID_DEP} ${Boost_CONTEXT_LIBRARY}") endif() endif() target_link_libraries(simgrid ${SIMGRID_DEP}) SimGrid-3.18/tools/cmake/Tests.cmake0000644000175000017500000001336413217757322017653 0ustar mquinsonmquinsonIF(enable_smpi AND NOT WIN32) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpicc) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpicxx) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpiff) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpif90) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpirun) ENDIF() SET(TESH_OPTION "--ignore-jenkins") SET(TESH_COMMAND "${PYTHON_EXECUTABLE}" ${CMAKE_BINARY_DIR}/bin/tesh) IF(enable_memcheck) INCLUDE(FindValgrind) ENDIF() #some tests may take forever on non futexes systems, using busy_wait with n cores < n workers # default to posix for these tests if futexes are not supported IF(NOT HAVE_FUTEX_H) SET(CONTEXTS_SYNCHRO --cfg contexts/synchro:posix) ENDIF() MACRO(ADD_TESH NAME) SET(ARGT ${ARGV}) LIST(REMOVE_AT ARGT 0) IF(WIN32) STRING(REPLACE "§" "\;" ARGT "${ARGT}") ENDIF() if(TESH_WRAPPER) ADD_TEST(${NAME} ${TESH_COMMAND} --wrapper "${TESH_WRAPPER}" ${TESH_OPTION} ${ARGT}) else() ADD_TEST(${NAME} ${TESH_COMMAND} ${TESH_OPTION} ${ARGT}) endif() ENDMACRO() MACRO(ADD_TESH_FACTORIES NAME FACTORIES) SET(ARGR ${ARGV}) LIST(REMOVE_AT ARGR 0) # remove name FOREACH(I ${FACTORIES}) # remove all factories LIST(REMOVE_AT ARGR 0) ENDFOREACH() FOREACH(FACTORY ${FACTORIES}) if ((${FACTORY} STREQUAL "thread" AND HAVE_THREAD_CONTEXTS) OR (${FACTORY} STREQUAL "boost" AND HAVE_BOOST_CONTEXTS) OR (${FACTORY} STREQUAL "raw" AND HAVE_RAW_CONTEXTS) OR (${FACTORY} STREQUAL "ucontext" AND HAVE_UCONTEXT_CONTEXTS)) ADD_TESH("${NAME}-${FACTORY}" "--cfg" "contexts/factory:${FACTORY}" ${ARGR}) ENDIF() ENDFOREACH() ENDMACRO() IF(enable_java) IF(WIN32) SET(TESH_CLASSPATH "${CMAKE_BINARY_DIR}/examples/java/\;${CMAKE_BINARY_DIR}/teshsuite/java/\;${SIMGRID_JAR}") STRING(REPLACE "\;" "§" TESH_CLASSPATH "${TESH_CLASSPATH}") ELSE() SET(TESH_CLASSPATH "${CMAKE_BINARY_DIR}/examples/java/:${CMAKE_BINARY_DIR}/teshsuite/java/:${SIMGRID_JAR}") ENDIF() ENDIF() IF(SIMGRID_HAVE_MC) ADD_TESH_FACTORIES(mc-bugged1 "ucontext;raw" --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1.tesh) ADD_TESH_FACTORIES(mc-bugged2 "ucontext;raw" --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged2.tesh) IF(HAVE_UCONTEXT_CONTEXTS AND SIMGRID_PROCESSOR_x86_64) # liveness model-checking works only on 64bits (for now ...) ADD_TESH(mc-bugged1-liveness-ucontext --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness.tesh) ADD_TESH(mc-bugged1-liveness-ucontext-sparse --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_sparse.tesh) ADD_TESH(mc-bugged1-liveness-visited-ucontext --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_visited.tesh) ADD_TESH(mc-bugged1-liveness-visited-ucontext-sparse --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_visited_sparse.tesh) IF(HAVE_C_STACK_CLEANER) # This test checks if the stack cleaner is making a difference: ADD_TEST(mc-bugged1-liveness-stack-cleaner ${CMAKE_HOME_DIRECTORY}/examples/msg/mc/bugged1_liveness_stack_cleaner ${CMAKE_HOME_DIRECTORY}/examples/msg/mc/ ${CMAKE_BINARY_DIR}/examples/msg/mc/) ENDIF() ENDIF() ENDIF() IF(enable_smpi_MPICH3_testsuite AND SMPI_FORTRAN AND HAVE_THREAD_CONTEXTS) ADD_TEST(test-smpi-mpich3-thread-f77 ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/teshsuite/smpi/mpich3-test/f77/ ${PERL_EXECUTABLE} ${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/runtests ${TESH_OPTION} -mpiexec=${CMAKE_BINARY_DIR}/smpi_script/bin/smpirun -srcdir=${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/f77/ -tests=testlist -privatization=${HAVE_PRIVATIZATION} -execarg=--cfg=contexts/stack-size:8000 -execarg=--cfg=contexts/factory:thread -execarg=--cfg=smpi/privatization:${HAVE_PRIVATIZATION}) SET_TESTS_PROPERTIES(test-smpi-mpich3-thread-f77 PROPERTIES PASS_REGULAR_EXPRESSION "tests passed!") ADD_TEST(test-smpi-mpich3-thread-f90 ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/teshsuite/smpi/mpich3-test/f90/ ${PERL_EXECUTABLE} ${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/runtests ${TESH_OPTION} -mpiexec=${CMAKE_BINARY_DIR}/smpi_script/bin/smpirun -srcdir=${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/f90/ -tests=testlist -privatization=${HAVE_PRIVATIZATION} -execarg=--cfg=smpi/privatization:${HAVE_PRIVATIZATION} -execarg=--cfg=contexts/factory:thread) SET_TESTS_PROPERTIES(test-smpi-mpich3-thread-f90 PROPERTIES PASS_REGULAR_EXPRESSION "tests passed!") ENDIF() IF(SIMGRID_HAVE_LUA) # Tests testing simulation from C but using lua for platform files. Executed like this # ~$ ./masterslave platform.lua deploy.lua ADD_TESH(lua-platform-masterslave --setenv srcdir=${CMAKE_HOME_DIRECTORY} --setenv bindir=${CMAKE_BINARY_DIR} --cd ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/teshsuite/lua/lua_platforms.tesh) SET_TESTS_PROPERTIES(lua-platform-masterslave PROPERTIES ENVIRONMENT "LUA_CPATH=${CMAKE_BINARY_DIR}/lib/lib?.${LIB_EXE}") ENDIF() ADD_TEST(testall ${CMAKE_BINARY_DIR}/testall) # New tests should use the Boost Unit Test Framework if(Boost_UNIT_TEST_FRAMEWORK_FOUND) add_executable (unit_tmgr src/surf/trace_mgr_test.cpp) target_link_libraries(unit_tmgr simgrid ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}) ADD_TEST(unit_tmgr ${CMAKE_BINARY_DIR}/unit_tmgr --build_info=yes) else() set(EXTRA_DIST ${EXTRA_DIST} src/surf/trace_mgr_test.cpp) endif() SimGrid-3.18/tools/cmake/Distrib.cmake0000644000175000017500000003624513217757322020154 0ustar mquinsonmquinson######################################### ### Fill in the "make install" target ### ######################################### # doc file(MAKE_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc/html/) install(DIRECTORY "${CMAKE_HOME_DIRECTORY}/doc/html/" DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/doc/simgrid/html/) # binaries if(enable_smpi) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/smpicc ${CMAKE_BINARY_DIR}/bin/smpicxx ${CMAKE_BINARY_DIR}/bin/smpirun ${CMAKE_BINARY_DIR}/bin/smpimain DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) if(SMPI_FORTRAN) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/smpif90 ${CMAKE_BINARY_DIR}/bin/smpiff DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) endif() endif() if(enable_model-checking) install( PROGRAMS ${CMAKE_BINARY_DIR}/bin/simgrid-mc DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) endif() install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/tesh DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/graphicator DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) install(PROGRAMS ${CMAKE_HOME_DIRECTORY}/tools/MSG_visualization/colorize.pl DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/ RENAME simgrid-colorizer) add_custom_target(simgrid-colorizer ALL COMMENT "Install ${CMAKE_BINARY_DIR}/bin/colorize" COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/tools/MSG_visualization/colorize.pl ${CMAKE_BINARY_DIR}/bin/colorize) install(PROGRAMS ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/ RENAME simgrid_update_xml) add_custom_target(simgrid_update_xml ALL COMMENT "Install ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml" COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml) # libraries install(TARGETS simgrid DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/lib/) if(enable_java) set(SIMGRID_JAR_TO_INSTALL "${SIMGRID_JAR}") install(TARGETS simgrid-java DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/lib/) install(FILES ${SIMGRID_JAR_TO_INSTALL} DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/java/ RENAME simgrid.jar) endif() # include files set(HEADERS ${headers_to_install} ${generated_headers_to_install}) foreach(file ${HEADERS}) get_filename_component(location ${file} PATH) string(REPLACE "${CMAKE_CURRENT_BINARY_DIR}/" "" location "${location}") install(FILES ${file} DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/${location}) endforeach(file ${HEADERS}) # example files foreach(file ${examples_to_install}) string(REPLACE "${CMAKE_CURRENT_SOURCE_DIR}/examples/" "" file ${file}) get_filename_component(location ${file} PATH) install(FILES "examples/${file}" DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/doc/simgrid/examples/${location}) endforeach(file ${examples_to_install}) ########################################### ### Fill in the "make uninstall" target ### ########################################### add_custom_target(uninstall COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/doc/simgrid COMMAND ${CMAKE_COMMAND} -E echo "uninstall doc ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/libsimgrid* COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/lua/5.1/simgrid* COMMAND ${CMAKE_COMMAND} -E echo "uninstall lib ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpicc COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpicxx COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpiff COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpif90 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpirun COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/tesh COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/simgrid-colorizer COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/simgrid_update_xml COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/graphicator COMMAND ${CMAKE_COMMAND} -E echo "uninstall bin ok" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/instr COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/msg COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simdag COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/smpi COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simix COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/surf COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/xbt COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/mc COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simgrid COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/simgrid.h COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/simgrid_config.h COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/xbt.h COMMAND ${CMAKE_COMMAND} -E echo "uninstall include ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/simgrid_update_xml.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/tesh.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpicc.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpicxx.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpirun.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpiff.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpif90.1 COMMAND ${CMAKE_COMMAND} -E echo "uninstall man ok" WORKING_DIRECTORY "${CMAKE_INSTALL_PREFIX}") if(SIMGRID_HAVE_LUA) add_custom_command(TARGET uninstall COMMAND ${CMAKE_COMMAND} -E echo "uninstall binding lua ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/lua/5.1/simgrid.${LIB_EXE} WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}/") endif() ################################################################ ## Build a sain "make dist" target to build a source package ### ## containing only the files that I explicitely state ### ## (instead of any cruft laying on my disk as CPack does) ### ################################################################ # This is the complete list of what will be added to the source archive set(source_to_pack ${headers_to_install} ${source_of_generated_headers} ${BINDINGS_SRC} ${JEDULE_SRC} ${JMSG_C_SRC} ${JMSG_JAVA_SRC} ${LUA_SRC} ${MC_SRC_BASE} ${MC_SRC} ${MC_SIMGRID_MC_SRC} ${MSG_SRC} ${S4U_SRC} ${NS3_SRC} ${PLUGINS_SRC} ${RNGSTREAM_SRC} ${SIMDAG_SRC} ${SIMGRID_SRC} ${SIMIX_SRC} ${SMPI_SRC} ${SURF_SRC} ${TRACING_SRC} ${XBT_RL_SRC} ${XBT_SRC} ${EXTRA_DIST} ${CMAKE_SOURCE_FILES} ${CMAKEFILES_TXT} ${DOC_FIGS} ${DOC_IMG} ${DOC_SOURCES} ${DOC_TOOLS} ${PLATFORMS_EXAMPLES} ${README_files} ${bin_files} ${examples_src} ${tesh_files} ${teshsuite_src} ${testsuite_src} ${tools_src} ${txt_files} ${xml_files} ) ########################################## ### Fill in the "make dist-dir" target ### ########################################## add_custom_target(dist-dir COMMENT "Generating the distribution directory" COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/ COMMAND ${CMAKE_COMMAND} -E remove ${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version}/doc/html/ COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_HOME_DIRECTORY}/doc/html/ ${PROJECT_NAME}-${release_version}/doc/html/ COMMAND rm -f `grep -rl " Reference" ${PROJECT_NAME}-${release_version}/doc/html/` # Doxygen, go away COMMAND rm -f `grep -rl "Member List" ${PROJECT_NAME}-${release_version}/doc/html/` # Doxygen, you're getting annoying ) add_dependencies(dist-dir maintainer_files) set(dirs_in_tarball "") foreach(file ${source_to_pack}) #message(${file}) # This damn prefix is still set somewhere (seems to be in subdirs) string(REPLACE "${CMAKE_HOME_DIRECTORY}/" "" file "${file}") # Create the directory on need get_filename_component(file_location ${file} PATH) string(REGEX MATCH ";${file_location};" OPERATION "${dirs_in_tarball}") if(NOT OPERATION) set(dirs_in_tarball "${dirs_in_tarball};${file_location};") add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version}/${file_location}/) endif() # Actually copy the file add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/${file} ${PROJECT_NAME}-${release_version}/${file_location}) endforeach(file ${source_to_pack}) add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E echo "${GIT_VERSION}" > ${PROJECT_NAME}-${release_version}/.gitversion) ########################################################## ### Link all sources to the bindir if srcdir != bindir ### ########################################################## add_custom_target(hardlinks COMMENT "Making the source files available from the bindir" ) if (NOT ${CMAKE_SOURCE_DIR} STREQUAL ${CMAKE_BINARY_DIR}) foreach(file ${source_to_pack}) #message(${file}) # This damn prefix is still set somewhere (seems to be in subdirs) string(REPLACE "${CMAKE_HOME_DIRECTORY}/" "" file "${file}") # Create the directory on need get_filename_component(file_location ${file} PATH) string(REGEX MATCH ";${file_location};" OPERATION "${dirs_in_bindir}") if(NOT OPERATION) set(dirs_in_tarball "${dirs_in_bindir};${file_location};") add_custom_command( TARGET hardlinks COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/${file_location}/) endif() # Actually copy the file add_custom_command( TARGET hardlinks COMMAND if test -f ${CMAKE_HOME_DIRECTORY}/${file} \; then rm -f ${CMAKE_BINARY_DIR}/${file}\; ln ${CMAKE_HOME_DIRECTORY}/${file} ${CMAKE_BINARY_DIR}/${file_location}\; fi ) endforeach(file ${source_to_pack}) endif() ###################################### ### Fill in the "make dist" target ### ###################################### add_custom_target(dist COMMENT "Removing the distribution directory" DEPENDS ${CMAKE_BINARY_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E echo ${PROJECT_NAME}-${release_version} > ${CMAKE_BINARY_DIR}/VERSION COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/) add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMENT "Compressing the archive from the distribution directory" COMMAND ${CMAKE_COMMAND} -E tar cf ${PROJECT_NAME}-${release_version}.tar ${PROJECT_NAME}-${release_version}/ COMMAND gzip -9v ${PROJECT_NAME}-${release_version}.tar COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/) add_dependencies(dist dist-dir) if(NOT enable_maintainer_mode) add_custom_target(echo-dist COMMAND ${CMAKE_COMMAND} -E echo "WARNING: ----------------------------------------------------" COMMAND ${CMAKE_COMMAND} -E echo "WARNING: Distrib is generated without option maintainer mode " COMMAND ${CMAKE_COMMAND} -E echo "WARNING: ----------------------------------------------------") add_dependencies(dist echo-dist) endif() ########################################### ### Fill in the "make distcheck" target ### ########################################### set(CMAKE_BINARY_TEST_DIR ${CMAKE_BINARY_DIR}) # Allow to test the "make dist" add_custom_target(distcheck COMMAND ${CMAKE_COMMAND} -E echo "XXX compare archive with git repository" COMMAND ${CMAKE_HOME_DIRECTORY}/tools/internal/check_dist_archive -batch ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E echo "XXX remove old copy" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E echo "XXX Untar distrib" COMMAND ${CMAKE_COMMAND} -E tar xf ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}.tar.gz ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E echo "XXX create build and install subtrees" COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst COMMAND ${CMAKE_COMMAND} -E echo "XXX Configure" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_COMMAND} -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst -Denable_lto=OFF .. COMMAND ${CMAKE_COMMAND} -E echo "XXX Build" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} -j 4 COMMAND ${CMAKE_COMMAND} -E echo "XXX Test" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ctest --output-on-failure -j 4 COMMAND ${CMAKE_COMMAND} -E echo "XXX Install" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} install COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst/lib/libsimgrid.so ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst/lib/libsimgridtest.so COMMAND ${CMAKE_COMMAND} -E echo "XXX Install with documentation" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} install COMMAND ${CMAKE_COMMAND} -E echo "XXX Remove temp directories" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} ) add_dependencies(distcheck dist) ####################################### ### Fill in the "make check" target ### ####################################### if(enable_memcheck) add_custom_target(check COMMAND ctest -D ExperimentalMemCheck) else() add_custom_target(check COMMAND make test) endif() ####################################### ### Fill in the "make xxx-clean" target ### ####################################### add_custom_target(maintainer-clean COMMAND ${CMAKE_COMMAND} -E remove -f src/config_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/cunit_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/dict_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/dynar_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/ex_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/set_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/simgrid_units_main.c COMMAND ${CMAKE_COMMAND} -E remove -f src/xbt_str_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/xbt_synchro_unit.c WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}") include(CPack) SimGrid-3.18/tools/cmake/GCCFlags.cmake0000644000175000017500000002613413217757322020121 0ustar mquinsonmquinson## ## This file is in charge of setting our paranoid flags with regard to warnings and optimization. ## ## It is only used for gcc and clang. MSVC builds don't load this file. ## ## These flags do break some classical CMake tests, so you don't ## want to do so before the very end of the configuration. ## ## Other compiler flags (C/C++ standard version) are tested and set ## by the beginning of the configuration, directly in ~/CMakeList.txt set(warnCFLAGS "") set(optCFLAGS "") set(warnCXXFLAGS "") if(enable_compile_warnings) set(warnCFLAGS "-fno-common -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wcomment -Wformat -Wwrite-strings -Wno-unused-function -Wno-unused-parameter -Wno-strict-aliasing") if(CMAKE_COMPILER_IS_GNUCC AND (NOT (CMAKE_C_COMPILER_VERSION VERSION_LESS "5.0"))) set(warnCFLAGS "${warnCFLAGS} -Wformat-signedness") endif() if(CMAKE_C_COMPILER_ID MATCHES "Clang|GCC") set(warnCFLAGS "${warnCFLAGS} -Wno-format-nonliteral") endif() if(CMAKE_COMPILER_IS_GNUCC) set(warnCFLAGS "${warnCFLAGS} -Wclobbered -Wno-error=clobbered -Wno-unused-local-typedefs -Wno-error=attributes") endif() set(warnCXXFLAGS "${warnCFLAGS} -Wall -Wextra -Wunused -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wcomment -Wformat -Wwrite-strings -Wno-unused-function -Wno-unused-parameter -Wno-strict-aliasing") if(CMAKE_COMPILER_IS_GNUCXX AND (NOT (CMAKE_CXX_COMPILER_VERSION VERSION_LESS "5.0"))) set(warnCFLAGS "${warnCFLAGS} -Wformat-signedness") endif() if(CMAKE_CXX_COMPILER_ID MATCHES "Clang|GCC") set(warnCXXFLAGS "${warnCXXFLAGS} -Wno-format-nonliteral") endif() if(CMAKE_COMPILER_IS_GNUCXX) set(warnCXXFLAGS "${warnCXXFLAGS} -Wclobbered -Wno-error=clobbered -Wno-unused-local-typedefs -Wno-error=attributes") endif() if (CMAKE_CXX_COMPILER_ID MATCHES "Clang") # don't care about class that become struct, avoid issue of empty C structs # size (coming from libunwind.h) set(warnCXXFLAGS "${warnCXXFLAGS} -Wno-mismatched-tags -Wno-extern-c-compat") endif() # the one specific to C but refused by C++ set(warnCFLAGS "${warnCFLAGS} -Wmissing-prototypes") if(CMAKE_Fortran_COMPILER_ID MATCHES "GCC|PGI") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -Wall") endif() if(CMAKE_Fortran_COMPILER_ID MATCHES "Intel") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -warn all") endif() set(CMAKE_JAVA_COMPILE_FLAGS "-Xlint") endif() # NDEBUG gives a lot of "initialized but unused variables" errors. Don't die anyway. if(enable_compile_warnings AND enable_debug) set(warnCFLAGS "${warnCFLAGS} -Werror") set(warnCXXFLAGS "${warnCXXFLAGS} -Werror") endif() # Activate the warnings on #if FOOBAR when FOOBAR has no value # It breaks on FreeBSD within Boost headers, so activate this only in Pure Hardcore debug mode. if(enable_maintainer_mode) set(warnCFLAGS "${warnCFLAGS} -Wundef") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wundef") endif() # Se the optimisation flags # NOTE, we should CMAKE_BUILD_TYPE for this if(enable_compile_optimizations) set(optCFLAGS "-O3 -funroll-loops -fno-strict-aliasing ") else() set(optCFLAGS "-O0 ") endif() if(enable_compile_optimizations AND CMAKE_COMPILER_IS_GNUCC AND (NOT enable_model-checking)) # This is redundant (already in -03): set(optCFLAGS "${optCFLAGS} -finline-functions ") endif() # Do not leak the current directory into the binaries if(CMAKE_COMPILER_IS_GNUCC) execute_process(COMMAND realpath --relative-to=${CMAKE_BINARY_DIR} ${CMAKE_SOURCE_DIR} RESULT_VARIABLE RESULT OUTPUT_VARIABLE RELATIVE_SOURCE_DIR ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE) if(RESULT EQUAL 0) message(STATUS "Relative source directory is \"${RELATIVE_SOURCE_DIR}\".") else() message(WARNING "Failed to find relative source directory. Using \".\".") set(RELATIVE_SOURCE_DIR ".") endif() set(optCFLAGS "${optCFLAGS} -fdebug-prefix-map=${CMAKE_SOURCE_DIR}=${RELATIVE_SOURCE_DIR}") endif() # Configure LTO # NOTE, cmake 3.0 has a INTERPROCEDURAL_OPTIMIZATION target # property for this (http://www.cmake.org/cmake/help/v3.0/prop_tgt/INTERPROCEDURAL_OPTIMIZATION.html) if(enable_lto) # User wants LTO. Try if we can do that set(enable_lto OFF) if(enable_compile_optimizations AND CMAKE_COMPILER_IS_GNUCC AND (NOT enable_model-checking)) # On windows, we need 4.8 or higher to enable lto because of http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50293 # We are experiencing assertion failures even with 4.8 on MinGW. # Push the support forward: will see if 4.9 works when we test it. # # On Linux, we got the following with GCC 4.8.4 on Centos and Ubuntu # lto1: internal compiler error: in output_die, at dwarf2out.c:8478 # Please submit a full bug report, with preprocessed source if appropriate. # So instead, we push the support forward if ( (CMAKE_C_COMPILER_VERSION VERSION_GREATER "4.8.5") AND (LINKER_VERSION VERSION_GREATER "2.22")) set(enable_lto ON) endif() endif() if(enable_lto) message(STATUS "LTO seems usable.") else() if(NOT enable_compile_optimizations) message(STATUS "LTO disabled: Compile-time optimizations turned off.") else() if(enable_model-checking) message(STATUS "LTO disabled when compiling with model-checking.") else() message(STATUS "LTO does not seem usable -- try updating your build chain.") endif() endif() endif() else() message(STATUS "LTO disabled on the command line.") endif() if(enable_lto) # User wants LTO, and it seems usable. Go for it set(optCFLAGS "${optCFLAGS} -flto ") # See https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ#ar.2C_nm_and_ranlib: # "Since version 4.9 gcc produces slim object files that only contain # the intermediate representation. In order to handle archives of # these objects you have to use the gcc wrappers: # gcc-ar, gcc-nm and gcc-ranlib." if(${CMAKE_C_COMPILER_ID} STREQUAL "GNU" AND CMAKE_C_COMPILER_VERSION VERSION_GREATER "4.8") set (CMAKE_AR gcc-ar) set (CMAKE_RANLIB gcc-ranlib) endif() endif() if(enable_model-checking AND enable_compile_optimizations) # Forget it, do not optimize the code (because it confuses the MC): set(optCFLAGS "-O0 ") # But you can still optimize this: foreach(s src/kernel/lmm/fair_bottleneck.cpp src/kernel/lmm/lagrange.cpp src/kernel/lmm/maxmin.cpp src/xbt/mmalloc/mm.c src/xbt/log.c src/xbt/xbt_log_appender_file.c src/xbt/xbt_log_layout_format.c src/xbt/xbt_log_layout_simple.c src/xbt/dict.cpp src/xbt/dict_elm.c src/xbt/dict_cursor.c src/xbt/dynar.cpp src/xbt/xbt_str.cpp src/xbt/snprintf.c src/xbt/xbt_os_time.c src/xbt/xbt_os_thread.c src/xbt/backtrace_linux.cpp ${MC_SRC_BASE} ${MC_SRC}) set (mcCFLAGS "-O3 -funroll-loops -fno-strict-aliasing") if(CMAKE_COMPILER_IS_GNUCC) set (mcCFLAGS "${mcCFLAGS} -finline-functions") endif() set_source_files_properties(${s} PROPERTIES COMPILE_FLAGS ${mcCFLAGS}) endforeach() endif() if(NOT enable_debug) set(CMAKE_C_FLAGS "-DNDEBUG ${CMAKE_C_FLAGS}") set(CMAKE_CXX_FLAGS "-DNDEBUG ${CMAKE_CXX_FLAGS}") endif() set(CMAKE_C_FLAGS "${warnCFLAGS} ${CMAKE_C_FLAGS} ${optCFLAGS}") set(CMAKE_CXX_FLAGS "${warnCXXFLAGS} ${CMAKE_CXX_FLAGS} ${optCFLAGS}") # Try to make Mac a bit more complient to open source standards if(CMAKE_SYSTEM_NAME MATCHES "Darwin") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_XOPEN_SOURCE=700 -D_DARWIN_C_SOURCE") endif() set(TESH_OPTION "") if(enable_coverage) find_program(GCOV_PATH gcov) if(GCOV_PATH) set(COVERAGE_COMMAND "${GCOV_PATH}" CACHE TYPE FILEPATH FORCE) set(COVERAGE_EXTRA_FLAGS "-l -p") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DCOVERAGE") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fprofile-arcs -ftest-coverage") add_definitions(-fprofile-arcs -ftest-coverage) endif() endif() if(enable_address_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address -fno-omit-frame-pointer") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=address -fno-omit-frame-pointer") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=address") set(TESH_OPTION --enable-sanitizers) try_compile(HAVE_SANITIZE_ADDRESS ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_asan.cpp) try_compile(HAVE_SANITIZE_ADDRESS_FIBER_SUPPORT ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_asan.cpp COMPILE_DEFINITIONS -DCHECK_FIBER_SUPPORT) else() set(HAVE_SANITIZE_ADDRESS FALSE CACHE INTERNAL "") set(HAVE_SANITIZE_ADDRESS_FIBER_SUPPORT FALSE CACHE INTERNAL "") endif() if(enable_thread_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=thread -fno-omit-frame-pointer -no-pie") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=thread -fno-omit-frame-pointer -no-pie") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=thread -no-pie") endif() if(enable_undefined_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=undefined -fno-omit-frame-pointer") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=undefined -fno-omit-frame-pointer") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=undefined") endif() if(NOT $ENV{CFLAGS} STREQUAL "") message(STATUS "Add CFLAGS: \"$ENV{CFLAGS}\" to CMAKE_C_FLAGS") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} $ENV{CFLAGS}") endif() if(NOT $ENV{CXXFLAGS} STREQUAL "") message(STATUS "Add CXXFLAGS: \"$ENV{CXXFLAGS}\" to CMAKE_CXX_FLAGS") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} $ENV{CXXFLAGS}") endif() if(NOT $ENV{LDFLAGS} STREQUAL "") message(STATUS "Add LDFLAGS: \"$ENV{LDFLAGS}\" to CMAKE_C_LINK_FLAGS") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} $ENV{LDFLAGS}") endif() if(MINGW) # http://stackoverflow.com/questions/10452262/create-64-bit-jni-under-windows # We don't want to ship libgcc_s_seh-1.dll nor libstdc++-6.dll set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -static-libgcc") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -static-libgcc -static-libstdc++") set(CMAKE_SHARED_LIBRARY_LINK_C_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_C_FLAGS} -static-libgcc") set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS} -static-libgcc -static-libstdc++") # JNI searches for stdcalls set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_SHARED_LIBRARY_LINK_C_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_C_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS} -Wl,--add-stdcall-alias") # Specify the data model that we are using (yeah it may help Java) if(CMAKE_SIZEOF_VOID_P EQUAL 4) # 32 bits set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -m32") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m32") else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -m64") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m64") endif() endif() SimGrid-3.18/tools/cmake/CTestConfig.cmake0000644000175000017500000000310413217757322020710 0ustar mquinsonmquinson# Configure CTest. For details, see: # http://www.cmake.org/Wiki/CMake_Testing_With_CTest#Customizing_CTest if(APPLE) SET(BUILDNAME "APPLE" CACHE INTERNAL "Buildname" FORCE) else() SET(BUILDNAME "UNIX" CACHE INTERNAL "Buildname" FORCE) if(WIN32) SET(BUILDNAME "WINDOWS" CACHE INTERNAL "Buildname" FORCE) endif() endif() if(NOT enable_memcheck) set(DART_TESTING_TIMEOUT "500") #TIMEOUT FOR EACH TEST else() set(DART_TESTING_TIMEOUT "3000") #TIMEOUT FOR EACH TEST endif() if(enable_compile_warnings AND enable_compile_optimizations) SET(BUILDNAME "FULL_FLAGS" CACHE INTERNAL "Buildname" FORCE) endif() if(SIMGRID_HAVE_MC) SET(BUILDNAME "MODEL-CHECKING" CACHE INTERNAL "Buildname" FORCE) endif() if(enable_memcheck) SET(BUILDNAME "MEMCHECK" CACHE INTERNAL "Buildname" FORCE) endif() set(osname ${CMAKE_SYSTEM_NAME}) set(cpu ${CMAKE_SYSTEM_PROCESSOR}) set(DISTRIB2 ${CMAKE_SYSTEM_VERSION}) SET(SITE "${osname}_${DISTRIB2}_${cpu}") SET(CTEST_SITE "${osname}_${DISTRIB2}_${cpu}") SET(CTEST_PROJECT_NAME "${PROJECT_NAME}") SET(CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE "3000000") SET(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE "3000000") set(PATTERN_CTEST_IGNORED "") if(enable_coverage) set(PATTERN_CTEST_IGNORED "/tools/" "/buildtools/" "/include/" "/teshsuite/" "/src/bindings/" ) if(NOT release) set(PATTERN_CTEST_IGNORED ${PATTERN_CTEST_IGNORED} "/examples/" ) endif() endif() CONFIGURE_FILE(${CMAKE_HOME_DIRECTORY}/tools/cmake/CTestCustom.cmake ${CMAKE_BINARY_DIR}/CTestCustom.cmake @ONLY) SimGrid-3.18/tools/cmake/DefinePackages.cmake0000644000175000017500000011026213217757322021375 0ustar mquinsonmquinson### define source packages set(EXTRA_DIST src/bindings/java/MANIFEST.in src/include/instr/instr_interface.h src/include/mc/datatypes.h src/include/mc/mc.h src/include/simgrid/sg_config.h src/include/surf/surf.hpp src/include/xbt/parmap.hpp src/mc/mc_mmu.hpp src/mc/mc_record.hpp src/mc/PageStore.hpp src/msg/msg_private.hpp src/simdag/dax.dtd src/simdag/dax_dtd.c src/simdag/dax_dtd.h src/simdag/simdag_private.hpp src/simix/simcalls.in src/simix/simcalls.py src/simix/popping_private.hpp src/simix/popping_bodies.cpp src/simix/popping_generated.cpp src/simix/popping_enum.h src/simix/popping_accessors.hpp src/simix/smx_host_private.hpp src/simix/smx_io_private.hpp src/simix/smx_network_private.hpp src/simix/smx_private.hpp src/simix/smx_synchro_private.hpp src/smpi/colls/coll_tuned_topo.hpp src/smpi/colls/colls_private.hpp src/smpi/colls/smpi_mvapich2_selector_stampede.hpp src/smpi/include/private.hpp src/smpi/include/smpi_utils.hpp src/surf/cpu_cas01.hpp src/surf/cpu_interface.hpp src/surf/cpu_ti.hpp src/surf/network_cm02.hpp src/surf/network_constant.hpp src/surf/network_interface.hpp src/surf/network_ns3.hpp src/surf/network_smpi.hpp src/surf/network_ib.hpp src/surf/ns3/ns3_interface.hpp src/surf/ns3/ns3_simulator.hpp src/surf/trace_mgr_test.cpp src/surf/xml/simgrid.dtd src/surf/xml/simgrid_dtd.h src/surf/xml/simgrid_dtd.c src/surf/xml/surfxml_sax_cb.cpp src/surf/StorageImpl.hpp src/surf/storage_n11.hpp src/surf/surf_interface.hpp src/surf/surf_private.hpp src/surf/PropertyHolder.hpp src/surf/host_clm03.hpp src/surf/HostImpl.hpp src/surf/ptask_L07.hpp src/xbt/automaton/automaton_lexer.yy.c src/xbt/automaton/parserPromela.lex src/xbt/automaton/parserPromela.tab.cacc src/xbt/automaton/parserPromela.tab.hacc src/xbt/automaton/parserPromela.yacc src/xbt/backtrace_dummy.cpp src/xbt/backtrace_linux.cpp src/xbt/dict_private.h src/xbt/log_private.h src/xbt/mallocator_private.h src/xbt/mmalloc/mfree.c src/xbt/mmalloc/mm.c src/xbt/mmalloc/mm_legacy.c src/xbt/mmalloc/mm_module.c src/xbt/mmalloc/mmalloc.c src/xbt/mmalloc/mmalloc.info src/xbt/mmalloc/mmalloc.texi src/xbt/mmalloc/mmorecore.c src/xbt/mmalloc/mmprivate.h src/xbt/mmalloc/mrealloc.c src/xbt/mmalloc/swag.c src/xbt/mmalloc/swag.h tools/tesh/generate_tesh tools/lualib.patch teshsuite/lua/lua_platforms.tesh examples/smpi/mc/only_send_deterministic.tesh examples/smpi/mc/non_deterministic.tesh examples/java/.classpath examples/java/.project ) set(SMPI_SRC src/smpi/smpi_main.c src/smpi/bindings/smpi_mpi.cpp src/smpi/bindings/smpi_pmpi.cpp src/smpi/bindings/smpi_pmpi_coll.cpp src/smpi/bindings/smpi_pmpi_comm.cpp src/smpi/bindings/smpi_pmpi_group.cpp src/smpi/bindings/smpi_pmpi_info.cpp src/smpi/bindings/smpi_pmpi_op.cpp src/smpi/bindings/smpi_pmpi_request.cpp src/smpi/bindings/smpi_pmpi_topo.cpp src/smpi/bindings/smpi_pmpi_type.cpp src/smpi/bindings/smpi_pmpi_win.cpp src/smpi/bindings/smpi_f77.cpp src/smpi/bindings/smpi_f77_coll.cpp src/smpi/bindings/smpi_f77_comm.cpp src/smpi/bindings/smpi_f77_request.cpp src/smpi/bindings/smpi_f77_type.cpp src/smpi/colls/allgather/allgather-2dmesh.cpp src/smpi/colls/allgather/allgather-3dmesh.cpp src/smpi/colls/allgather/allgather-GB.cpp src/smpi/colls/allgather/allgather-NTSLR-NB.cpp src/smpi/colls/allgather/allgather-NTSLR.cpp src/smpi/colls/allgather/allgather-SMP-NTS.cpp src/smpi/colls/allgather/allgather-bruck.cpp src/smpi/colls/allgather/allgather-loosely-lr.cpp src/smpi/colls/allgather/allgather-ompi-neighborexchange.cpp src/smpi/colls/allgather/allgather-pair.cpp src/smpi/colls/allgather/allgather-mvapich-smp.cpp src/smpi/colls/allgather/allgather-rdb.cpp src/smpi/colls/allgather/allgather-rhv.cpp src/smpi/colls/allgather/allgather-ring.cpp src/smpi/colls/allgather/allgather-smp-simple.cpp src/smpi/colls/allgather/allgather-spreading-simple.cpp src/smpi/colls/allgatherv/allgatherv-GB.cpp src/smpi/colls/allgatherv/allgatherv-mpich-rdb.cpp src/smpi/colls/allgatherv/allgatherv-mpich-ring.cpp src/smpi/colls/allgatherv/allgatherv-ompi-bruck.cpp src/smpi/colls/allgatherv/allgatherv-ompi-neighborexchange.cpp src/smpi/colls/allgatherv/allgatherv-pair.cpp src/smpi/colls/allgatherv/allgatherv-ring.cpp src/smpi/colls/allreduce/allreduce-lr.cpp src/smpi/colls/allreduce/allreduce-ompi-ring-segmented.cpp src/smpi/colls/allreduce/allreduce-rab-rdb.cpp src/smpi/colls/allreduce/allreduce-rab1.cpp src/smpi/colls/allreduce/allreduce-rab2.cpp src/smpi/colls/allreduce/allreduce-rdb.cpp src/smpi/colls/allreduce/allreduce-redbcast.cpp src/smpi/colls/allreduce/allreduce-smp-binomial-pipeline.cpp src/smpi/colls/allreduce/allreduce-smp-binomial.cpp src/smpi/colls/allreduce/allreduce-smp-rdb.cpp src/smpi/colls/allreduce/allreduce-smp-rsag-lr.cpp src/smpi/colls/allreduce/allreduce-smp-rsag-rab.cpp src/smpi/colls/allreduce/allreduce-smp-rsag.cpp src/smpi/colls/allreduce/allreduce-mvapich-rs.cpp src/smpi/colls/allreduce/allreduce-mvapich-two-level.cpp src/smpi/colls/alltoall/alltoall-basic-linear.cpp src/smpi/colls/alltoall/alltoall-2dmesh.cpp src/smpi/colls/alltoall/alltoall-3dmesh.cpp src/smpi/colls/alltoall/alltoall-bruck.cpp src/smpi/colls/alltoall/alltoall-pair-light-barrier.cpp src/smpi/colls/alltoall/alltoall-pair-mpi-barrier.cpp src/smpi/colls/alltoall/alltoall-pair-one-barrier.cpp src/smpi/colls/alltoall/alltoall-pair.cpp src/smpi/colls/alltoall/alltoall-rdb.cpp src/smpi/colls/alltoall/alltoall-ring-light-barrier.cpp src/smpi/colls/alltoall/alltoall-ring-mpi-barrier.cpp src/smpi/colls/alltoall/alltoall-ring-one-barrier.cpp src/smpi/colls/alltoall/alltoall-ring.cpp src/smpi/colls/alltoall/alltoall-mvapich-scatter-dest.cpp src/smpi/colls/alltoallv/alltoallv-bruck.cpp src/smpi/colls/alltoallv/alltoallv-ompi-basic-linear.cpp src/smpi/colls/alltoallv/alltoallv-pair-light-barrier.cpp src/smpi/colls/alltoallv/alltoallv-pair-mpi-barrier.cpp src/smpi/colls/alltoallv/alltoallv-pair-one-barrier.cpp src/smpi/colls/alltoallv/alltoallv-pair.cpp src/smpi/colls/alltoallv/alltoallv-ring-light-barrier.cpp src/smpi/colls/alltoallv/alltoallv-ring-mpi-barrier.cpp src/smpi/colls/alltoallv/alltoallv-ring-one-barrier.cpp src/smpi/colls/alltoallv/alltoallv-ring.cpp src/smpi/colls/barrier/barrier-ompi.cpp src/smpi/colls/barrier/barrier-mvapich2-pair.cpp src/smpi/colls/bcast/bcast-NTSB.cpp src/smpi/colls/bcast/bcast-NTSL-Isend.cpp src/smpi/colls/bcast/bcast-NTSL.cpp src/smpi/colls/bcast/bcast-SMP-binary.cpp src/smpi/colls/bcast/bcast-SMP-binomial.cpp src/smpi/colls/bcast/bcast-SMP-linear.cpp src/smpi/colls/bcast/bcast-arrival-pattern-aware-wait.cpp src/smpi/colls/bcast/bcast-arrival-pattern-aware.cpp src/smpi/colls/bcast/bcast-arrival-scatter.cpp src/smpi/colls/bcast/bcast-binomial-tree.cpp src/smpi/colls/bcast/bcast-flattree-pipeline.cpp src/smpi/colls/bcast/bcast-flattree.cpp src/smpi/colls/bcast/bcast-ompi-pipeline.cpp src/smpi/colls/bcast/bcast-ompi-split-bintree.cpp src/smpi/colls/bcast/bcast-mvapich-smp.cpp src/smpi/colls/bcast/bcast-scatter-LR-allgather.cpp src/smpi/colls/bcast/bcast-scatter-rdb-allgather.cpp src/smpi/colls/coll_tuned_topo.cpp src/smpi/colls/colls_global.cpp src/smpi/colls/gather/gather-ompi.cpp src/smpi/colls/gather/gather-mvapich.cpp src/smpi/colls/reduce/reduce-NTSL.cpp src/smpi/colls/reduce/reduce-arrival-pattern-aware.cpp src/smpi/colls/reduce/reduce-binomial.cpp src/smpi/colls/reduce/reduce-flat-tree.cpp src/smpi/colls/reduce/reduce-ompi.cpp src/smpi/colls/reduce/reduce-scatter-gather.cpp src/smpi/colls/reduce_scatter/reduce_scatter-mpich.cpp src/smpi/colls/reduce_scatter/reduce_scatter-ompi.cpp src/smpi/colls/reduce/reduce-mvapich-knomial.cpp src/smpi/colls/reduce/reduce-mvapich-two-level.cpp src/smpi/colls/reduce/reduce-rab.cpp src/smpi/colls/scatter/scatter-ompi.cpp src/smpi/colls/scatter/scatter-mvapich-two-level.cpp src/smpi/colls/smpi_automatic_selector.cpp src/smpi/colls/smpi_default_selector.cpp src/smpi/colls/smpi_mpich_selector.cpp src/smpi/colls/smpi_intel_mpi_selector.cpp src/smpi/colls/smpi_openmpi_selector.cpp src/smpi/colls/smpi_mvapich2_selector.cpp src/smpi/colls/smpi_coll.cpp src/smpi/internals/instr_smpi.cpp src/smpi/internals/smpi_bench.cpp src/smpi/internals/smpi_memory.cpp src/smpi/internals/smpi_shared.cpp src/smpi/internals/smpi_static_variables.cpp src/smpi/internals/smpi_deployment.cpp src/smpi/internals/smpi_dvfs.cpp src/smpi/internals/smpi_global.cpp src/smpi/internals/SmpiHost.cpp src/smpi/internals/smpi_replay.cpp src/smpi/internals/smpi_process.cpp src/smpi/internals/smpi_utils.cpp src/smpi/mpi/smpi_comm.cpp src/smpi/mpi/smpi_datatype.cpp src/smpi/mpi/smpi_datatype_derived.cpp src/smpi/mpi/smpi_f2c.cpp src/smpi/mpi/smpi_group.cpp src/smpi/mpi/smpi_info.cpp src/smpi/mpi/smpi_keyvals.cpp src/smpi/mpi/smpi_op.cpp src/smpi/mpi/smpi_request.cpp src/smpi/mpi/smpi_status.cpp src/smpi/mpi/smpi_topo.cpp src/smpi/mpi/smpi_win.cpp src/smpi/include/smpi_coll.hpp src/smpi/include/smpi_comm.hpp src/smpi/include/smpi_f2c.hpp src/smpi/include/smpi_group.hpp src/smpi/include/SmpiHost.hpp src/smpi/include/smpi_datatype.hpp src/smpi/include/smpi_info.hpp src/smpi/include/smpi_keyvals.hpp src/smpi/include/smpi_datatype_derived.hpp src/smpi/include/smpi_op.hpp src/smpi/include/smpi_process.hpp src/smpi/include/smpi_request.hpp src/smpi/include/smpi_status.hpp src/smpi/include/smpi_win.hpp src/smpi/include/smpi_topo.hpp src/surf/network_smpi.cpp src/surf/network_ib.cpp ) set(XBT_SRC src/xbt/RngStream.c src/xbt/automaton/automaton.c src/xbt/automaton/automatonparse_promela.c src/xbt/backtrace.cpp src/xbt/config.cpp src/xbt/cunit.cpp src/xbt/dict.cpp src/xbt/dict_cursor.c src/xbt/dict_elm.c src/xbt/dynar.cpp src/xbt/ex.cpp src/xbt/exception.cpp src/xbt/graph.c src/xbt/log.c src/xbt/mallocator.c src/xbt/memory_map.cpp src/xbt/memory_map.hpp src/xbt/parmap.cpp src/xbt/snprintf.c src/xbt/string.cpp src/xbt/xbt_log_appender_file.c src/xbt/xbt_log_layout_format.c src/xbt/xbt_log_layout_simple.c src/xbt/xbt_main.cpp src/xbt/xbt_os_file.cpp src/xbt/xbt_os_synchro.cpp src/xbt/xbt_os_time.c src/xbt/xbt_replay.cpp src/xbt/xbt_str.cpp src/xbt/xbt_virtu.c src/xbt_modinter.h ) if(HAVE_MMALLOC) set(XBT_SRC ${XBT_SRC} src/xbt/mmalloc/mm.c ) endif() set(NS3_SRC src/surf/network_ns3.cpp src/surf/ns3/ns3_simulator.cpp ) set(SURF_SRC src/kernel/lmm/fair_bottleneck.cpp src/kernel/lmm/lagrange.cpp src/kernel/lmm/maxmin.hpp src/kernel/lmm/maxmin.cpp src/kernel/routing/ClusterZone.cpp src/kernel/routing/ClusterZone.hpp src/kernel/routing/DijkstraZone.cpp src/kernel/routing/DijkstraZone.hpp src/kernel/routing/DragonflyZone.cpp src/kernel/routing/DragonflyZone.hpp src/kernel/routing/EmptyZone.cpp src/kernel/routing/EmptyZone.hpp src/kernel/routing/FatTreeZone.cpp src/kernel/routing/FatTreeZone.hpp src/kernel/routing/FloydZone.cpp src/kernel/routing/FloydZone.hpp src/kernel/routing/FullZone.cpp src/kernel/routing/FullZone.hpp src/kernel/routing/NetPoint.cpp src/kernel/routing/NetPoint.hpp src/kernel/routing/NetZoneImpl.cpp src/kernel/routing/NetZoneImpl.hpp src/kernel/routing/RoutedZone.cpp src/kernel/routing/RoutedZone.hpp src/kernel/routing/TorusZone.cpp src/kernel/routing/TorusZone.hpp src/kernel/routing/VivaldiZone.cpp src/kernel/routing/VivaldiZone.hpp src/kernel/EngineImpl.cpp src/kernel/EngineImpl.hpp src/surf/cpu_cas01.cpp src/surf/cpu_interface.cpp src/surf/cpu_ti.cpp src/surf/instr_routing.cpp src/surf/instr_surf.cpp src/surf/network_cm02.cpp src/surf/network_constant.cpp src/surf/network_interface.cpp src/surf/plugins/dirty_page_tracking.cpp src/surf/plugins/host_energy.cpp src/surf/plugins/link_energy.cpp src/surf/plugins/host_load.cpp src/surf/PropertyHolder.cpp src/surf/sg_platf.cpp src/surf/StorageImpl.cpp src/surf/storage_n11.cpp src/surf/surf_c_bindings.cpp src/surf/surf_interface.cpp src/surf/xml/platf.hpp src/surf/xml/platf_private.hpp src/surf/xml/surfxml_sax_cb.cpp src/surf/xml/surfxml_parseplatf.cpp src/surf/trace_mgr.hpp src/surf/trace_mgr.cpp src/surf/host_clm03.cpp src/surf/HostImpl.cpp src/surf/ptask_L07.cpp ) set(PLUGINS_SRC src/plugins/file_system/FileSystem.hpp src/plugins/file_system/s4u_FileSystem.cpp src/plugins/vm/VirtualMachineImpl.hpp src/plugins/vm/s4u_VirtualMachine.cpp src/plugins/vm/VirtualMachineImpl.hpp src/plugins/vm/VirtualMachineImpl.cpp src/plugins/vm/VmHostExt.hpp src/plugins/vm/VmHostExt.cpp ) set(SIMIX_GENERATED_SRC src/simix/popping_generated.cpp ) set(SIMIX_SRC src/kernel/future.cpp src/simix/libsmx.cpp src/simix/smx_context.cpp src/kernel/context/Context.cpp src/kernel/context/Context.hpp src/kernel/context/ContextRaw.cpp src/kernel/context/ContextRaw.hpp src/simix/smx_deployment.cpp src/simix/smx_environment.cpp src/simix/smx_global.cpp src/simix/smx_host.cpp src/simix/smx_io.cpp src/simix/smx_network.cpp src/simix/ActorImpl.cpp src/simix/ActorImpl.hpp src/simix/smx_synchro.cpp src/simix/popping.cpp src/kernel/activity/ActivityImpl.cpp src/kernel/activity/ActivityImpl.hpp src/kernel/activity/CommImpl.cpp src/kernel/activity/CommImpl.hpp src/kernel/activity/ExecImpl.cpp src/kernel/activity/ExecImpl.hpp src/kernel/activity/MailboxImpl.cpp src/kernel/activity/MailboxImpl.hpp src/kernel/activity/SleepImpl.cpp src/kernel/activity/SleepImpl.hpp src/kernel/activity/SynchroIo.cpp src/kernel/activity/SynchroIo.hpp src/kernel/activity/SynchroRaw.cpp src/kernel/activity/SynchroRaw.hpp ${SIMIX_GENERATED_SRC} ) # Boost context may not be available if (HAVE_BOOST_CONTEXTS) set(SIMIX_SRC ${SIMIX_SRC} src/kernel/context/ContextBoost.hpp src/kernel/context/ContextBoost.cpp) else() set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextBoost.hpp src/kernel/context/ContextBoost.cpp) endif() set(S4U_SRC src/s4u/s4u_actor.cpp src/s4u/s4u_activity.cpp src/s4u/s4u_conditionVariable.cpp src/s4u/s4u_comm.cpp src/s4u/s4u_engine.cpp src/s4u/s4u_exec.cpp src/s4u/s4u_host.cpp src/s4u/s4u_link.cpp src/s4u/s4u_mailbox.cpp src/s4u/s4u_mutex.cpp src/s4u/s4u_netzone.cpp src/s4u/s4u_storage.cpp ) set(SIMGRID_SRC src/simgrid/sg_config.cpp src/simgrid/host.cpp src/simgrid/util.hpp ) set(MSG_SRC src/msg/instr_msg_process.cpp src/msg/instr_msg_task.cpp src/msg/msg_actions.cpp src/msg/msg_deployment.cpp src/msg/msg_environment.cpp src/msg/msg_global.cpp src/msg/msg_gos.cpp src/msg/msg_host.cpp src/msg/msg_io.cpp src/msg/msg_mailbox.cpp src/msg/msg_process.cpp src/msg/msg_synchro.cpp src/msg/msg_task.cpp src/msg/msg_vm.cpp ) set(SIMDAG_SRC src/simdag/sd_daxloader.cpp src/simdag/sd_dotloader.cpp src/simdag/sd_global.cpp src/simdag/sd_task.cpp ) set(BINDINGS_SRC src/bindings/lua/lua_private.hpp src/bindings/lua/lua_utils.hpp src/bindings/lua/simgrid_lua.hpp ) set(JMSG_C_SRC src/bindings/java/jmsg.cpp src/bindings/java/jmsg.hpp src/bindings/java/jmsg_as.cpp src/bindings/java/jmsg_as.hpp src/bindings/java/jmsg_comm.cpp src/bindings/java/jmsg_comm.h src/bindings/java/jmsg_file.cpp src/bindings/java/jmsg_file.h src/bindings/java/jmsg_host.cpp src/bindings/java/jmsg_host.h src/bindings/java/jmsg_process.cpp src/bindings/java/jmsg_process.h src/bindings/java/jmsg_rngstream.cpp src/bindings/java/jmsg_rngstream.h src/bindings/java/jmsg_synchro.cpp src/bindings/java/jmsg_synchro.h src/bindings/java/jmsg_task.cpp src/bindings/java/jmsg_task.h src/bindings/java/jmsg_vm.cpp src/bindings/java/jmsg_vm.h src/bindings/java/jxbt_utilities.cpp src/bindings/java/jxbt_utilities.hpp src/bindings/java/JavaContext.cpp src/bindings/java/JavaContext.hpp src/bindings/java/jmsg_storage.cpp src/bindings/java/jmsg_storage.h ) set(JMSG_JAVA_SRC src/bindings/java/org/simgrid/NativeLib.java src/bindings/java/org/simgrid/msg/As.java src/bindings/java/org/simgrid/msg/Comm.java src/bindings/java/org/simgrid/msg/File.java src/bindings/java/org/simgrid/msg/Host.java src/bindings/java/org/simgrid/msg/HostFailureException.java src/bindings/java/org/simgrid/msg/HostNotFoundException.java src/bindings/java/org/simgrid/msg/JniException.java src/bindings/java/org/simgrid/msg/Msg.java src/bindings/java/org/simgrid/msg/MsgException.java src/bindings/java/org/simgrid/msg/Mutex.java src/bindings/java/org/simgrid/msg/Process.java src/bindings/java/org/simgrid/msg/ProcessKilledError.java src/bindings/java/org/simgrid/msg/ProcessNotFoundException.java src/bindings/java/org/simgrid/msg/RngStream.java src/bindings/java/org/simgrid/msg/Semaphore.java src/bindings/java/org/simgrid/msg/Storage.java src/bindings/java/org/simgrid/msg/StorageNotFoundException.java src/bindings/java/org/simgrid/msg/Task.java src/bindings/java/org/simgrid/msg/TaskCancelledException.java src/bindings/java/org/simgrid/msg/TimeoutException.java src/bindings/java/org/simgrid/msg/TransferFailureException.java src/bindings/java/org/simgrid/msg/VM.java ) set(JTRACE_C_SRC src/bindings/java/jtrace.cpp src/bindings/java/jtrace.h ) set(JTRACE_JAVA_SRC src/bindings/java/org/simgrid/trace/Trace.java) list(APPEND JMSG_C_SRC ${JTRACE_C_SRC}) list(APPEND JMSG_JAVA_SRC ${JTRACE_JAVA_SRC}) set(LUA_SRC src/bindings/lua/lua_host.cpp src/bindings/lua/lua_platf.cpp src/bindings/lua/lua_debug.cpp src/bindings/lua/simgrid_lua.cpp ) set(TRACING_SRC src/instr/instr_config.cpp src/instr/instr_interface.cpp src/instr/instr_paje_containers.cpp src/instr/instr_paje_containers.hpp src/instr/instr_paje_events.cpp src/instr/instr_paje_events.hpp src/instr/instr_paje_header.cpp src/instr/instr_paje_trace.cpp src/instr/instr_paje_types.cpp src/instr/instr_paje_types.hpp src/instr/instr_paje_values.cpp src/instr/instr_paje_values.hpp src/instr/instr_private.hpp src/instr/instr_smpi.hpp src/instr/instr_resource_utilization.cpp ) set(JEDULE_SRC include/simgrid/jedule/jedule_events.hpp include/simgrid/jedule/jedule.hpp include/simgrid/jedule/jedule_platform.hpp include/simgrid/jedule/jedule_sd_binding.h src/instr/jedule/jedule_events.cpp src/instr/jedule/jedule.cpp src/instr/jedule/jedule_platform.cpp src/instr/jedule/jedule_sd_binding.cpp ) set(MC_SRC_BASE src/mc/mc_base.cpp src/mc/mc_base.h src/mc/mc_record.hpp src/mc/mc_replay.hpp src/mc/mc_record.cpp src/mc/mc_config.cpp src/mc/mc_global.cpp ) set(MC_SRC src/mc/checker/Checker.cpp src/mc/checker/Checker.hpp src/mc/checker/CommunicationDeterminismChecker.cpp src/mc/checker/CommunicationDeterminismChecker.hpp src/mc/checker/SafetyChecker.cpp src/mc/checker/SafetyChecker.hpp src/mc/checker/LivenessChecker.cpp src/mc/checker/LivenessChecker.hpp src/mc/remote/Channel.cpp src/mc/remote/Channel.hpp src/mc/remote/Client.cpp src/mc/remote/Client.hpp src/mc/remote/RemoteClient.hpp src/mc/remote/RemoteClient.cpp src/mc/remote/RemotePtr.hpp src/mc/remote/mc_protocol.h src/mc/remote/mc_protocol.cpp src/mc/AddressSpace.hpp src/mc/Frame.hpp src/mc/Frame.cpp src/mc/ModelChecker.hpp src/mc/ModelChecker.cpp src/mc/ObjectInformation.hpp src/mc/ObjectInformation.cpp src/mc/PageStore.hpp src/mc/PageStore.cpp src/mc/ChunkedData.hpp src/mc/ChunkedData.cpp src/mc/RegionSnapshot.cpp src/mc/RegionSnapshot.hpp src/mc/Type.hpp src/mc/Variable.hpp src/mc/mc_forward.hpp src/mc/Session.cpp src/mc/Session.hpp src/mc/mc_unw.hpp src/mc/mc_unw.cpp src/mc/mc_unw_vmread.cpp src/mc/mc_checkpoint.cpp src/mc/mc_snapshot.hpp src/mc/mc_snapshot.cpp src/mc/mc_page_snapshot.cpp src/mc/mc_comm_pattern.cpp src/mc/mc_comm_pattern.hpp src/mc/compare.cpp src/mc/mc_dwarf.hpp src/mc/mc_dwarf.cpp src/mc/mc_dwarf_attrnames.cpp src/mc/DwarfExpression.hpp src/mc/DwarfExpression.cpp src/mc/mc_dwarf_tagnames.cpp src/mc/mc_hash.hpp src/mc/mc_hash.cpp src/mc/mc_ignore.h src/mc/LocationList.hpp src/mc/LocationList.cpp src/mc/mc_record.cpp src/mc/mc_member.cpp src/mc/mc_memory.cpp src/mc/mc_private.hpp src/mc/mc_request.hpp src/mc/mc_request.cpp src/mc/mc_safety.hpp src/mc/mc_state.hpp src/mc/mc_state.cpp src/mc/VisitedState.cpp src/mc/VisitedState.hpp src/mc/mc_client_api.cpp src/mc/mc_smx.hpp src/mc/mc_smx.cpp src/mc/mc_xbt.hpp src/mc/mc_xbt.cpp src/mc/mc_exit.hpp src/mc/Transition.hpp ) set(MC_SIMGRID_MC_SRC src/mc/checker/simgrid_mc.cpp) set(headers_to_install include/simgrid/chrono.hpp include/simgrid/plugins/energy.h include/simgrid/plugins/file_system.h include/simgrid/plugins/live_migration.h include/simgrid/plugins/load.h include/simgrid/instr.h include/simgrid/msg.h include/simgrid/simdag.h include/simgrid/datatypes.h include/simgrid/modelchecker.h include/simgrid/forward.h include/simgrid/simix.h include/simgrid/simix.hpp include/simgrid/simix/blocking_simcall.hpp include/simgrid/kernel/future.hpp include/simgrid/host.h include/simgrid/link.h include/simgrid/s4u/forward.hpp include/simgrid/s4u/Activity.hpp include/simgrid/s4u/Actor.hpp include/simgrid/s4u/Comm.hpp include/simgrid/s4u/ConditionVariable.hpp include/simgrid/s4u/Engine.hpp include/simgrid/s4u/Exec.hpp include/simgrid/s4u/Host.hpp include/simgrid/s4u/Link.hpp include/simgrid/s4u/Mailbox.hpp include/simgrid/s4u/Mutex.hpp include/simgrid/s4u/NetZone.hpp include/simgrid/s4u/Storage.hpp include/simgrid/s4u/VirtualMachine.hpp include/simgrid/s4u.hpp include/smpi/mpi.h include/smpi/smpi.h include/smpi/smpi_main.h include/smpi/smpi_extended_traces.h include/smpi/smpi_extended_traces_fortran.h include/smpi/forward.hpp include/xbt.h include/xbt/algorithm.hpp include/xbt/asserts.h include/xbt/automaton.h include/xbt/automaton.hpp include/xbt/backtrace.h include/xbt/backtrace.hpp include/xbt/base.h include/xbt/config.h include/xbt/config.hpp include/xbt/cunit.h include/xbt/dict.h include/xbt/dynar.h include/xbt/dynar.hpp include/xbt/ex.h include/xbt/ex.hpp include/xbt/exception.hpp include/xbt/Extendable.hpp include/xbt/file.hpp include/xbt/functional.hpp include/xbt/function_types.h include/xbt/future.hpp include/xbt/graph.h include/xbt/log.h include/xbt/log.hpp include/xbt/mallocator.h include/xbt/misc.h include/xbt/mmalloc.h include/xbt/module.h include/xbt/parmap.h include/xbt/range.hpp include/xbt/replay.hpp include/xbt/RngStream.h include/xbt/signal.hpp include/xbt/str.h include/xbt/string.hpp include/xbt/synchro.h include/xbt/sysdep.h include/xbt/system_error.hpp include/xbt/utility.hpp include/xbt/virtu.h include/xbt/xbt_os_thread.h include/xbt/xbt_os_time.h ) set(source_of_generated_headers include/simgrid_config.h.in include/smpi/mpif.h.in) ### depend of some variables setted upper # -->HAVE_THREAD_CONTEXTS HAVE_UCONTEXT_CONTEXTS if(${HAVE_THREAD_CONTEXTS}) #pthread set(SURF_SRC ${SURF_SRC} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp ) else() # NOT pthread set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp ) endif() if(${HAVE_THREAD_CONTEXTS}) #pthread set(SURF_SRC ${SURF_SRC} src/xbt/xbt_os_thread.c) else() # NOT pthread set(EXTRA_DIST ${EXTRA_DIST} src/xbt/xbt_os_thread.c ) endif() if(${HAVE_UCONTEXT_CONTEXTS}) #ucontext set(SURF_SRC ${SURF_SRC} src/kernel/context/ContextUnix.hpp src/kernel/context/ContextUnix.cpp) else() # NOT ucontext set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextUnix.hpp src/kernel/context/ContextUnix.cpp) endif() ### Simgrid Lib sources set(simgrid_sources ${PLUGINS_SRC} ${BINDINGS_SRC} ${MC_SRC_BASE} ${MSG_SRC} ${S4U_SRC} ${SIMDAG_SRC} ${SIMGRID_SRC} ${SIMIX_SRC} ${SURF_SRC} ${TRACING_SRC} ${XBT_SRC} ) if(${SIMGRID_HAVE_JEDULE}) set(simgrid_sources ${simgrid_sources} ${JEDULE_SRC}) else() set(EXTRA_DIST ${EXTRA_DIST} ${JEDULE_SRC}) endif() if(enable_smpi) set(simgrid_sources ${simgrid_sources} ${SMPI_SRC}) endif() if(SIMGRID_HAVE_MC) set(simgrid_sources ${simgrid_sources} ${MC_SRC}) endif() if(SIMGRID_HAVE_NS3) set(simgrid_sources ${simgrid_sources} ${NS3_SRC}) endif() # WINDOWS if(WIN32) set(simgrid_sources ${simgrid_sources} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp src/xbt/xbt_os_thread.c ) endif() if(SIMGRID_HAVE_LUA) set(simgrid_sources ${simgrid_sources} ${LUA_SRC}) else() set(EXTRA_DIST ${EXTRA_DIST} ${LUA_SRC}) endif() set(DOC_SOURCES doc/Doxyfile.in doc/Layout.xml doc/sg_thread_model.fig doc/simix.fig doc/surf_nutshell.fig doc/surf++.png doc/surf++.pdf doc/surf++.graphml doc/surf++.uml doc/triva-graph_configuration.png doc/triva-graph_configuration.svg doc/triva-graph_visualization.png doc/triva-graph_visualization.svg doc/triva-time_interval.png doc/triva-time_interval.svg doc/doxygen/FAQ.doc doc/doxygen/application.doc doc/doxygen/community.doc doc/doxygen/deployment.doc doc/doxygen/examples.doc doc/doxygen/footer.html doc/doxygen/getting_started.doc doc/doxygen/header.html doc/doxygen/howtos.doc doc/doxygen/index.doc doc/doxygen/inside.doc doc/doxygen/inside_tests.doc doc/doxygen/inside_cmake.doc doc/doxygen/inside_doxygen.doc doc/doxygen/inside_extending.doc doc/doxygen/inside_release.doc doc/doxygen/install.doc doc/doxygen/install_yours.doc doc/doxygen/java.doc doc/doxygen/tutorial_msg.doc doc/doxygen/tutorial_smpi.doc doc/doxygen/models.doc doc/doxygen/module-msg.doc doc/doxygen/module-s4u.doc doc/doxygen/module-sd.doc doc/doxygen/module-simix.doc doc/doxygen/module-smpi.doc doc/doxygen/module-surf.doc doc/doxygen/module-trace.doc doc/doxygen/module-xbt.doc doc/doxygen/module-index.doc doc/doxygen/ns3.doc doc/doxygen/options.doc doc/doxygen/outcomes.doc doc/doxygen/outcomes_logs.doc doc/doxygen/outcomes_MC.doc doc/doxygen/outcomes_vizu.doc doc/doxygen/platform.doc doc/doxygen/platform_lua.doc doc/doxygen/scenario.doc doc/doxygen/stylesheet.css doc/doxygen/uhood.doc doc/doxygen/uhood_switch.doc doc/doxygen/uhood_arch.doc doc/manpage/smpicc.1 doc/manpage/smpicxx.1 doc/manpage/smpif90.1 doc/manpage/smpiff.1 doc/manpage/smpirun.1 doc/manpage/tesh.pod doc/msg-tuto-src/deployment0.xml doc/msg-tuto-src/deployment1.xml doc/msg-tuto-src/deployment2.xml doc/msg-tuto-src/deployment3.xml doc/msg-tuto-src/deployment_general.xml doc/msg-tuto-src/Makefile doc/msg-tuto-src/masterworker0.c doc/msg-tuto-src/masterworker1.c doc/msg-tuto-src/masterworker2.c doc/msg-tuto-src/masterworker3.c doc/msg-tuto-src/masterworker4.c CITATION.bib ) set(DOC_FIGS ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules.fig ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules2.fig ) set(DOC_TOOLS tools/doxygen/fig2dev_postprocessor.pl tools/doxygen/xbt_log_extract_hierarchy.pl tools/doxygen/list_routing_models_examples.sh ) # these files get copied automatically to the html documentation set(DOC_IMG ${CMAKE_HOME_DIRECTORY}/doc/sc3-description.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/AS_hierarchy.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/eclipseScreenShot.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot.jpg ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot_thn.jpg ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.gif ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.icns ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.ico ${CMAKE_HOME_DIRECTORY}/doc/webcruft/awstats_logo3.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/output.goal.pdf ${CMAKE_HOME_DIRECTORY}/doc/webcruft/poster_thumbnail.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/storage_sample_scenario.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011.gif ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011_small.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_win.bmp ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_win_2011.bmp ${CMAKE_HOME_DIRECTORY}/doc/webcruft/smpi_simgrid_alltoall_pair_16.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/smpi_simgrid_alltoall_ring_16.png ) set(bin_files ${bin_files} src/smpi/smpicc.in src/smpi/smpicxx.in src/smpi/smpiff.in src/smpi/smpif90.in src/smpi/smpirun.in src/smpi/smpitools.sh ) set(txt_files ${txt_files} AUTHORS COPYING README.md ChangeLog INSTALL LICENSE-LGPL-2.1 NEWS TODO ) # The list of cmake build directories is constructed from the following list. # Add your CMakeLists file here to see your subdir built. set(CMAKEFILES_TXT examples/java/CMakeLists.txt examples/msg/CMakeLists.txt examples/msg/mc/CMakeLists.txt examples/s4u/CMakeLists.txt examples/simdag/CMakeLists.txt examples/smpi/CMakeLists.txt examples/smpi/NAS/CMakeLists.txt examples/smpi/smpi_msg_masterslave/CMakeLists.txt examples/smpi/replay_multiple/CMakeLists.txt examples/smpi/energy/f77/CMakeLists.txt examples/smpi/energy/f90/CMakeLists.txt teshsuite/java/CMakeLists.txt teshsuite/mc/CMakeLists.txt teshsuite/msg/CMakeLists.txt teshsuite/s4u/CMakeLists.txt teshsuite/simdag/CMakeLists.txt teshsuite/simix/CMakeLists.txt teshsuite/smpi/CMakeLists.txt teshsuite/surf/CMakeLists.txt teshsuite/xbt/CMakeLists.txt teshsuite/smpi/isp/umpire/CMakeLists.txt teshsuite/smpi/mpich3-test/CMakeLists.txt teshsuite/smpi/mpich3-test/attr/CMakeLists.txt teshsuite/smpi/mpich3-test/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/comm/CMakeLists.txt teshsuite/smpi/mpich3-test/datatype/CMakeLists.txt # teshsuite/smpi/mpich3-test/f77/attr/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/info/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/comm/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/datatype/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/ext/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/init/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/util/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/topo/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/datatype/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/info/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/init/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/util/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/group/CMakeLists.txt teshsuite/smpi/mpich3-test/info/CMakeLists.txt teshsuite/smpi/mpich3-test/init/CMakeLists.txt teshsuite/smpi/mpich3-test/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/topo/CMakeLists.txt teshsuite/smpi/mpich3-test/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/perf/CMakeLists.txt tools/CMakeLists.txt tools/graphicator/CMakeLists.txt tools/tesh/CMakeLists.txt ) set(CMAKE_SOURCE_FILES CMakeLists.txt FindSimGrid.cmake tools/cmake/Tests.cmake tools/cmake/CTestConfig.cmake tools/cmake/CTestCustom.cmake tools/cmake/DefinePackages.cmake tools/cmake/Distrib.cmake tools/cmake/GCCFlags.cmake tools/cmake/Documentation.cmake tools/cmake/MaintainerMode.cmake tools/cmake/Java.cmake tools/cmake/MakeLib.cmake tools/cmake/MakeLibWin.cmake tools/cmake/Modules/FindGraphviz.cmake tools/cmake/Modules/FindLibdw.cmake tools/cmake/Modules/FindLibelf.cmake tools/cmake/Modules/FindLibunwind.cmake tools/cmake/Modules/FindLibevent.cmake tools/cmake/Modules/FindLuaSimgrid.cmake tools/cmake/Modules/FindNS3.cmake tools/cmake/Modules/FindPAPI.cmake tools/cmake/Modules/FindRngStream.cmake tools/cmake/Modules/FindValgrind.cmake tools/cmake/Option.cmake tools/cmake/scripts/my_valgrind.pl tools/cmake/scripts/update_tesh.pl tools/cmake/UnitTesting.cmake tools/cmake/src/internal_config.h.in tools/cmake/test_prog/prog_asan.cpp tools/cmake/test_prog/prog_gnu_dynlinker.c tools/cmake/test_prog/prog_makecontext.c tools/cmake/test_prog/prog_mutex_timedlock.c tools/cmake/test_prog/prog_sem_init.c tools/cmake/test_prog/prog_sem_open.c tools/cmake/test_prog/prog_sem_timedwait.c tools/cmake/test_prog/prog_snprintf.c tools/cmake/test_prog/prog_stackgrowth.c tools/cmake/test_prog/prog_stacksetup.c tools/cmake/test_prog/prog_thread_storage.c tools/cmake/test_prog/prog_vsnprintf.c tools/cmake/cross-mingw.cmake tools/smpi/generate_smpi_defines.pl tools/stack-cleaner/as tools/stack-cleaner/cc tools/stack-cleaner/c++ tools/stack-cleaner/fortran tools/stack-cleaner/clean-stack-filter tools/stack-cleaner/compiler-wrapper tools/stack-cleaner/README ) set(PLATFORMS_EXAMPLES examples/platforms/bypassASroute.xml examples/platforms/bypassRoute.xml examples/platforms/cloud.xml examples/platforms/cluster.xml examples/platforms/cluster_backbone.xml examples/platforms/cluster_and_one_host.xml examples/platforms/cluster_prototype.lua examples/platforms/cluster_no_backbone.xml examples/platforms/cluster_torus.xml examples/platforms/cluster_fat_tree.xml examples/platforms/cluster_dragonfly.xml examples/platforms/crosstraffic.xml examples/platforms/optorsim/gridpp_grid_2004.conf examples/platforms/optorsim/lcg_sept2004_grid.conf examples/platforms/optorsim/transform_optorsim_platform.pl examples/platforms/config.xml examples/platforms/config_tracing.xml examples/platforms/trace/bourassa_state.trace examples/platforms/trace/fafard_state.trace examples/platforms/trace/faulty_host.trace examples/platforms/trace/ginette_state.trace examples/platforms/trace/jupiter_speed.trace examples/platforms/trace/jupiter_state.trace examples/platforms/trace/link1_avail.trace examples/platforms/trace/link3_state.trace examples/platforms/trace/link4_state.trace examples/platforms/trace/linkBandwidth7.bw examples/platforms/trace/trace_A_failure.txt examples/platforms/trace/trace_A.txt examples/platforms/trace/trace_B.txt examples/platforms/data_center.xml examples/platforms/dogbone.xml examples/platforms/energy_platform.xml examples/platforms/energy_cluster.xml examples/platforms/faulty_host.xml examples/platforms/g5k.xml examples/platforms/griffon.xml examples/platforms/meta_cluster.xml examples/platforms/multicore_machine.xml examples/platforms/onelink.xml examples/platforms/prop.xml examples/platforms/routing_cluster.xml examples/platforms/routing_cluster.lua examples/platforms/routing_none.xml examples/platforms/simulacrum_7_hosts.xml examples/platforms/storage/content/small_content.txt examples/platforms/storage/content/storage_content.txt examples/platforms/storage/content/win_storage_content.txt examples/platforms/storage/remote_io.xml examples/platforms/storage/storage.xml examples/platforms/small_platform.xml examples/platforms/small_platform.lua examples/platforms/small_platform_constant.xml examples/platforms/small_platform_fatpipe.xml examples/platforms/small_platform_one_link_routes.xml examples/platforms/small_platform_with_failures.xml examples/platforms/small_platform_with_routers.xml examples/platforms/syscoord/generate_peer_platform.pl examples/platforms/syscoord/median_harvard.syscoord examples/platforms/syscoord/median_meridian.syscoord examples/platforms/syscoord/median_p2psim.syscoord examples/platforms/three_multicore_hosts.xml examples/platforms/two_hosts.xml examples/platforms/two_hosts_platform_shared.xml examples/platforms/two_hosts_platform_with_availability.xml examples/platforms/two_hosts_platform_with_availability_included.xml examples/platforms/two_peers.xml examples/platforms/vivaldi.xml ) set(generated_src_files src/xbt/automaton/automaton_lexer.yy.c src/xbt/automaton/parserPromela.tab.cacc src/xbt/automaton/parserPromela.tab.hacc ) foreach(file ${generated_src_files}) set_source_files_properties(${file} PROPERTIES GENERATED true) endforeach(file ${generated_src_files}) SimGrid-3.18/tools/cmake/CTestCustom.cmake0000644000175000017500000000007213217757322020756 0ustar mquinsonmquinsonSET(CTEST_CUSTOM_COVERAGE_EXCLUDE @PATTERN_CTEST_IGNORED@)SimGrid-3.18/tools/cmake/src/0000755000175000017500000000000013217757322016327 5ustar mquinsonmquinsonSimGrid-3.18/tools/cmake/src/internal_config.h.in0000644000175000017500000000675313217757322022261 0ustar mquinsonmquinson/* internal_config.h -- characteristics of the platform, as probed by Cmake */ /* Warning: The file internal_config.h is AUTOMATICALLY GENERATED by Cmake. * Edit the template instead: tools/cmake/src/internal_config.h.in */ /* Copyright (c) 2004-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid_config.h" /* what was compiled in? */ /* Non-standard header files */ /* */ #cmakedefine01 HAVE_EXECINFO_H /* */ #cmakedefine01 HAVE_FUTEX_H /* */ #cmakedefine01 HAVE_SIGNAL_H /* */ #cmakedefine01 HAVE_UNISTD_H /* */ #cmakedefine01 HAVE_UCONTEXT_H /* */ #cmakedefine01 HAVE_VALGRIND_H /* Address Sanitizer */ #cmakedefine01 HAVE_SANITIZE_ADDRESS #cmakedefine01 HAVE_SANITIZE_ADDRESS_FIBER_SUPPORT /* Time portability */ /* Function gettimeofday */ #cmakedefine01 HAVE_GETTIMEOFDAY /* Function clock_gettime */ #cmakedefine01 HAVE_POSIX_GETTIME /* Function nanosleep */ #cmakedefine01 HAVE_NANOSLEEP /* The usable context factories */ #define HAVE_BOOST_CONTEXTS @HAVE_BOOST_CONTEXTS@ #cmakedefine01 HAVE_RAW_CONTEXTS #cmakedefine01 HAVE_THREAD_CONTEXTS #cmakedefine01 HAVE_UCONTEXT_CONTEXTS /* Variables for the thread contexts (and parallel mode of raw contexts) */ /* Define to 1 if threads are usable . */ #cmakedefine01 HAVE_PTHREAD /* Does not seems defined on Mac nor Windows */ #cmakedefine01 HAVE_PTHREAD_SETAFFINITY /* If __thread is available */ #cmakedefine01 HAVE_THREAD_LOCAL_STORAGE /* Variables for the raw contexts (to select the right assembly code) */ #cmakedefine01 SIMGRID_PROCESSOR_i686 #cmakedefine01 SIMGRID_PROCESSOR_x86_64 /* Variables for the SysV contexts */ @sg_makecontext_stack_addr@ @sg_makecontext_stack_size@ /* Variable for SysV, raw and Boost contexts */ /* Does the stack growth upward, or downward? */ #define PTH_STACKGROWTH @PTH_STACKGROWTH@ /* MC variables */ /* getdtablesize: get descriptor table size */ #cmakedefine01 HAVE_GETDTABLESIZE /* Did we compile mmalloc in? */ #cmakedefine01 HAVE_MMALLOC /* process_vm_readv: transfer data between process address spaces */ #cmakedefine01 HAVE_PROCESS_VM_READV /* Set to true if enable_model-checking is true and the dependencies available */ #cmakedefine01 SIMGRID_HAVE_MC /* SMPI variables */ /* SMPI enabled */ #cmakedefine01 HAVE_SMPI /* Fortran language is available for SMPI */ #cmakedefine01 SMPI_FORTRAN /* We have mmap and objdump to handle privatization */ #cmakedefine01 HAVE_PRIVATIZATION /* We have PAPI to fine-grain trace execution time */ #cmakedefine01 HAVE_PAPI /* We have sendfile to efficiently copy files for dl-open privatization */ #cmakedefine01 HAVE_SENDFILE /* Other function checks */ /* Function backtrace */ #cmakedefine01 HAVE_BACKTRACE /* Function dlfunc */ #cmakedefine01 HAVE_DLFUNC /* Function mmap */ #cmakedefine01 HAVE_MMAP /* Function mremap */ #cmakedefine01 HAVE_MREMAP /* Function sem_init (part of XPG6 standard only) */ #cmakedefine01 HAVE_SEM_INIT /* Function popen */ #cmakedefine01 HAVE_POPEN /* Function sysconf */ #cmakedefine01 HAVE_SYSCONF /* Function vasprintf */ #cmakedefine01 HAVE_VASPRINTF /* Other checks */ /* Path to the addr2line tool */ #cmakedefine ADDR2LINE "@ADDR2LINE@" /* The graphviz library */ #cmakedefine01 HAVE_GRAPHVIZ /* The lib unwind library (for MC and backtrace display) */ #cmakedefine01 HAVE_LIBUNWIND SimGrid-3.18/tools/cmake/Modules/0000755000175000017500000000000013217757322017150 5ustar mquinsonmquinsonSimGrid-3.18/tools/cmake/Modules/FindNS3.cmake0000644000175000017500000000617613217757322021370 0ustar mquinsonmquinson# Try to find the NS3 library. # # The user can hint a path using the NS3_HINT option. # # Once done, the following will be defined: # # NS3_INCLUDE_DIRS - the NS3 include directories # NS3_LIBRARY_PATH - path to the libs # NS3_LIBRARIES - link these to use NS3 (full path to libs) # # This could be improved in many ways (patches welcome): # - No way to specify a minimal version (v3.10 is hardcoded). # - No proper find_package() integration ## ADDING A NS3 VERSION. # - Add "ns3.${version}-core ns3.${version}-core-debug ns3.${version}-core-optimized" to the NAME line of the find_library below # - Add "include/ns3{version}" to the PATH_SUFFIXES line of the find_path below set(SIMGRID_HAVE_NS3 0) set(NS3_HINT ${ns3_path} CACHE PATH "Path to search for NS3 lib and include") find_library(NS3_LIBRARIES NAME ns3-core ns3.14-core ns3.15-core ns3.16-core ns3.17-core ns3.18-core ns3.19-core ns3.20-core ns3.21-core ns3.22-core ns3.25-core ns3.26-core ns3.26-core-optimized ns3.26-core-debug ns3.27-core ns3.27-core-optimized ns3.27-core-debug ns3.28-core ns3.28-core-optimized ns3.28-core-debug PATH_SUFFIXES lib64 lib ns3/lib PATHS ${NS3_HINT} ) find_path(NS3_INCLUDE_DIR NAME ns3/core-module.h PATH_SUFFIXES include ns3/include include/ns3.14 include/ns3.15 include/ns3.16 include/ns3.17 include/ns3.18 include/ns3.19 include/ns3.20 include/ns3.21 include/ns3.22 include/ns3.25 include/ns3.26 include/ns3.27 include/ns3.28 PATHS ${NS3_HINT} ) if(NS3_INCLUDE_DIR) message(STATUS "Looking for ns3/core-module.h - found") else() message(STATUS "Looking for ns3/core-module.h - not found") endif() mark_as_advanced(NS3_INCLUDE_DIR) message(STATUS "Looking for lib ns3-core") if(NS3_LIBRARIES) message(STATUS "Looking for lib ns3-core - found") else() message(STATUS "Looking for lib ns3-core - not found") endif() mark_as_advanced(NS3_LIBRARIES) if(NS3_INCLUDE_DIR) if(NS3_LIBRARIES) set(SIMGRID_HAVE_NS3 1) if(NS3_LIBRARIES MATCHES "-optimized") set (NS3_SUFFIX "-optimized") elseif(NS3_LIBRARIES MATCHES "-debug") set (NS3_SUFFIX "-debug") else() set (NS3_SUFFIX "") endif() string(REGEX REPLACE ".*ns([.0-9]+)-core.*" "\\1" NS3_VERSION "${NS3_LIBRARIES}") get_filename_component(NS3_LIBRARY_PATH "${NS3_LIBRARIES}" PATH) # Compute NS3_PATH string(REGEX REPLACE "(.*)/lib" "\\1" NS3_PATH "${NS3_LIBRARY_PATH}") message(STATUS "NS-3 found (v3.${NS3_VERSION} in ${NS3_PATH}).") if (NOT NS3_LIBRARY_PATH STREQUAL "/usr/lib") string(REGEX MATCH "${NS3_LIBRARY_PATH}" MatchResult "$ENV{LD_LIBRARY_PATH}") if(NOT MatchResult) message(STATUS "Warning: NS3 not installed in system path, and not listed in LD_LIBRARY_PATH." " You want to: export LD_LIBRARY_PATH=${NS3_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:\$LD_LIBRARY_PATH}") endif() endif() endif() endif() mark_as_advanced(NS3_LIBRARY_PATH) if(SIMGRID_HAVE_NS3) link_directories(${NS3_LIBRARY_PATH}) include_directories(${NS3_INCLUDE_DIR}) else() message(STATUS "Warning: To use NS-3 Please install ns3 at least version 3.10 (http://www.nsnam.org/releases/)") endif() SimGrid-3.18/tools/cmake/Modules/FindLuaSimgrid.cmake0000644000175000017500000000472213217757322023020 0ustar mquinsonmquinson# Search for the Lua 5.3 include files and libraries # # Input variable: # LUA_HINT: path to Lua installation -- only needed for non-standard installs # Output variable: # SIMGRID_HAVE_LUA : if Lua was found # LUA_LIBRARY : the path to the dynamic library # LUA_INCLUDE_DIR : where to find lua.h # LUA_VERSION_MAJOR: First part of the version (often, 5) # LUA_VERSION_MINOR: Second part of the version (3 when we have 5.3) find_path(LUA_INCLUDE_DIR lua.h HINTS ENV LUA_HINT PATH_SUFFIXES include/lua53 include/lua5.3 include/lua-5.3 include/lua include PATHS ~/Library/Frameworks /Library/Frameworks /sw # Fink /opt/local # DarwinPorts /opt/csw # Blastwave /opt ) find_library(LUA_LIBRARY NAMES # I don't want a lua.a lua53.so lua5.3.so lua-5.3.so lua53.dynlib lua5.3.dynlib lua-5.3.dynlib lua53.dll lua5.3.dll lua-5.3.dll lua.so lua.dynlib lua.dll lua53 lua5.3 lua-5.3 lua HINTS ENV LUA_HINT PATH_SUFFIXES lib PATHS ~/Library/Frameworks /Library/Frameworks /sw /opt/local /opt/csw /opt ) if (NOT LUA_LIBRARY) message(FATAL_ERROR "Error: Lua library not found. Please install that package (and set LUA_HINT) or disable Lua.") endif() if (NOT LUA_INCLUDE_DIR OR NOT EXISTS "${LUA_INCLUDE_DIR}/lua.h") message(FATAL_ERROR "Error: Lua header file not found. Please install that package (and set LUA_HINT) or disable Lua.") endif() # Extract the version info out of the header file file(STRINGS "${LUA_INCLUDE_DIR}/lua.h" lua_version_str REGEX "^#define[ \t]+LUA_VERSION_MAJOR[ \t]+\"[456]+\"") string(REGEX REPLACE "^#define[ \t]+LUA_VERSION_MAJOR[ \t]+\"([^\"]+)\"" "\\1" LUA_VERSION_MAJOR "${lua_version_str}") file(STRINGS "${LUA_INCLUDE_DIR}/lua.h" lua_version_str REGEX "^#define[ \t]+LUA_VERSION_MINOR[ \t]+\"[0123456789]+\"") string(REGEX REPLACE "^#define[ \t]+LUA_VERSION_MINOR[ \t]+\"([^\"]+)\"" "\\1" LUA_VERSION_MINOR "${lua_version_str}") unset(lua_version_str) # Check that we have a sufficient version of Lua if(LUA_VERSION_MAJOR EQUAL 5 AND LUA_VERSION_MINOR EQUAL 3) set(SIMGRID_HAVE_LUA 1) include_directories(${LUA_INCLUDE_DIR}) else() message(FATAL_ERROR "Error: Lua version 5.3 is required, but version ${LUA_VERSION_MAJOR}.${LUA_VERSION_MINOR} found instead.") endif() message(STATUS "Lua version: ${LUA_VERSION_MAJOR}.${LUA_VERSION_MINOR}") message(STATUS "Lua library: ${LUA_LIBRARY}") mark_as_advanced(LUA_INCLUDE_DIR) mark_as_advanced(LUA_LIBRARY) SimGrid-3.18/tools/cmake/Modules/FindGraphviz.cmake0000644000175000017500000000234513217757322022551 0ustar mquinsonmquinsonfind_path(HAVE_CGRAPH_H cgraph.h HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES include/graphviz include PATHS /opt;/opt/local;/opt/csw;/sw;/usr ) find_library(HAVE_CGRAPH_LIB NAME cgraph HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES lib/graphviz lib PATHS /opt;/opt/local;/opt/csw;/sw;/usr) find_library(HAVE_CDT_LIB NAME cdt HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES lib/graphviz lib PATHS /opt;/opt/local;/opt/csw;/sw;/usr) if(HAVE_CDT_LIB AND HAVE_CGRAPH_LIB AND HAVE_CGRAPH_H) string(REGEX REPLACE "/libcgraph.*" "" lib_graphviz ${HAVE_CGRAPH_LIB}) string(REPLACE "/graphviz/cgraph.h" "" file_graphviz_h ${HAVE_CGRAPH_H}) string(REPLACE "/graphviz" "" file_graphviz_h ${file_graphviz_h}) include_directories(${file_graphviz_h} ${file_graphviz_h}/graphviz) link_directories(${lib_graphviz}) set(HAVE_GRAPHVIZ "1") else() set(HAVE_GRAPHVIZ "0") endif() mark_as_advanced(HAVE_GRAPHVIZ) unset(HAVE_CGRAPH_H) unset(HAVE_CGRAPH_LIB) unset(HAVE_CDT_LIB) message(STATUS "Looking for graphviz") if(HAVE_GRAPHVIZ) message(STATUS "Looking for graphviz - found") else() message(STATUS "Looking for graphviz - not found (try installing libgraphviz-dev)") endif() SimGrid-3.18/tools/cmake/Modules/FindValgrind.cmake0000644000175000017500000000213013217757322022515 0ustar mquinsonmquinsonfind_program(VALGRIND_EXE NAME valgrind PATH_SUFFIXES bin/ PATHS /opt /opt/local /opt/csw /sw /usr ) mark_as_advanced(VALGRIND_EXE) if(enable_memcheck) if (NOT VALGRIND_EXE MATCHES "NOTFOUND") execute_process(COMMAND ${VALGRIND_EXE} --version OUTPUT_VARIABLE "VALGRIND_VERSION") message(STATUS "Valgrind version: ${VALGRIND_VERSION}") set(TESH_WRAPPER ${CMAKE_HOME_DIRECTORY}/tools/cmake/scripts/my_valgrind.pl) set(TESH_WRAPPER ${TESH_WRAPPER}\ --trace-children=yes\ --trace-children-skip=/usr/bin/*,/bin/*\ --leak-check=full\ --show-reachable=yes\ --track-origins=no\ --read-var-info=no\ --num-callers=20\ --suppressions=${CMAKE_HOME_DIRECTORY}/tools/simgrid.supp\ ) if(enable_memcheck_xml) SET(TESH_WRAPPER ${TESH_WRAPPER}\ --xml=yes\ --xml-file=memcheck_test_%p.memcheck\ --child-silent-after-fork=yes\ ) endif() # message(STATUS "tesh wrapper: ${TESH_WRAPPER}") mark_as_advanced(TESH_WRAPPER) else() set(enable_memcheck false) message(STATUS "Error: Command valgrind not found --> enable_memcheck autoset to false.") endif() endif() SimGrid-3.18/tools/cmake/Modules/FindRngStream.cmake0000644000175000017500000000225513217757322022661 0ustar mquinsonmquinsonfind_path(HAVE_RNGSTREAM_H NAME RngStream.h HINTS $ENV{HOME} PATH_SUFFIXES include PATHS /opt /opt/local /opt/csw /sw /usr ) find_library(HAVE_RNGSTREAM_LIB NAME rngstreams HINTS $ENV{HOME} PATH_SUFFIXES lib64 lib lib32 PATHS /opt /opt/local /opt/csw /sw /usr ) message(STATUS "Looking for RngStream.h") if(HAVE_RNGSTREAM_H) message(STATUS "Looking for RngStream.h - found") string(REGEX MATCH "-I${HAVE_RNGSTREAM_H} " operation "${CMAKE_C_FLAGS}") if(NOT operation) SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}-I${HAVE_RNGSTREAM_H} ") endif() else() message(STATUS "Looking for RngStream.h - not found") endif() message(STATUS "Looking for lib rngstreams") if(HAVE_RNGSTREAM_LIB) message(STATUS "Looking for lib rngstreams - found") string(REGEX REPLACE "/librngstreams.*" "" HAVE_RNGSTREAM_LIB "${HAVE_RNGSTREAM_LIB}") string(REGEX MATCH "-L${HAVE_RNGSTREAM_LIB} " operation "${CMAKE_C_FLAGS}") if(NOT operation) SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}-L${HAVE_RNGSTREAM_LIB} ") endif() else() message(STATUS "Looking for lib rngstreams - not found") endif() mark_as_advanced(HAVE_RNGSTREAM_LIB) mark_as_advanced(HAVE_RNGSTREAM_H)SimGrid-3.18/tools/cmake/Modules/FindLibunwind.cmake0000644000175000017500000000336613217757322022716 0ustar mquinsonmquinsonif(SIMGRID_PROCESSOR_x86_64) find_library(PATH_LIBUNWIND_LIB NAMES unwind-x86_64 HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib lib/system PATHS /opt /opt/local /opt/csw /sw /usr) endif() if(NOT PATH_LIBUNWIND_LIB) find_library(PATH_LIBUNWIND_LIB NAMES unwind HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib lib/system PATHS /opt /opt/local /opt/csw /sw /usr /usr/lib/) endif() find_path(PATH_LIBUNWIND_H "libunwind.h" HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES include/ GnuWin32/include PATHS /opt /opt/local /opt/csw /sw /usr) message(STATUS "Looking for libunwind.h") if(PATH_LIBUNWIND_H) message(STATUS "Looking for libunwind.h - found") else() message(STATUS "Looking for libunwind.h - not found") endif() message(STATUS "Looking for libunwind") if(PATH_LIBUNWIND_LIB) message(STATUS "Looking for libunwind - found") else() message(STATUS "Looking for libunwind - not found") endif() if(PATH_LIBUNWIND_LIB AND PATH_LIBUNWIND_H) string(REGEX REPLACE "/libunwind.*[.]${LIB_EXE}$" "" PATH_LIBUNWIND_LIB "${PATH_LIBUNWIND_LIB}") string(REGEX REPLACE "/libunwind.h" "" PATH_LIBUNWIND_H "${PATH_LIBUNWIND_H}") include_directories(${PATH_LIBUNWIND_H}) link_directories(${PATH_LIBUNWIND_LIB}) SET(HAVE_LIBUNWIND 1) else() SET(HAVE_LIBUNWIND 0) endif() mark_as_advanced(PATH_LIBDW_H) mark_as_advanced(PATH_LIBDW_LIB) mark_as_advanced(PATH_LIBUNWIND_LIB) mark_as_advanced(PATH_LIBUNWIND_H) SimGrid-3.18/tools/cmake/Modules/FindPAPI.cmake0000644000175000017500000000360213217757322021505 0ustar mquinsonmquinson# Try to find PAPI headers and LIBRARY. # # Usage of this module as follows: # # find_package(PAPI) # # Variables used by this module, they can change the default behaviour and need # to be set before calling find_package: # # PAPI_PREFIX Set this variable to the root installation of # libpapi if the module has problems finding the # proper installation path. # # Variables defined by this module: # # PAPI_FOUND System has PAPI LIBRARY and headers # PAPI_LIBRARY The PAPI library # PAPI_INCLUDE_DIRS The location of PAPI headers set(HAVE_PAPI 0) set(PAPI_HINT ${papi_path} CACHE PATH "Path to search for PAPI headers and library") find_path(PAPI_PREFIX NAMES include/papi.h PATHS ${PAPI_HINT} ) message(STATUS "Looking for libpapi") find_library(PAPI_LIBRARY NAMES libpapi papi PATH_SUFFIXES lib64 lib # HINTS gets searched before PATHS HINTS ${PAPI_PREFIX}/lib ) if(PAPI_LIBRARY) message(STATUS "Looking for libpapi - found at ${PAPI_LIBRARY}") else() message(STATUS "Looking for libpapi - not found") endif() message(STATUS "Looking for papi.h") find_path(PAPI_INCLUDE_DIRS NAMES papi.h # HINTS gets searched before PATHS HINTS ${PAPI_PREFIX}/include ) if(PAPI_INCLUDE_DIRS) message(STATUS "Looking for papi.h - found at ${PAPI_INCLUDE_DIRS}") else() message(STATUS "Looking for papi.h - not found") endif() if (PAPI_LIBRARY) if(PAPI_INCLUDE_DIRS) set(HAVE_PAPI 1) mark_as_advanced(HAVE_PAPI) endif() endif() if(NOT HAVE_PAPI) message(FATAL_ERROR, "Could not find PAPI LIBRARY and/or papi.h. Make sure they are correctly installed!") endif() #include(FindPackageHandleStandardArgs) #find_package_handle_standard_args(PAPI DEFAULT_MSG # PAPI_LIBRARY # PAPI_INCLUDE_DIRS #) mark_as_advanced( PAPI_PREFIX_DIRS PAPI_LIBRARY PAPI_INCLUDE_DIRS ) SimGrid-3.18/tools/cmake/Modules/FindLibevent.cmake0000644000175000017500000000062713217757322022530 0ustar mquinsonmquinsonfind_path(LIBEVENT_INCLUDE_DIR event2/event.h) find_library(LIBEVENT_LIBRARY NAMES event) find_library(LIBEVENT_THREADS_LIBRARY NAMES event_pthreads) set(LIBEVENT_LIBRARIES "${LIBEVENT_LIBRARY}") include(FindPackageHandleStandardArgs) find_package_handle_standard_args( Libevent DEFAULT_MSG LIBEVENT_LIBRARIES LIBEVENT_INCLUDE_DIR) mark_as_advanced(LIBEVENT_INCLUDE_DIR LIBEVENT_LIBRARIES) SimGrid-3.18/tools/cmake/Modules/FindLibelf.cmake0000644000175000017500000000126113217757322022150 0ustar mquinsonmquinsonfind_path(LIBELF_INCLUDE_DIR "libelf.h" HINTS $ENV{SIMGRID_LIBELF_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBELF_LIBRARY_PATH} PATH_SUFFIXES include/ GnuWin32/include PATHS /opt /opt/local /opt/csw /sw /usr) find_library(LIBELF_LIBRARY NAMES elf HINTS $ENV{SIMGRID_LIBELF_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBELF_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib PATHS /opt /opt/local /opt/csw /sw /usr) set(LIBELF_LIBRARIES "${LIBELF_LIBRARY}") include(FindPackageHandleStandardArgs) find_package_handle_standard_args( Libelf DEFAULT_MSG LIBELF_LIBRARIES LIBELF_INCLUDE_DIR) mark_as_advanced(LIBELF_INCLUDE_DIR LIBELF_LIBRARIES) SimGrid-3.18/tools/cmake/Modules/FindLibdw.cmake0000644000175000017500000000125313217757322022015 0ustar mquinsonmquinsonfind_path(LIBDW_INCLUDE_DIR "elfutils/libdw.h" HINTS $ENV{SIMGRID_LIBDW_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBDW_LIBRARY_PATH} PATH_SUFFIXES include/ GnuWin32/include PATHS /opt /opt/local /opt/csw /sw /usr) find_library(LIBDW_LIBRARY NAMES dw HINTS $ENV{SIMGRID_LIBDW_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBDW_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib PATHS /opt /opt/local /opt/csw /sw /usr) set(LIBDW_LIBRARIES "${LIBDW_LIBRARY}") include(FindPackageHandleStandardArgs) find_package_handle_standard_args( Libdw DEFAULT_MSG LIBDW_LIBRARIES LIBDW_INCLUDE_DIR) mark_as_advanced(LIBDW_INCLUDE_DIR LIBDW_LIBRARIES) SimGrid-3.18/tools/cmake/Documentation.cmake0000644000175000017500000001652313217757322021362 0ustar mquinsonmquinson### ### Generate all parts of the documentation on non-Windows systems ### ### - HTML with doxygen (reference and manual) ### - Javadoc (reference) ### - manpages (reference of tools) ### ### This file is not loaded on windows #### Generate the html documentation if (enable_documentation) find_package(Doxygen REQUIRED) else() find_package(Doxygen) endif() find_path(FIG2DEV_PATH NAMES fig2dev PATHS NO_DEFAULT_PATHS) if(enable_documentation) ADD_CUSTOM_TARGET(documentation COMMENT "Generating the SimGrid documentation..." DEPENDS ${DOC_SOURCES} ${DOC_FIGS} ${source_doxygen} COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_HOME_DIRECTORY}/doc/html COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/doc/html WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) message(STATUS "Doxygen version: ${DOXYGEN_VERSION}") # This is a workaround for older cmake versions (such as 2.8.7 on Ubuntu 12.04). These cmake versions do not provide # the DOXYGEN_VERSION variable and hence, building the documentation will always fail. This code is the same as used # in the cmake library, version 3. if(DOXYGEN_EXECUTABLE) execute_process(COMMAND ${DOXYGEN_EXECUTABLE} "--version" OUTPUT_VARIABLE DOXYGEN_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE) endif() if(DOXYGEN_VERSION VERSION_LESS "1.8") ADD_CUSTOM_TARGET(error_doxygen COMMAND ${CMAKE_COMMAND} -E echo "Doxygen must be at least version 1.8 to generate documentation. Version found: ${DOXYGEN_VERSION}" COMMAND false ) add_dependencies(documentation error_doxygen) endif() configure_file(${CMAKE_HOME_DIRECTORY}/doc/Doxyfile.in ${CMAKE_HOME_DIRECTORY}/doc/Doxyfile @ONLY) foreach(file ${DOC_FIGS}) string(REPLACE ".fig" ".png" tmp_file ${file}) string(REPLACE "${CMAKE_HOME_DIRECTORY}/doc/shared/fig/" "${CMAKE_HOME_DIRECTORY}/doc/html/" tmp_file ${tmp_file}) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${FIG2DEV_PATH}/fig2dev -Lpng -S 4 ${file} ${tmp_file}) endforeach() foreach(file ${DOC_IMG}) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${CMAKE_COMMAND} -E copy ${file} ${CMAKE_HOME_DIRECTORY}/doc/html/) endforeach() ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${FIG2DEV_PATH}/fig2dev -Lmap ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules.fig | perl -pe 's/imagemap/simgrid_modules/g'| perl -pe 's/ ${CMAKE_HOME_DIRECTORY}/doc/simgrid_modules.map COMMAND pwd COMMAND ${CMAKE_COMMAND} -E echo "XX Generate the index files" COMMAND ${CMAKE_COMMAND} -E remove ${CMAKE_HOME_DIRECTORY}/doc/doxygen/logcategories.doc COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/xbt_log_extract_hierarchy.pl > ${CMAKE_HOME_DIRECTORY}/doc/doxygen/logcategories.doc COMMAND ${CMAKE_COMMAND} -E echo "XX Generate list of files in examples/ for routing models" COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/doc/example_lists/ COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh Floyd > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_floyd COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh Dijkstra > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_dijkstra COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh DijkstraCache > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_dijkstra_cache COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="None"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_none COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Cluster"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_cluster COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Vivaldi"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_vivaldi COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Full"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_full COMMAND ${CMAKE_COMMAND} -E echo "XX Generate list of files in examples/ for XML tags" COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh ' ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_xmltag_mount COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh ' ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_xmltag_linkctn COMMAND ${CMAKE_COMMAND} -E echo "XX Run doxygen" COMMAND ${DOXYGEN_EXECUTABLE} Doxyfile COMMAND ${CMAKE_COMMAND} -E remove ${CMAKE_HOME_DIRECTORY}/doc/simgrid_modules.map WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) ### Fill in the "make gforge-gforge" target ### set(RSYNC_CMD rsync --verbose --cvs-exclude --compress --delete --delete-excluded --rsh=ssh --ignore-times --recursive --links --times --omit-dir-times --perms --chmod=a+rX,ug+w,o-w,Dg+s) add_custom_target(gforge-sync COMMAND ssh scm.gforge.inria.fr mkdir -p -m 2775 /home/groups/simgrid/htdocs/simgrid/${release_version}/ || true COMMAND ${RSYNC_CMD} doc/html/ scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/doc/ || true COMMAND ${RSYNC_CMD} doc/html/simgrid_modules2.png doc/html/simgrid_modules.png doc/webcruft/simgrid_logo_2011.png doc/webcruft/simgrid_logo_2011_small.png scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/ COMMAND ${RSYNC_CMD} src/surf/xml/simgrid.dtd scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/ COMMAND ${RSYNC_CMD} src/surf/xml/simgrid.dtd scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/simgrid.dtd WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}" ) add_dependencies(gforge-sync documentation) else(enable_documentation) ADD_CUSTOM_TARGET(documentation COMMENT "The generation of the SimGrid documentation was disabled in cmake" ) endif(enable_documentation) if (Java_FOUND) find_path(JAVADOC_PATH NAMES javadoc PATHS NO_DEFAULT_PATHS) mark_as_advanced(JAVADOC_PATH) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${CMAKE_COMMAND} -E echo "XX Javadoc pass" COMMAND ${JAVADOC_PATH}/javadoc -quiet -d ${CMAKE_HOME_DIRECTORY}/doc/html/javadoc/ ${CMAKE_HOME_DIRECTORY}/src/bindings/java/org/simgrid/*.java ${CMAKE_HOME_DIRECTORY}/src/bindings/java/org/simgrid/*/*.java WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) endif() #### Generate the manpages if( NOT MANPAGE_DIR) set( MANPAGE_DIR ${CMAKE_BINARY_DIR}/manpages ) endif() add_custom_target(manpages ALL COMMAND ${CMAKE_COMMAND} -E make_directory ${MANPAGE_DIR} COMMAND pod2man ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl > ${MANPAGE_DIR}/simgrid_update_xml.1 COMMAND pod2man ${CMAKE_HOME_DIRECTORY}/doc/manpage/tesh.pod > ${MANPAGE_DIR}/tesh.1 COMMENT "Generating manpages" ) install(FILES ${MANPAGE_DIR}/simgrid_update_xml.1 ${MANPAGE_DIR}/tesh.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpicc.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpicxx.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpif90.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpiff.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpirun.1 DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/share/man/man1) SimGrid-3.18/tools/cmake/Option.cmake0000644000175000017500000001006113217757322020010 0ustar mquinsonmquinson### ARGs use -D[var]=[ON/OFF] or [1/0] or [true/false](see below) ### ex: cmake -Denable_java=ON -Denable_ns3=ON ./ set(BIBTEX2HTML ${BIBTEX2HTML} CACHE PATH "Path to bibtex2html") if(NOT CMAKE_INSTALL_PREFIX) set(CMAKE_INSTALL_PREFIX "/usr/local/simgrid/" CACHE PATH "Path where this project should be installed") else() set(CMAKE_INSTALL_PREFIX ${CMAKE_INSTALL_PREFIX} CACHE PATH "Path where this project should be installed") endif() option(release "Whether Release Mode is activated (disable tests on experimental parts)" on) # How to build ### option(enable_compile_optimizations "Whether to produce efficient code for the SimGrid library" on) option(enable_compile_warnings "Whether compilation warnings should be turned into errors." off) option(enable_lto "Whether we should try to activate the LTO (link time optimisation)" on) option(enable_mallocators "Enable mallocators (disable only for debugging purpose)." on) option(enable_maintainer_mode "Whether flex and flexml files should be rebuilt." off) option(enable_debug "Turn this off to remove all debug messages at compile time (faster, but no debug activatable)" on) # Optional modules ### option(enable_documentation "Whether to produce documentation" on) option(enable_ns3 "Whether ns3 model is activated." off) option(enable_java "Whether the Java bindings are activated." off) option(enable_lib_in_jar "Whether the native libraries are bundled in a Java jar file" on) option(enable_lua "Whether the Lua bindings are activated." off) option(enable_model-checking "Turn this on to experiment with our prototype of model-checker (hinders the simulation's performance even if turned off at runtime)" off) option(enable_jedule "Jedule output of SimDAG." off) if(WIN32) option(enable_smpi "Whether SMPI in included in library." off) option(enable_smpi_MPICH3_testsuite "Whether the test suite form MPICH 3 should be built" off) else() option(enable_smpi "Whether SMPI in included in library." on) # PAPI does not support windows (they did in 3.7, but not anymore in 5.x) # See http://icl.cs.utk.edu/papi/custom/index.html?lid=62&slid=96 option(enable_smpi_papi "Whether SMPI supports PAPI bindings." off) option(enable_smpi_MPICH3_testsuite "Whether the test suite form MPICH 3 should be built" off) endif() option(enable_smpi_ISP_testsuite "Whether the test suite from ISP should be built." off) # Internal targets used by jenkins ### option(enable_fortran "Whether fortran is used with SMPI. Turned on by default if gfortran is found." on) option(enable_coverage "Whether coverage should be enabled." off) mark_as_advanced(enable_coverage) option(enable_memcheck "Enable memcheck." off) mark_as_advanced(enable_memcheck) option(enable_memcheck_xml "Enable memcheck with xml output." off) mark_as_advanced(enable_memcheck_xml) option(enable_address_sanitizer "Whether address sanitizer is turned on." off) mark_as_advanced(enable_address_sanitizer) option(enable_thread_sanitizer "Whether thread sanitizer is turned on." off) mark_as_advanced(enable_thread_sanitizer) option(enable_undefined_sanitizer "Whether undefined sanitizer is turned on." off) mark_as_advanced(enable_undefined_sanitizer) # Cmake, Y U NO hide your garbage?? ### mark_as_advanced(HAVE_SSH) mark_as_advanced(HAVE_RSYNC) mark_as_advanced(BIBTEX2HTML_PATH) mark_as_advanced(BUILDNAME) mark_as_advanced(ADDR2LINE) mark_as_advanced(BIBTOOL_PATH) mark_as_advanced(BUILD_TESTING) mark_as_advanced(CMAKE_BUILD_TYPE) mark_as_advanced(DART_ROOT) mark_as_advanced(DOXYGEN_PATH) mark_as_advanced(FIG2DEV_PATH) mark_as_advanced(FLEXML_EXE) mark_as_advanced(FLEX_EXE) mark_as_advanced(GCOV_PATH) mark_as_advanced(ICONV_PATH) mark_as_advanced(MAKE_PATH) mark_as_advanced(SVN) mark_as_advanced(GIT) mark_as_advanced(CMAKE_OSX_ARCHITECTURES) mark_as_advanced(CMAKE_OSX_DEPLOYMENT_TARGET) mark_as_advanced(CMAKE_OSX_SYSROOT) mark_as_advanced(SED_EXE) mark_as_advanced(BIBTEX2HTML) mark_as_advanced(CMAKE_C_LINK_FLAGS) mark_as_advanced(CMAKE_CXX_FLAGS) mark_as_advanced(CMAKE_Fortran_LINK_FLAGS) SimGrid-3.18/tools/cmake/UnitTesting.cmake0000644000175000017500000000347013217757322021023 0ustar mquinsonmquinson# This file is in charge of the unit testing in SimGrid. # See http://simgrid.gforge.inria.fr/simgrid/3.13/doc/inside_tests.html#inside_tests_add_units # To register a file containing unit tests, simply add it to # FILES_CONTAINING_UNITTESTS and have a pleasant day. set(FILES_CONTAINING_UNITTESTS src/xbt/cunit.cpp src/xbt/ex.cpp src/xbt/dynar.cpp src/xbt/dict.cpp src/xbt/xbt_str.cpp src/xbt/config.cpp ) if(SIMGRID_HAVE_MC) set(FILES_CONTAINING_UNITTESTS ${FILES_CONTAINING_UNITTESTS} src/mc/PageStore.cpp src/mc/mc_snapshot.cpp ) endif() #### Nothing to change below this line to add a new tested file ################################################################ foreach(file ${FILES_CONTAINING_UNITTESTS}) get_filename_component(basename ${file} NAME_WE) get_filename_component(ext ${file} EXT) set(EXTRACTED_TEST_SOURCE_FILES ${EXTRACTED_TEST_SOURCE_FILES} ${CMAKE_CURRENT_BINARY_DIR}/src/${basename}_unit${ext}) endforeach() set(EXTRACTED_TEST_SOURCE_FILES ${EXTRACTED_TEST_SOURCE_FILES} ${CMAKE_CURRENT_BINARY_DIR}/src/simgrid_units_main.c) set_source_files_properties(${EXTRACTED_TEST_SOURCE_FILES} PROPERTIES GENERATED true) ADD_CUSTOM_COMMAND( OUTPUT ${EXTRACTED_TEST_SOURCE_FILES} DEPENDS ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl ${FILES_CONTAINING_UNITTESTS} COMMENT "Generating *_units files for testall..." WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY} COMMAND ${CMAKE_COMMAND} -E remove -f ${EXTRACTED_TEST_SOURCE_FILES} COMMAND chmod +x ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl COMMAND ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl --root=src/ --outdir=${CMAKE_CURRENT_BINARY_DIR}/src/ ${FILES_CONTAINING_UNITTESTS} ) add_executable (testall ${EXTRACTED_TEST_SOURCE_FILES}) target_link_libraries(testall simgrid) SimGrid-3.18/tools/cmake/Java.cmake0000644000175000017500000001547513217757322017437 0ustar mquinsonmquinson## ## The Cmake definitions for the use of Java (and Scala) ## This file is loaded only if the Java option is activated ## find_package(Java 1.7 COMPONENTS Runtime Development) if (NOT ${Java_FOUND}) message(FATAL_ERROR "Java not found (need at least Java7). Please install the JDK or disable that option") endif() set(Java_FOUND 1) include(UseJava) find_package(JNI REQUIRED) message(STATUS "[Java] JNI found: ${JNI_FOUND}") message(STATUS "[Java] JNI include dirs: ${JNI_INCLUDE_DIRS}") if(WIN32) execute_process(COMMAND java -d64 -version OUTPUT_VARIABLE JVM_IS_64_BITS) if("${JVM_IS_64_BITS}" MATCHES "Error") message(fatal_error "SimGrid can only use Java 64 bits") endif() endif() # Rules to build libsimgrid-java ################################ add_library(simgrid-java SHARED ${JMSG_C_SRC}) set_target_properties(simgrid-java PROPERTIES VERSION ${libsimgrid-java_version}) set_target_properties(simgrid-java PROPERTIES SKIP_BUILD_RPATH ON) target_link_libraries(simgrid-java simgrid) get_target_property(COMMON_INCLUDES simgrid-java INCLUDE_DIRECTORIES) if (COMMON_INCLUDES) set_target_properties(simgrid-java PROPERTIES INCLUDE_DIRECTORIES "${COMMON_INCLUDES};${JNI_INCLUDE_DIRS}") else() set_target_properties(simgrid-java PROPERTIES INCLUDE_DIRECTORIES "${JNI_INCLUDE_DIRS}") endif() get_target_property(CHECK_INCLUDES simgrid-java INCLUDE_DIRECTORIES) message(STATUS "[Java] simgrid-java includes: ${CHECK_INCLUDES}") # Rules to build simgrid.jar ############################ ## Files to include in simgrid.jar ## set(SIMGRID_JAR "${CMAKE_BINARY_DIR}/simgrid.jar") set(MANIFEST_IN_FILE "${CMAKE_HOME_DIRECTORY}/src/bindings/java/MANIFEST.in") set(MANIFEST_FILE "${CMAKE_BINARY_DIR}/src/bindings/java/MANIFEST.MF") set(LIBSIMGRID_SO libsimgrid${CMAKE_SHARED_LIBRARY_SUFFIX}) set(LIBSIMGRID_JAVA_SO ${CMAKE_SHARED_LIBRARY_PREFIX}simgrid-java${CMAKE_SHARED_LIBRARY_SUFFIX}) ## Here is how to build simgrid.jar ## if(CMAKE_VERSION VERSION_LESS "2.8.12") set(CMAKE_JAVA_TARGET_OUTPUT_NAME simgrid) add_jar(simgrid-java_jar ${JMSG_JAVA_SRC}) else() add_jar(simgrid-java_jar ${JMSG_JAVA_SRC} OUTPUT_NAME simgrid) endif() if(enable_lib_in_jar) add_dependencies(simgrid-java_jar simgrid-java) add_dependencies(simgrid-java_jar simgrid) endif() if (enable_documentation) add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the documentation into simgrid.jar..." DEPENDS ${MANIFEST_IN_FILE} ${CMAKE_HOME_DIRECTORY}/COPYING ${CMAKE_HOME_DIRECTORY}/ChangeLog ${CMAKE_HOME_DIRECTORY}/NEWS ${CMAKE_HOME_DIRECTORY}/LICENSE-LGPL-2.1 COMMAND ${CMAKE_COMMAND} -E copy ${MANIFEST_IN_FILE} ${MANIFEST_FILE} COMMAND ${CMAKE_COMMAND} -E echo "Specification-Version: \\\"${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}.${SIMGRID_VERSION_PATCH}\\\"" >> ${MANIFEST_FILE} COMMAND ${CMAKE_COMMAND} -E echo "Implementation-Version: \\\"${GIT_VERSION}\\\"" >> ${MANIFEST_FILE} COMMAND ${Java_JAVADOC_EXECUTABLE} -quiet -d doc/javadoc -sourcepath ${CMAKE_HOME_DIRECTORY}/src/bindings/java/ org.simgrid.msg org.simgrid.trace COMMAND ${JAVA_ARCHIVE} -uvmf ${MANIFEST_FILE} ${SIMGRID_JAR} doc/javadoc COMMAND ${JAVA_ARCHIVE} -uvf ${SIMGRID_JAR} -C ${CMAKE_HOME_DIRECTORY} COPYING -C ${CMAKE_HOME_DIRECTORY} ChangeLog -C ${CMAKE_HOME_DIRECTORY} LICENSE-LGPL-2.1 -C ${CMAKE_HOME_DIRECTORY} NEWS ) endif() ### ### Pack the java libraries into the jarfile if asked to do so ### if(enable_lib_in_jar) set(SG_SYSTEM_NAME ${CMAKE_SYSTEM_NAME}) if(${SG_SYSTEM_NAME} MATCHES "kFreeBSD") set(SG_SYSTEM_NAME GNU/kFreeBSD) endif() set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/${CMAKE_SYSTEM_PROCESSOR}) if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "^i[3-6]86$") set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/x86) endif() if( (${CMAKE_SYSTEM_PROCESSOR} MATCHES "x86_64") OR (${CMAKE_SYSTEM_PROCESSOR} MATCHES "AMD64") ) set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/amd64) endif() if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "armv7l") set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/arm) # Default arm (soft-float ABI) endif() add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the native libs into simgrid.jar..." DEPENDS simgrid simgrid-java ${JAVALIBS} COMMAND ${CMAKE_COMMAND} -E make_directory ${JAVA_NATIVE_PATH} COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/lib/${LIBSIMGRID_SO} ${JAVA_NATIVE_PATH}/${LIBSIMGRID_SO} COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/lib/${LIBSIMGRID_JAVA_SO} ${JAVA_NATIVE_PATH}/${LIBSIMGRID_JAVA_SO} ) if(WIN32) add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the windows-specific native libs into simgrid.jar..." DEPENDS simgrid simgrid-java ${JAVALIBS} # There is no way to disable the dependency of mingw-64 on that lib, unfortunately nor to script cmake -E properly # So let's be brutal and copy it in any case (even on non-windows builds) from the location where chocolatey installs it. # The copy is only expected to work on the appveyor builder, but that's all we need right now # since our users are directed to download that file as nightly build. COMMAND ${CMAKE_COMMAND} -E copy_if_different C:/tools/mingw64/bin/libwinpthread-1.dll ${JAVA_NATIVE_PATH}/libwinpthread-1.dll || true ) endif() if(APPLE) add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the apple-specific native libs into simgrid.jar..." DEPENDS simgrid simgrid-java ${JAVALIBS} # We need to fix the rpath of the simgrid-java library so that it # searches the simgrid library in the right location # # Since we don't officially install the lib before copying it in # the jarfile, the lib is searched for where it was built. Given # how we unpack it, we need to instruct simgrid-java to forget # about the build path, and search in its current directory # instead. # # This has to be done with the classical Apple tools, as follows: COMMAND install_name_tool -change ${CMAKE_BINARY_DIR}/lib/libsimgrid.${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}${CMAKE_SHARED_LIBRARY_SUFFIX} @loader_path/libsimgrid.dylib ${JAVA_NATIVE_PATH}/${LIBSIMGRID_JAVA_SO} ) endif(APPLE) add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Packing back the simgrid.jar with the native libs..." DEPENDS simgrid simgrid-java ${JAVALIBS} COMMAND ${JAVA_ARCHIVE} -uvf ${SIMGRID_JAR} ${JAVA_NATIVE_PATH} COMMAND ${CMAKE_COMMAND} -E echo "-- Cmake put the native code in ${JAVA_NATIVE_PATH}" COMMAND "${Java_JAVA_EXECUTABLE}" -classpath "${SIMGRID_JAR}" org.simgrid.NativeLib ) endif(enable_lib_in_jar) include_directories(${JNI_INCLUDE_DIRS} ${JAVA_INCLUDE_PATH} ${JAVA_INCLUDE_PATH2}) SimGrid-3.18/tools/cmake/test_prog/0000755000175000017500000000000013217757322017546 5ustar mquinsonmquinsonSimGrid-3.18/tools/cmake/test_prog/prog_makecontext.c0000644000175000017500000000211013217757322023255 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014, 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifdef __APPLE__ #define _XOPEN_SOURCE 700 #endif #include #include #include ucontext_t uc_child; ucontext_t uc_main; static void child(void) { /* switch back to the main context */ if (swapcontext(&uc_child, &uc_main) != 0) exit(2); } int main(int argc, char *argv[]) { void *stack = malloc(64 * 1024); /* configure a child user-space context */ if (stack == NULL) exit(3); if (getcontext(&uc_child) != 0) exit(4); uc_child.uc_link = NULL; uc_child.uc_stack.ss_sp = (char *) stack + (32 * 1024); uc_child.uc_stack.ss_size = 32 * 1024; uc_child.uc_stack.ss_flags = 0; makecontext(&uc_child, child, 0); /* switch into the user context */ if (swapcontext(&uc_main, &uc_child) != 0) exit(5); /* Fine, child came home */ printf("yes\n"); /* die successfully */ return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_sem_init.c0000644000175000017500000000071113217757322022547 0ustar mquinsonmquinson/* Copyright (c) 2010-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include int main(void) { sem_t s; if (sem_init(&s, 0, 0) != 0) return 1; return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_asan.cpp0000644000175000017500000000160613217757322022226 0ustar mquinsonmquinson/* Copyright (c) 2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Check availability of AddressSanitizer */ #if defined(__has_feature) #define HAS_FEATURE(x) __has_feature(x) #else #define HAS_FEATURE(x) 0 #endif #if not HAS_FEATURE(address_sanitizer) && not defined(__SANITIZE_ADDRESS__) #error "ASan feature not found." #endif #include #if defined(CHECK_FIBER_SUPPORT) // Verify the existence of the fiber annotation interface, with the expected signature void (*start_fiber)(void**, const void*, size_t) = __sanitizer_start_switch_fiber; void (*finish_fiber)(void*, const void**, size_t*) = __sanitizer_finish_switch_fiber; #endif int main(void) { } SimGrid-3.18/tools/cmake/test_prog/prog_stacksetup.c0000644000175000017500000000354313217757322023134 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifdef __APPLE__ #define _XOPEN_SOURCE 700 #endif #include #include #include #include union alltypes { long l; double d; void *vp; void (*fp) (void); char *cp; }; static char *handler_addr = (char *) 0xDEAD; static ucontext_t uc_handler; static ucontext_t uc_main; void handler(void) { char garbage[1024]; auto int dummy; for (int i = 0; i < 1024; i++) garbage[i] = 'X'; handler_addr = (char *) &dummy; swapcontext(&uc_handler, &uc_main); return; } int main(int argc, char *argv[]) { int sksize = 32768; char *skbuf = (char *) malloc(sksize * 2 + 2 * sizeof(union alltypes)); if (skbuf == NULL) exit(1); for (int i = 0; i < sksize * 2 + 2 * sizeof(union alltypes); i++) skbuf[i] = 'A'; char *skaddr = skbuf + sizeof(union alltypes); if (getcontext(&uc_handler) != 0) exit(1); uc_handler.uc_link = NULL; uc_handler.uc_stack.ss_sp = (void *) (skaddr + sksize); uc_handler.uc_stack.ss_size = sksize; uc_handler.uc_stack.ss_flags = 0; makecontext(&uc_handler, handler, 0); swapcontext(&uc_main, &uc_handler); if (handler_addr == (char *) 0xDEAD) exit(1); if (handler_addr < skaddr + sksize) { /* stack was placed into lower area */ if (*(skaddr + sksize) != 'A') printf("(skaddr)+(sksize)-%d;(sksize)-%d", sizeof(union alltypes), sizeof(union alltypes)); else printf("(skaddr)+(sksize);(sksize)"); } else { /* stack was placed into higher area */ if (*(skaddr + sksize * 2) != 'A') printf("(skaddr);(sksize)-%d", sizeof(union alltypes)); else printf("(skaddr);(sksize)"); } return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_sem_timedwait.c0000644000175000017500000000060013217757322023570 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main() { sem_t *s; const struct timespec *t; sem_timedwait(s, t); } SimGrid-3.18/tools/cmake/test_prog/prog_gnu_dynlinker.c0000644000175000017500000000176213217757322023617 0ustar mquinsonmquinson/* prog_gnu_dynlinker.c -- check that RTLD_NEXT is defined as in GNU linker */ /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #define _GNU_SOURCE 1 #include #include #include #include static void * (*real_malloc) (size_t); int main(void) { char *error; dlerror(); // clear any previous error real_malloc = (void * (*) (size_t)) dlsym(RTLD_NEXT, "malloc"); error = dlerror(); if (!error && real_malloc) { char *A = real_malloc(20); strcpy(A,"epic success"); free(A); return 0; // SUCCESS } else { if (error) printf("Error while checking for dlsym: %s\n",error); else printf("dlsym did not return any error, but failed to find malloc()\n"); return 1; // FAILED } } SimGrid-3.18/tools/cmake/test_prog/prog_vsnprintf.c0000644000175000017500000000150113217757322022767 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include int my_vsnprintf(char *buf, const char *tmpl, ...) { int i; va_list args; va_start(args, tmpl); i = vsnprintf(buf, 2, tmpl, args); va_end(args); return i; } int main(void) { char bufs[5] = { 'x', 'x', 'x', '\0', '\0' }; char bufd[5] = { 'x', 'x', 'x', '\0', '\0' }; int i; i = my_vsnprintf(bufs, "%s", "111"); if (strcmp(bufs, "1")) exit(1); if (i != 3) exit(1); i = my_vsnprintf(bufd, "%d", 111); if (strcmp(bufd, "1")) exit(1); if (i != 3) exit(1); exit(0); } SimGrid-3.18/tools/cmake/test_prog/prog_mutex_timedlock.c0000644000175000017500000000064213217757322024140 0ustar mquinsonmquinson/* Copyright (c) 2010-2011, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main(void) { pthread_mutex_t s; const struct timespec t; sem_timedlock(&s, &t); return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_thread_storage.c0000644000175000017500000000071513217757322023737 0ustar mquinsonmquinson/* Copyright (c) 2010-2011, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include __thread int thread_specific_variable = 0; int main(void) { thread_specific_variable++; printf("%d\n", thread_specific_variable); return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_snprintf.c0000644000175000017500000000120113217757322022576 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main(void) { char bufs[5] = { 'x', 'x', 'x', '\0', '\0' }; char bufd[5] = { 'x', 'x', 'x', '\0', '\0' }; int i; i = snprintf(bufs, 2, "%s", "111"); if (strcmp(bufs, "1")) exit(1); if (i != 3) exit(1); i = snprintf(bufd, 2, "%d", 111); if (strcmp(bufd, "1")) exit(1); if (i != 3) exit(1); exit(0); } SimGrid-3.18/tools/cmake/test_prog/prog_stackgrowth.c0000644000175000017500000000176513217757322023312 0ustar mquinsonmquinson/* Copyright (c) 2010, 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include static int iterate = 10; static int growsdown(int *x) { int y = (x > &y); if (--iterate > 0) y = growsdown(&y); /* The stack sometimes changes at the 0th level. * Original version did fail in this case, but I changed this around SimGrid 3.13 because of https://bugs.debian.org/814272 * Every arch failed on that day :( */ if (iterate != 0 && y != (x > &y)) { fprintf(stderr, "The stack changed its direction! (Iteration: %d. It was growing %s; &y=%p; &prevY=%p)\n", (10 - iterate), y ? "down" : "up", &y, x); exit(1); } return y; } int main(int argc, char *argv[]) { int x; printf("%s", growsdown(&x) ? "down" : "up"); return 0; } SimGrid-3.18/tools/cmake/test_prog/prog_sem_open.c0000644000175000017500000000124713217757322022552 0ustar mquinsonmquinson/* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #ifndef SEM_FAILED #define SEM_FAILED (-1) #endif int main(void) { #ifdef WIN32 int s; #else sem_t * s; #endif s = sem_open("/0", O_CREAT, 0644, 10); if (s == SEM_FAILED){ // printf("sem_open failed\n"); return 1; } // printf("sem_open succeeded\n"); return 0; } SimGrid-3.18/tools/cmake/cross-mingw.cmake0000644000175000017500000000104713217757322021014 0ustar mquinsonmquinson# cmake -DCMAKE_TOOLCHAIN_FILE=tools/cmake/cross-mingw.cmake .. set (CMAKE_SYSTEM_NAME Windows) set (CMAKE_SYSTEM_PROCESSOR x86_64) set (CMAKE_C_COMPILER /usr/bin/x86_64-w64-mingw32-gcc-win32) set (CMAKE_CXX_COMPILER /usr/bin/x86_64-w64-mingw32-g++-win32) set (CMAKE_Fortran_COMPILER /usr/bin/x86_64-w64-mingw32-gfortran-win32) set (CMAKE_FIND_ROOT_PATH /usr/lib/gcc/x86_64-w64-mingw32/ /usr/i686-w64-mingw32/) set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) SimGrid-3.18/tools/cmake/MakeLib.cmake0000644000175000017500000001004713217757322020050 0ustar mquinsonmquinson### Make Libs # On Mac OSX, specify that rpath is useful to look for the dependencies # See https://cmake.org/Wiki/CMake_RPATH_handling and Java.cmake set(MACOSX_RPATH ON) if(APPLE) SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE) # When installed, use system path set(CMAKE_SKIP_BUILD_RPATH FALSE) # When executing from build tree, take the lib from the build path if exists set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE) # When executing from build tree, take the lib from the system path if exists # add the current location of libsimgrid-java.dynlib as a location for libsimgrid.dynlib # (useful when unpacking the native libraries from the jarfile) set(CMAKE_INSTALL_RPATH "@loader_path/.;@rpath/.") endif() ############################### # Declare the library content # ############################### # Actually declare our libraries add_library(simgrid SHARED ${simgrid_sources}) set_target_properties(simgrid PROPERTIES VERSION ${libsimgrid_version}) add_dependencies(simgrid maintainer_files) if(enable_model-checking) add_executable(simgrid-mc ${MC_SIMGRID_MC_SRC}) target_link_libraries(simgrid-mc simgrid) set_target_properties(simgrid-mc PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin") endif() # Compute the dependencies of SimGrid ##################################### if (HAVE_BOOST_CONTEXTS) set(SIMGRID_DEP "${SIMGRID_DEP} ${Boost_CONTEXT_LIBRARY}") endif() if(CMAKE_USE_PTHREADS_INIT AND ${HAVE_THREAD_CONTEXTS}) set(SIMGRID_DEP "${SIMGRID_DEP} ${CMAKE_THREAD_LIBS_INIT}") endif() if(SIMGRID_HAVE_LUA) ADD_CUSTOM_TARGET(link_simgrid_lua ALL DEPENDS simgrid ) SET(SIMGRID_DEP "${SIMGRID_DEP} ${LUA_LIBRARY} ${DL_LIBRARY}") endif() if(HAVE_PAPI) SET(SIMGRID_DEP "${SIMGRID_DEP} -lpapi") endif() if(HAVE_GRAPHVIZ) if(HAVE_CGRAPH_LIB) SET(SIMGRID_DEP "${SIMGRID_DEP} -lcgraph") else() if(HAVE_AGRAPH_LIB) SET(SIMGRID_DEP "${SIMGRID_DEP} -lagraph -lcdt") endif() endif() endif() if(SIMGRID_HAVE_MC AND HAVE_GNU_LD AND NOT ${DL_LIBRARY} STREQUAL "") SET(SIMGRID_DEP "${SIMGRID_DEP} ${DL_LIBRARY}") endif() if(SIMGRID_HAVE_NS3) SET(SIMGRID_DEP "${SIMGRID_DEP} -lns${NS3_VERSION}-core${NS3_SUFFIX} -lns${NS3_VERSION}-csma${NS3_SUFFIX} -lns${NS3_VERSION}-point-to-point${NS3_SUFFIX} -lns${NS3_VERSION}-internet${NS3_SUFFIX} -lns${NS3_VERSION}-network${NS3_SUFFIX} -lns${NS3_VERSION}-applications${NS3_SUFFIX}") endif() if(HAVE_POSIX_GETTIME) SET(SIMGRID_DEP "${SIMGRID_DEP} -lrt") endif() if("${CMAKE_SYSTEM}" MATCHES "FreeBSD") set(SIMGRID_DEP "${SIMGRID_DEP} -lprocstat") endif() # Compute the dependencies of SMPI ################################## if(enable_smpi) if(NOT ${DL_LIBRARY} STREQUAL "") set(SIMGRID_DEP "${SIMGRID_DEP} ${DL_LIBRARY}") # for privatization endif() add_executable(smpimain src/smpi/smpi_main.c) target_link_libraries(smpimain simgrid) set_target_properties(smpimain PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) endif() if(enable_smpi AND APPLE) set(SIMGRID_DEP "${SIMGRID_DEP} -Wl,-U -Wl,_smpi_simulated_main") endif() # See https://github.com/HewlettPackard/foedus_code/blob/master/foedus-core/cmake/FindGccAtomic.cmake FIND_LIBRARY(GCCLIBATOMIC_LIBRARY NAMES atomic atomic.so.1 libatomic.so.1 HINTS $ENV{HOME}/local/lib64 $ENV{HOME}/local/lib /usr/local/lib64 /usr/local/lib /opt/local/lib64 /opt/local/lib /usr/lib64 /usr/lib /lib64 /lib ) # Fix a FTBFS on armel, mips, mipsel and friends (Debian's #872881) if(CMAKE_COMPILER_IS_GNUCC AND GCCLIBATOMIC_LIBRARY) set(SIMGRID_DEP "${SIMGRID_DEP} -Wl,--as-needed -latomic -Wl,--no-as-needed") endif() target_link_libraries(simgrid ${SIMGRID_DEP}) # Dependencies from maintainer mode ################################### if(enable_maintainer_mode) add_dependencies(simgrid smpi_generated_headers_call_location_tracing) endif() if(enable_maintainer_mode AND PYTHON_EXE) add_dependencies(simgrid simcalls_generated_src) endif() if(enable_maintainer_mode AND BISON_EXE AND LEX_EXE) add_dependencies(simgrid automaton_generated_src) endif() SimGrid-3.18/tools/cmake/MaintainerMode.cmake0000644000175000017500000003221113217757322021435 0ustar mquinsonmquinson# Change the following when we need a recent enough version of flexml to get the maintainer mode working set(FLEXML_MIN_MAJOR 1) set(FLEXML_MIN_MINOR 9) set(FLEXML_MIN_PATCH 6) # Change the following when we need a recent enough version of flex to get the maintainer mode working. # Maintainer mode does NOT work with version 2.5.35! set(FLEX_MIN_MAJOR 2) set(FLEX_MIN_MINOR 5) set(FLEX_MIN_PATCH 39) # the rest should only be changed if you understand what you're doing if(enable_maintainer_mode AND NOT WIN32) if (PYTHON_EXECUTABLE) add_custom_command( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h ${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.hpp DEPENDS ${CMAKE_HOME_DIRECTORY}/src/simix/simcalls.py ${CMAKE_HOME_DIRECTORY}/src/simix/simcalls.in COMMENT "Generating simcalls source files" COMMAND ${PYTHON_EXECUTABLE} simcalls.py WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/src/simix/ ) add_custom_target(simcalls_generated_src DEPENDS ${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h ${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.hpp ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h;${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp;${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp;${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.hpp" ) endif() endif() # Let's generate header files required by SMPI when the call location tracing # has been activated. if(enable_maintainer_mode AND NOT WIN32) add_custom_command(OUTPUT ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h COMMENT "Generating header files for call-location tracing with SMPI" # Make sure there is no space after the redirection operator (>). I received # error messages in that case on my Debian system. COMMAND "${CMAKE_HOME_DIRECTORY}/tools/smpi/generate_smpi_defines.pl" "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi.h >${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h" COMMAND "${CMAKE_HOME_DIRECTORY}/tools/smpi/generate_smpi_defines.pl" "-f" "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi.h >${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h" ) add_custom_target(smpi_generated_headers_call_location_tracing DEPENDS ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h;${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h" ) endif() if(enable_maintainer_mode AND NOT WIN32) find_program(FLEX_EXE NAMES flex) find_program(FLEXML_EXE NAMES flexml) find_program(SED_EXE NAMES sed) find_program(BISON_EXE NAMES bison) find_program(LEX_EXE NAMES lex) mark_as_advanced(BISON_EXE) mark_as_advanced(LEX_EXE) if(BISON_EXE AND LEX_EXE) add_custom_command( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_lexer.yy.c ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc DEPENDS ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.lex ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.yacc COMMENT "Generating automaton source files" COMMAND ${BISON_EXE} --name-prefix=xbt_automaton_parser_ -d parserPromela.yacc COMMAND ${LEX_EXE} --prefix=xbt_automaton_parser_ --outfile=automaton_lexer.yy.c parserPromela.lex WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/ ) add_custom_target(automaton_generated_src DEPENDS ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_lexer.yy.c ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc;${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc;${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_parse.yy.c" ) endif() message(STATUS "Found flex: ${FLEX_EXE}") IF(FLEX_EXE) set(HAVE_FLEX 1) execute_process(COMMAND ${FLEX_EXE} --version OUTPUT_VARIABLE FLEX_VERSION) string(REGEX MATCH "[0-9]+[.]+[0-9]+[.]+[0-9]+" FLEX_VERSION "${FLEX_VERSION}") string(REGEX MATCH "^[0-9]+" FLEX_MAJOR_VERSION "${FLEX_VERSION}") string(REGEX MATCH "[0-9]+[.]+[0-9]+$" FLEX_VERSION "${FLEX_VERSION}") string(REGEX MATCH "^[0-9]+" FLEX_MINOR_VERSION "${FLEX_VERSION}") string(REGEX MATCH "[0-9]+$" FLEX_PATCH_VERSION "${FLEX_VERSION}") ENDIF() message(STATUS "Found flexml: ${FLEXML_EXE}") IF(FLEXML_EXE) set(HAVE_FLEXML 1) execute_process(COMMAND ${FLEXML_EXE} --version OUTPUT_VARIABLE FLEXML_VERSION) if (FLEXML_VERSION MATCHES "version Id:") message(FATAL_ERROR "You have an ancient flexml version (${FLEXML_VERSION}). You need at least v${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH} to compile in maintainer mode. Upgrade your flexml, or disable the Maintainer mode option in cmake.") endif() string(REGEX MATCH "[0-9]+[.]+[0-9]+[.]+[0-9]+" FLEXML_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "^[0-9]*" FLEXML_MAJOR_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "[0-9]+[.]+[0-9]+$" FLEXML_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "^[0-9]+" FLEXML_MINOR_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "[0-9]+$" FLEXML_PATCH_VERSION "${FLEXML_VERSION}") ENDIF() message(STATUS "Found sed: ${SED_EXE}") if(HAVE_FLEXML AND HAVE_FLEX AND SED_EXE) message(STATUS "Flex version: ${FLEX_MAJOR_VERSION}.${FLEX_MINOR_VERSION}.${FLEX_PATCH_VERSION} (need at least version ${FLEX_MIN_MAJOR}.${FLEX_MIN_MINOR}.${FLEX_MIN_PATCH})") message(STATUS "Flexml version: ${FLEXML_MAJOR_VERSION}.${FLEXML_MINOR_VERSION}.${FLEXML_PATCH_VERSION} (need at least version ${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH})") IF( (${FLEXML_MAJOR_VERSION} LESS ${FLEXML_MIN_MAJOR}) OR ((${FLEXML_MAJOR_VERSION} EQUAL ${FLEXML_MIN_MAJOR}) AND (${FLEXML_MINOR_VERSION} LESS ${FLEXML_MIN_MINOR}) ) OR ( (${FLEXML_MAJOR_VERSION} EQUAL ${FLEXML_MIN_MAJOR}) AND (${FLEXML_MINOR_VERSION} EQUAL ${FLEXML_MIN_MINOR}) AND (${FLEXML_PATCH_VERSION} LESS ${FLEXML_MIN_PATCH}) )) message(FATAL_ERROR "Your flexml version is too old to compile in maintainer mode (need at least v${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH}). Upgrade your flexml, or disable the Maintainer mode option in cmake.") ENDIF() IF( (${FLEX_MAJOR_VERSION} LESS ${FLEX_MIN_MAJOR}) OR ((${FLEX_MAJOR_VERSION} EQUAL ${FLEX_MIN_MAJOR}) AND (${FLEX_MINOR_VERSION} LESS ${FLEX_MIN_MINOR}) ) OR ( (${FLEX_MAJOR_VERSION} EQUAL ${FLEX_MIN_MAJOR}) AND (${FLEX_MINOR_VERSION} EQUAL ${FLEX_MIN_MINOR}) AND (${FLEX_PATCH_VERSION} LESS ${FLEX_MIN_PATCH}) )) message(FATAL_ERROR "Your flex version is too old to compile in maintainer mode (need at least v${FLEX_MIN_MAJOR}.${FLEX_MIN_MINOR}.${FLEX_MIN_PATCH}). Upgrade your flex, or disable the Maintainer mode option in cmake (run 'ccmake').") ENDIF() set(string1 "'s/extern *\\([^(]*\\)\\( \\|\\( \\*\\)\\)/XBT_PUBLIC_DATA(\\1\\3) /'") set(string2 "'s/XBT_PUBLIC_DATA(\\([^)]*\\)) *\\([^(]*\\)(/XBT_PUBLIC(\\1) \\2(/'") set(string5 "'s/SET(DOCTYPE)/SET(ROOT_dax__adag)/'") set(string9 "'s/#include /#if defined(_WIN32)\\n# ifndef __STRICT_ANSI__\\n# include \\n# include \\n# endif\\n#else\\n# include \\n#endif/g'") set(string14 "'\\!^ \\* Generated [0-9/]\\{10\\} [0-9:]\\{8\\}\\.$$!d'") ADD_CUSTOM_COMMAND( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c DEPENDS ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd #${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.l: ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/src/surf/xml COMMAND ${FLEXML_EXE} --root-tags platform -b 1000000 -P surfxml --sysid=http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd -S src/surf/xml/simgrid_dtd.l -L src/surf/xml/simgrid.dtd COMMAND ${SED_EXE} -i ${string14} src/surf/xml/simgrid_dtd.l COMMAND ${SED_EXE} -i "'s/FAIL(\"Bad declaration %s.\",yytext)/FAIL(\"Bad declaration %s.\\\\nIf your are using a XML v3 file (check the version attribute in ), please update it with tools\\/simgrid_update_xml.pl\",yytext)/'" src/surf/xml/simgrid_dtd.l COMMAND ${CMAKE_COMMAND} -E echo " Generated src/surf/xml/simgrid_dtd.l" #${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.l: ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd COMMAND ${FLEXML_EXE} -b 1000000 --root-tags adag -P dax_ --sysid=dax.dtd -S src/simdag/dax_dtd.l -L src/simdag/dax.dtd COMMAND ${SED_EXE} -i ${string5} src/simdag/dax_dtd.l COMMAND ${SED_EXE} -i ${string14} src/simdag/dax_dtd.l COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.l" #${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h: ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/include/surf/simgrid.h COMMAND ${FLEXML_EXE} --root-tags platform -P surfxml --sysid=http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd -H src/surf/xml/simgrid_dtd.h -L src/surf/xml/simgrid.dtd COMMAND ${SED_EXE} -i ${string1} src/surf/xml/simgrid_dtd.h COMMAND ${SED_EXE} -i ${string2} src/surf/xml/simgrid_dtd.h COMMAND ${SED_EXE} -i ${string14} src/surf/xml/simgrid_dtd.h COMMAND ${CMAKE_COMMAND} -E echo " Generated src/surf/xml/simgrid_dtd.h" #${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h: ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h COMMAND ${FLEXML_EXE} --root-tags adag -P dax_ --sysid=dax.dtd -H src/simdag/dax_dtd.h -L src/simdag/dax.dtd COMMAND ${SED_EXE} -i ${string1} src/simdag/dax_dtd.h COMMAND ${SED_EXE} -i ${string2} src/simdag/dax_dtd.h COMMAND ${SED_EXE} -i ${string14} src/simdag/dax_dtd.h COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.h" #surf/xml/simgrid_dtd.c: surf/xml/simgrid_dtd.l COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c COMMAND ${FLEX_EXE} -o src/surf/xml/simgrid_dtd.c -Psurf_parse_ --noline src/surf/xml/simgrid_dtd.l COMMAND ${SED_EXE} -i ${string9} src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i 's/int yyl\;/unsigned int yyl\;/' src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i 's/int surf_parse_leng\;/unsigned int surf_parse_leng\;/' src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i 's/n = 0\; n < max_size/n = 0\; n < (size_t) max_size/' src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i "s/register //" src/surf/xml/simgrid_dtd.c COMMAND ${CMAKE_COMMAND} -E echo " Generated surf/xml/simgrid_dtd.c" #simdag/dax_dtd.c: simdag/dax_dtd.l COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/src/simdag COMMAND ${FLEX_EXE} -o src/simdag/dax_dtd.c -Pdax_ --noline src/simdag/dax_dtd.l COMMAND ${SED_EXE} -i ${string9} src/simdag/dax_dtd.c COMMAND ${SED_EXE} -i 's/int yyl\;/unsigned int yyl\;/' src/simdag/dax_dtd.c COMMAND ${SED_EXE} -i 's/int dax_leng\;/unsigned int dax_leng\;/' src/simdag/dax_dtd.c COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.c" WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY} COMMENT "Generating files in maintainer mode..." ) else() if(NOT HAVE_FLEXML) message(STATUS "Error : Install flexml to use the maintainer mode.") endif() if(NOT HAVE_FLEX) message(STATUS "Error : Install flex to use the maintainer mode.") endif() if(NOT SED_EXE) message(STATUS "Error : Install sed before use maintainer mode.") endif() message(FATAL_ERROR STATUS "Error : Need to install all tools for maintainer mode !!!") endif() endif() add_custom_target(maintainer_files DEPENDS ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c ) SimGrid-3.18/tools/cmake/scripts/0000755000175000017500000000000013217757322017227 5ustar mquinsonmquinsonSimGrid-3.18/tools/cmake/scripts/update_tesh.pl0000755000175000017500000000215413217757322022076 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright (c) 2012, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; if($#ARGV!=1) { die "Usage: perl update_tesh.pl \n"; } my($directory)=$ARGV[0]; my($old)=$ARGV[1]; chdir("$directory"); open SH_LIGNE, $old or die "Unable to open $old. $!\n"; my($line); my($line_exec); my($l); my($tmp); while(defined($line=)) { if($line =~ /^\$(.*)$/) { $line_exec = $line; $line =~ s/\$\{srcdir\:\=\.\}/./g; $line =~ s/\(/\\(/g; $line =~ s/\)/\\)/g; $line =~ s/\$SG_TEST_EXENV//g; $line =~ s/\$EXEEXT//g; $line =~ s/^\$\ */.\//g; $line =~ s/^.\/lua/lua/g; $line =~ s/^.\/ruby/ruby/g; $line =~ s/--log=([^ ]*)/--log="$1"/g; print "$line_exec"; chomp $line; open (FILE, "$line 2>&1|"); while(defined($l=)) { chomp $l; print "\> $l\n"; } close(FILE); } else { if($line =~ /^\>(.*)$/) { } else { print "$line"; } } } close(SH_LIGNE); SimGrid-3.18/tools/cmake/scripts/my_valgrind.pl0000755000175000017500000000131013217757322022075 0ustar mquinsonmquinson#!/usr/bin/env perl # Copyright (c) 2012-2016. The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; my @argv = ("valgrind"); my $count = 0; while (defined(my $arg = shift)) { # print "arg($count)$arg\n"; if ($arg =~ m!/smpirun$!) { @argv = ( $arg, "-wrapper", "@argv" ); } elsif ($arg eq "--cd") { $arg = shift; print "cd $arg\n"; chdir ($arg); $count++; } else { push @argv, $arg; } $count++; } # print "COMMAND:"; # map { print "$_ " } @argv; # print "\n"; system @argv; SimGrid-3.18/tools/simgrid_update_xml.pl0000755000175000017500000001553013217757325020707 0ustar mquinsonmquinson#! /usr/bin/env perl eval 'exec perl -S $0 ${1+"$@"}' if $running_under_some_shell; # This script updates the simgrid XML file passed as argument (modification in place) # It is built to do the conversion incrementally. # Copyright (c) 2006-2016. The SimGrid Team. # All rights reserved. # # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. =encoding UTF-8 =head1 NAME simgrid_update_xml - updates simgrid XML files to latest version =head1 SYNOPSIS B I =head1 DESCRIPTION simgrid_update_xml updates the simgrid XML file passed as argument. The file is modified in place, without any kind of backup. You may want to save a copy before running the script. In SimGrid XML files, the standard version is indicated in the version attribute of the platform tag. Current version is 4. Here is a list of major changes in each version. =over 4 =item B Used before SimGrid 3.3 =item B Introduced in SimGrid 3.3 =over 4 =item The version attribute of platform were added to allow file versioning. =item The link bandwidth changed from Mb/s to b/s; and the CPU power were changed from MFlop/s to Flop/s =back =item B Introduced in SimGrid 3.4 =over =item Several tags were renamed: CPU -> HOST NETWORK_LINK -> LINK ROUTE_ELEMENT -> LINK_CTN PLATFORM_DESCRIPTION -> PLATFORM =back =item B Introduced in SimGrid 3.5 =over 4 =item The AS tag were introduced. Every platform should now contain an englobing AS tag. =item Routes are now symmetric by default. =item Several tags were renamed (for sake of XML sanity): LINK:CTN -> LINK_CTN TRACE:CONNECT -> TRACE_CONNECT =back =item B Introduced in SimGrid 3.13 =over 4 =item Rename the attributes describing the amount of flop that a host / peer / cluster / cabinet can deliver per second. -> =item In , attribute kind="POWER" is now kind="SPEED". =item The DOCTYPE points to the right URL: http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd =item Units are now mandatory in attributes. USE THE SCRIPT sg_xml_unit_converter.py TO CONVERT THIS - speed. Old default: 'f' or 'flops'. Also defined: 'Yf', 'Zf', 'Ef', 'Pf', 'Tf', 'Gf', 'Mf', 'kf' 'yottaflops', 'zettaflops', 'exaflops', 'petaflops', 'teraflops', 'gigaflops', 'megaflops', 'kiloflops' - bandwidth. Old default: 'Bps' bytes per second (or 'bps' but 1 Bps = 8 bps) Also defined in bytes: 'TiBps', 'GiBps', 'MiBps', 'KiBps', 'TBps', 'GBps', 'MBps', 'kBps', 'Bps' And the same in bits: 'Tibps', 'Gibps', 'Mibps', 'Kibps', 'Tbps', 'Gbps', 'Mbps', 'kbps', 'bps' - latency. Old default: 's' second. Also defined: 'w' week, 'd' day, 'h' hour, 'm' minute, 'ms' millisecond, 'us' microsecond, 'ns' nanosecond, 'ps' picosecond =back =item B Introduced in SimGrid 3.16 (this is the current version) =over 4 =item Rename a few tags, but in a backward-compatible manner: the old names are still accepted. AS -> zone ASroute -> zoneRoute bypassAsRoute -> bypassZoneRoute process -> actor =back =back =head1 AUTHORS The SimGrid team (simgrid-devel@lists.gforge.inria.fr) =head1 COPYRIGHT AND LICENSE Copyright (c) 2006-2016. The SimGrid Team. All rights reserved. This program is free software; you may redistribute it and/or modify it under the terms of GNU LGPL (v2.1) license. =cut use strict; my $fromversion=-1; my $toversion=4.1; my $filename = $ARGV[0] or die "Usage: simgrid_update_xml.pl file_to_convert.xml\nPlease provide an XML to convert as a parameter.\n"; open INPUT, "$filename" or die "Cannot open input file $filename: $!\n"; my $output_string = "\n". "\n". "\n"; my($AS_opened)=0; my $line; while (defined($line = )) { chomp $line; # eat the header, whatever form it has next if ($line =~ s/<\?xml[^>]*>// && ! $line =~ /\S/); # just in case several tags are on the same line next if ($line =~ s/]*>// && ! $line =~ /\S/); if ($line =~ s///) { $fromversion = 0; print "$filename was using version 0\n"; next if !$line =~ /\S/; } elsif ($line =~ s///) { $fromversion = $1; if ($fromversion == $toversion) { warn "Input platform file $filename is already conformant to version $fromversion. This should be a no-op.\n"; } if ($fromversion > $toversion) { die "Input platform file $filename is more recent than this script (file version: $fromversion; script version: $toversion)\n"; } next if !$line =~ /\S/; print "$filename was using version $fromversion\n"; } if ($fromversion == 0) { while ($line =~ m|^(.*?)/) || ($line=~ /\n"; $AS_opened=1; } if($line=~/ $filename"; print OUTPUT $output_string; close OUTPUT; SimGrid-3.18/CITATION.bib0000644000175000017500000000074513217757324015220 0ustar mquinsonmquinson@Article{simgrid, hal_id = {hal-01017319}, url = {http://hal.inria.fr/hal-01017319}, title = {Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms}, author = {Casanova, Henri and Giersch, Arnaud and Legrand, Arnaud and Quinson, Martin and Suter, Fr{\'e}d{\'e}ric}, publisher = {Elsevier}, pages = {2899-2917}, journal = {Journal of Parallel and Distributed Computing}, volume = {74}, number = {10}, year = {2014}, month = Jun, } SimGrid-3.18/TODO0000644000175000017500000000167313217757340014017 0ustar mquinsonmquinson ************************************************ *** This file is a TODO. It is thus kinda *** *** outdated. You know the story, right? *** ************************************************ ### ### Urgent stuff: ### * Have a proper todo file ### ### Ongoing stuff ### * Clean up CMake files Non exhaustive list of subgoals: - Use genuine cmake mechanisms and variables when available, instead of reinventing the wheel. - Correctly determine system and architecture (e.g. x32). - Correctly determine compiler type and version (e.g. clang). - Use git to generate the dist archive. Either use git-archive to generate the tarball, or keep using cmake -E tar, but use git-ls-files to get the list of files to pack. * Replace XBT with the C++ standard library * Some XBT datacontainers seem to overlap. Kill some/all of them. - replace set with SWAG SimGrid-3.18/src/0000755000175000017500000000000013217757321014106 5ustar mquinsonmquinsonSimGrid-3.18/src/mc/0000755000175000017500000000000013217757321014505 5ustar mquinsonmquinsonSimGrid-3.18/src/mc/mc_mmu.hpp0000644000175000017500000000307213217757321016475 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_MMU_HPP #define SIMGRID_MC_MMU_HPP #include "xbt/misc.h" // xbt_pagesize... #include #include namespace simgrid { namespace mc { // TODO, do not depend on xbt_pagesize/xbt_pagebits but our own chunk size namespace mmu { static int chunkSize() { return xbt_pagesize; } /** @brief How many memory pages are necessary to store size bytes? * * @param size Byte size * @return Number of memory pages */ static XBT_ALWAYS_INLINE std::size_t chunkCount(std::size_t size) { size_t page_count = size >> xbt_pagebits; if (size & (xbt_pagesize - 1)) page_count++; return page_count; } /** @brief Split into chunk number and remaining offset */ static XBT_ALWAYS_INLINE std::pair split(std::uintptr_t offset) { return {offset >> xbt_pagebits, offset & (xbt_pagesize - 1)}; } /** Merge chunk number and remaining offset info a global offset */ static XBT_ALWAYS_INLINE std::uintptr_t join(std::size_t page, std::uintptr_t offset) { return ((std::uintptr_t)page << xbt_pagebits) + offset; } static XBT_ALWAYS_INLINE std::uintptr_t join(std::pair value) { return join(value.first, value.second); } static XBT_ALWAYS_INLINE bool sameChunk(std::uintptr_t a, std::uintptr_t b) { return (a >> xbt_pagebits) == (b >> xbt_pagebits); } } } } #endif SimGrid-3.18/src/mc/Frame.cpp0000644000175000017500000000220213217757316016243 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "xbt/sysdep.h" #include "src/mc/Frame.hpp" namespace simgrid { namespace mc { void* Frame::frame_base(unw_cursor_t& unw_cursor) const { simgrid::dwarf::Location location = simgrid::dwarf::resolve( frame_base_location, object_info, &unw_cursor, nullptr, nullptr, -1); if (location.in_memory()) return location.address(); else if (location.in_register()) { // This is a special case. // The register is not the location of the frame base // (a frame base cannot be located in a register). // Instead, DWARF defines this to mean that the register // contains the address of the frame base. unw_word_t word; unw_get_reg(&unw_cursor, location.register_id(), &word); return (void*) word; } else xbt_die("Unexpected location type"); } } } SimGrid-3.18/src/mc/mc_comm_pattern.hpp0000644000175000017500000000373413217757316020400 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_COMM_PATTERN_H #define SIMGRID_MC_COMM_PATTERN_H #include #include "smpi/smpi.h" #include "xbt/dynar.h" #include "src/mc/mc_state.hpp" namespace simgrid { namespace mc { struct PatternCommunicationList { unsigned int index_comm = 0; std::vector> list; }; } } extern "C" { /** * Type: `xbt_dynar_t` */ extern XBT_PRIVATE xbt_dynar_t initial_communications_pattern; /** * Type: `xbt_dynar_t>` */ extern XBT_PRIVATE xbt_dynar_t incomplete_communications_pattern; enum e_mc_call_type_t { MC_CALL_TYPE_NONE, MC_CALL_TYPE_SEND, MC_CALL_TYPE_RECV, MC_CALL_TYPE_WAIT, MC_CALL_TYPE_WAITANY, }; enum e_mc_comm_pattern_difference_t { NONE_DIFF, TYPE_DIFF, RDV_DIFF, TAG_DIFF, SRC_PROC_DIFF, DST_PROC_DIFF, DATA_SIZE_DIFF, DATA_DIFF, }; static inline e_mc_call_type_t MC_get_call_type(smx_simcall_t req) { switch (req->call) { case SIMCALL_COMM_ISEND: return MC_CALL_TYPE_SEND; case SIMCALL_COMM_IRECV: return MC_CALL_TYPE_RECV; case SIMCALL_COMM_WAIT: return MC_CALL_TYPE_WAIT; case SIMCALL_COMM_WAITANY: return MC_CALL_TYPE_WAITANY; default: return MC_CALL_TYPE_NONE; } } XBT_PRIVATE void MC_handle_comm_pattern(e_mc_call_type_t call_type, smx_simcall_t request, int value, xbt_dynar_t current_pattern, int backtracking); XBT_PRIVATE void MC_restore_communications_pattern(simgrid::mc::State* state); XBT_PRIVATE void MC_state_copy_incomplete_communications_pattern(simgrid::mc::State* state); XBT_PRIVATE void MC_state_copy_index_communications_pattern(simgrid::mc::State* state); } #endif SimGrid-3.18/src/mc/LocationList.hpp0000644000175000017500000000513713217757316017634 0ustar mquinsonmquinson/* Copyright (c) 2004-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_OBJECT_LOCATION_H #define SIMGRID_MC_OBJECT_LOCATION_H #include #include #include #include #include #include "xbt/base.h" #include "xbt/range.hpp" #include "src/mc/mc_base.h" #include "src/mc/mc_forward.hpp" #include "src/mc/DwarfExpression.hpp" namespace simgrid { namespace dwarf { /** A DWARF expression with optional validity constraints */ class LocationListEntry { public: typedef simgrid::xbt::Range range_type; private: DwarfExpression expression_; // By default, the expression is always valid: range_type range_ = {0, UINT64_MAX}; public: LocationListEntry() = default; LocationListEntry(DwarfExpression expression, range_type range) : expression_(std::move(expression)), range_(range) {} explicit LocationListEntry(DwarfExpression expression) : expression_(std::move(expression)), range_({0, UINT64_MAX}) { } DwarfExpression& expression() { return expression_; } DwarfExpression const& expression() const { return expression_; } bool valid_for_ip(unw_word_t ip) const { return range_.contain(ip); } }; typedef std::vector LocationList; /** Location of some variable in memory * * The variable is located either in memory of a register. */ class Location { private: void* memory_; int register_id_ = 0; public: explicit Location(void* x) : memory_(x) {} explicit Location(int register_id) : memory_(nullptr), register_id_(register_id) {} // Type of location: bool in_register() const { return memory_ == nullptr; } bool in_memory() const { return memory_ != nullptr; } // Get the location: void* address() const { return memory_; } int register_id() const { return register_id_; } }; XBT_PRIVATE Location resolve( simgrid::dwarf::DwarfExpression const& expression, simgrid::mc::ObjectInformation* object_info, unw_cursor_t* c, void* frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index); Location resolve( simgrid::dwarf::LocationList const& locations, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index); XBT_PRIVATE simgrid::dwarf::LocationList location_list( simgrid::mc::ObjectInformation& info, Dwarf_Attribute& attr); } } #endif SimGrid-3.18/src/mc/mc_memory.cpp0000644000175000017500000000146013217757316017205 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "xbt/log.h" #include "mc/mc.h" #include "src/mc/mc_private.hpp" #include "src/xbt/mmalloc/mmprivate.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_memory, mc, "Logging specific to MC (memory)"); extern "C" { /* Initialize the model-checker memory subsystem */ /* It creates the two heap regions: std_heap and mc_heap */ void MC_memory_init() { if (not malloc_use_mmalloc()) xbt_die("Model-checking support is not enabled: run with simgrid-mc."); } } SimGrid-3.18/src/mc/mc_state.cpp0000644000175000017500000002077113217757316017023 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "xbt/log.h" #include "xbt/sysdep.h" #include "src/mc/Transition.hpp" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_state.hpp" #include "src/mc/mc_xbt.hpp" #include "src/simix/smx_private.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_state, mc, "Logging specific to MC (state)"); namespace simgrid { namespace mc { State::State(unsigned long state_number) : num(state_number) { this->internal_comm.clear(); std::memset(&this->internal_req, 0, sizeof(this->internal_req)); std::memset(&this->executed_req, 0, sizeof(this->executed_req)); actorStates.resize(MC_smx_get_maxpid()); /* Stateful model checking */ if ((_sg_mc_checkpoint > 0 && (state_number % _sg_mc_checkpoint == 0)) || _sg_mc_termination) { system_state = simgrid::mc::take_snapshot(num); if (_sg_mc_comms_determinism || _sg_mc_send_determinism) { MC_state_copy_incomplete_communications_pattern(this); MC_state_copy_index_communications_pattern(this); } } } std::size_t State::interleaveSize() const { return boost::range::count_if(this->actorStates, [](simgrid::mc::ProcessState const& p) { return p.isTodo(); }); } Transition State::getTransition() const { return this->transition; } } } /* Search an enabled transition for the given process. * * This can be seen as an iterator returning the next transition of the process. * * We only consider the processes that are both * - marked "to be interleaved" in their ProcessState (controlled by the checker algorithm). * - which simcall can currently be executed (like a comm where the other partner is already known) * Once we returned the last enabled transition of a process, it is marked done. * * Things can get muddled with the WAITANY and TESTANY simcalls, that are rewritten on the fly to a bunch of WAIT * (resp TEST) transitions using the transition.argument field to remember what was the last returned sub-transition. */ static inline smx_simcall_t MC_state_get_request_for_process(simgrid::mc::State* state, smx_actor_t actor) { /* reset the outgoing transition */ simgrid::mc::ProcessState* procstate = &state->actorStates[actor->pid]; state->transition.pid = -1; state->transition.argument = -1; state->executed_req.call = SIMCALL_NONE; if (not simgrid::mc::actor_is_enabled(actor)) return nullptr; // Not executable in the application smx_simcall_t req = nullptr; switch (actor->simcall.call) { case SIMCALL_COMM_WAITANY: state->transition.argument = -1; while (procstate->times_considered < read_length(mc_model_checker->process(), remote(simcall_comm_waitany__get__comms(&actor->simcall)))) { if (simgrid::mc::request_is_enabled_by_idx(&actor->simcall, procstate->times_considered++)) { state->transition.argument = procstate->times_considered - 1; break; } } if (procstate->times_considered >= simgrid::mc::read_length(mc_model_checker->process(), simgrid::mc::remote(simcall_comm_waitany__get__comms(&actor->simcall)))) procstate->setDone(); if (state->transition.argument != -1) req = &actor->simcall; break; case SIMCALL_COMM_TESTANY: { unsigned start_count = procstate->times_considered; state->transition.argument = -1; while (procstate->times_considered < simcall_comm_testany__get__count(&actor->simcall)) if (simgrid::mc::request_is_enabled_by_idx(&actor->simcall, procstate->times_considered++)) { state->transition.argument = procstate->times_considered - 1; break; } if (procstate->times_considered >= simcall_comm_testany__get__count(&actor->simcall)) procstate->setDone(); if (state->transition.argument != -1 || start_count == 0) req = &actor->simcall; break; } case SIMCALL_COMM_WAIT: { simgrid::mc::RemotePtr remote_act = remote(static_cast(simcall_comm_wait__getraw__comm(&actor->simcall))); simgrid::mc::Remote temp_act; mc_model_checker->process().read(temp_act, remote_act); simgrid::kernel::activity::CommImpl* act = temp_act.getBuffer(); if (act->src_proc && act->dst_proc) state->transition.argument = 0; else if (act->src_proc == nullptr && act->type == SIMIX_COMM_READY && act->detached == 1) state->transition.argument = 0; else state->transition.argument = -1; procstate->setDone(); req = &actor->simcall; break; } case SIMCALL_MC_RANDOM: { int min_value = simcall_mc_random__get__min(&actor->simcall); state->transition.argument = procstate->times_considered + min_value; procstate->times_considered++; if (state->transition.argument == simcall_mc_random__get__max(&actor->simcall)) procstate->setDone(); req = &actor->simcall; break; } default: procstate->setDone(); state->transition.argument = 0; req = &actor->simcall; break; } if (not req) return nullptr; state->transition.pid = actor->pid; state->executed_req = *req; // Fetch the data of the request and translate it: state->internal_req = *req; /* The waitany and testany request are transformed into a wait or test request over the corresponding communication * action so it can be treated later by the dependence function. */ switch (req->call) { case SIMCALL_COMM_WAITANY: { state->internal_req.call = SIMCALL_COMM_WAIT; simgrid::kernel::activity::ActivityImpl* remote_comm; read_element(mc_model_checker->process(), &remote_comm, remote(simcall_comm_waitany__getraw__comms(req)), state->transition.argument, sizeof(remote_comm)); mc_model_checker->process().read(state->internal_comm, remote(static_cast(remote_comm))); simcall_comm_wait__set__comm(&state->internal_req, state->internal_comm.getBuffer()); simcall_comm_wait__set__timeout(&state->internal_req, 0); break; } case SIMCALL_COMM_TESTANY: state->internal_req.call = SIMCALL_COMM_TEST; if (state->transition.argument > 0) { simgrid::kernel::activity::ActivityImpl* remote_comm = mc_model_checker->process().read( remote(simcall_comm_testany__getraw__comms(req) + state->transition.argument)); mc_model_checker->process().read(state->internal_comm, remote(static_cast(remote_comm))); } simcall_comm_test__set__comm(&state->internal_req, state->internal_comm.getBuffer()); simcall_comm_test__set__result(&state->internal_req, state->transition.argument); break; case SIMCALL_COMM_WAIT: mc_model_checker->process().read_bytes(&state->internal_comm, sizeof(state->internal_comm), remote(simcall_comm_wait__getraw__comm(req))); simcall_comm_wait__set__comm(&state->executed_req, state->internal_comm.getBuffer()); simcall_comm_wait__set__comm(&state->internal_req, state->internal_comm.getBuffer()); break; case SIMCALL_COMM_TEST: mc_model_checker->process().read_bytes(&state->internal_comm, sizeof(state->internal_comm), remote(simcall_comm_test__getraw__comm(req))); simcall_comm_test__set__comm(&state->executed_req, state->internal_comm.getBuffer()); simcall_comm_test__set__comm(&state->internal_req, state->internal_comm.getBuffer()); break; default: /* No translation needed */ break; } return req; } smx_simcall_t MC_state_get_request(simgrid::mc::State* state) { for (auto& actor : mc_model_checker->process().actors()) { /* Only consider the actors that were marked as interleaving by the checker algorithm */ if (not state->actorStates[actor.copy.getBuffer()->pid].isTodo()) continue; smx_simcall_t res = MC_state_get_request_for_process(state, actor.copy.getBuffer()); if (res) return res; } return nullptr; } SimGrid-3.18/src/mc/mc_dwarf_tagnames.cpp0000644000175000017500000000671313217757316020665 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include #include "src/mc/mc_dwarf.hpp" namespace { const std::unordered_map tagname_map = { {0x00, "DW_TAG_invalid"}, {0x01, "DW_TAG_array_type"}, {0x02, "DW_TAG_class_type"}, {0x03, "DW_TAG_entry_point"}, {0x04, "DW_TAG_enumeration_type"}, {0x05, "DW_TAG_formal_parameter"}, {0x08, "DW_TAG_imported_declaration"}, {0x0a, "DW_TAG_label"}, {0x0b, "DW_TAG_lexical_block"}, {0x0d, "DW_TAG_member"}, {0x0f, "DW_TAG_pointer_type"}, {0x10, "DW_TAG_reference_type"}, {0x11, "DW_TAG_compile_unit"}, {0x12, "DW_TAG_string_type"}, {0x13, "DW_TAG_structure_type"}, {0x15, "DW_TAG_subroutine_type"}, {0x16, "DW_TAG_typedef"}, {0x17, "DW_TAG_union_type"}, {0x18, "DW_TAG_unspecified_parameters"}, {0x19, "DW_TAG_variant"}, {0x1a, "DW_TAG_common_block"}, {0x1b, "DW_TAG_common_inclusion"}, {0x1c, "DW_TAG_inheritance"}, {0x1d, "DW_TAG_inlined_subroutine"}, {0x1e, "DW_TAG_module"}, {0x1f, "DW_TAG_ptr_to_member_type"}, {0x20, "DW_TAG_set_type"}, {0x21, "DW_TAG_subrange_type"}, {0x22, "DW_TAG_with_stmt"}, {0x23, "DW_TAG_access_declaration"}, {0x24, "DW_TAG_base_type"}, {0x25, "DW_TAG_catch_block"}, {0x26, "DW_TAG_const_type"}, {0x27, "DW_TAG_constant"}, {0x28, "DW_TAG_enumerator"}, {0x29, "DW_TAG_file_type"}, {0x2a, "DW_TAG_friend"}, {0x2b, "DW_TAG_namelist"}, {0x2c, "DW_TAG_namelist_item"}, {0x2d, "DW_TAG_packed_type"}, {0x2e, "DW_TAG_subprogram"}, {0x2f, "DW_TAG_template_type_parameter"}, {0x30, "DW_TAG_template_value_parameter"}, {0x31, "DW_TAG_thrown_type"}, {0x32, "DW_TAG_try_block"}, {0x33, "DW_TAG_variant_part"}, {0x34, "DW_TAG_variable"}, {0x35, "DW_TAG_volatile_type"}, {0x36, "DW_TAG_dwarf_procedure"}, {0x37, "DW_TAG_restrict_type"}, {0x38, "DW_TAG_interface_type"}, {0x39, "DW_TAG_namespace"}, {0x3a, "DW_TAG_imported_module"}, {0x3b, "DW_TAG_unspecified_type"}, {0x3c, "DW_TAG_partial_unit"}, {0x3d, "DW_TAG_imported_unit"}, {0x3f, "DW_TAG_condition"}, {0x40, "DW_TAG_shared_type"}, {0x41, "DW_TAG_type_unit"}, {0x42, "DW_TAG_rvalue_reference_type"}, {0x43, "DW_TAG_template_alias"}, {0x47, "DW_TAG_atomic_type"}, {0x4080, "DW_TAG_lo_user"}, {0x4081, "DW_TAG_MIPS_loop"}, {0x4101, "DW_TAG_format_label"}, {0x4102, "DW_TAG_function_template"}, {0x4103, "DW_TAG_class_template"}, {0x4104, "DW_TAG_GNU_BINCL"}, {0x4105, "DW_TAG_GNU_EINCL"}, {0x4106, "DW_TAG_GNU_template_template_param"}, {0x4107, "DW_TAG_GNU_template_parameter_pack"}, {0x4108, "DW_TAG_GNU_formal_parameter_pack"}, {0x4109, "DW_TAG_GNU_call_site"}, {0x410a, "DW_TAG_GNU_call_site_parameter"}, {0xffff, "DW_TAG_hi_user"}, }; } namespace simgrid { namespace dwarf { /** \brief Get the name of a dwarf tag (DW_TAG_*) from its code * * \param tag tag code (see the DWARF specification) * \return name of the tag */ XBT_PRIVATE const char *tagname(int tag) { auto name = tagname_map.find(tag); return name == tagname_map.end() ? "DW_TAG_unknown" : name->second; } } } SimGrid-3.18/src/mc/ModelChecker.hpp0000644000175000017500000000460613217757316017555 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_MODEL_CHECKER_HPP #define SIMGRID_MC_MODEL_CHECKER_HPP #include #include #include #include #include #include "xbt/base.h" #include #include "src/mc/PageStore.hpp" #include "src/mc/Transition.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/remote/RemoteClient.hpp" #include "src/mc/remote/mc_protocol.h" namespace simgrid { namespace mc { /** State of the model-checker (global variables for the model checker) */ class ModelChecker { struct event_base *base_; struct event* socket_event_; struct event* signal_event_; /** String pool for host names */ // TODO, use std::set with heterogeneous comparison lookup (C++14)? std::set hostnames_; // This is the parent snapshot of the current state: PageStore page_store_; std::unique_ptr process_; Checker* checker_ = nullptr; public: std::shared_ptr parent_snapshot_; ModelChecker(ModelChecker const&) = delete; ModelChecker& operator=(ModelChecker const&) = delete; explicit ModelChecker(std::unique_ptr process); ~ModelChecker(); RemoteClient& process() { return *process_; } PageStore& page_store() { return page_store_; } std::string const& get_host_name(const char* hostname) { return *this->hostnames_.insert(hostname).first; } std::string const& get_host_name(std::string const& hostname) { return *this->hostnames_.insert(hostname).first; } void start(); void shutdown(); void resume(simgrid::mc::RemoteClient& process); void loop(); void handle_events(int fd, short events); void wait_for_requests(); void handle_simcall(Transition const& transition); void exit(int status); bool checkDeadlock(); Checker* getChecker() const { return checker_; } void setChecker(Checker* checker) { checker_ = checker; } private: void setup_ignore(); bool handle_message(char* buffer, ssize_t size); void handle_waitpid(); void on_signal(int signo); public: unsigned long visited_states = 0; unsigned long executed_transitions = 0; }; } } #endif SimGrid-3.18/src/mc/mc_forward.hpp0000644000175000017500000000146713217757316017355 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** @file mc_forward.hpp * * Forward definitions for MC types */ #ifndef SIMGRID_MC_FORWARD_HPP #define SIMGRID_MC_FORWARD_HPP namespace simgrid { namespace mc { class PageStore; class ChunkedData; class ModelChecker; class AddressSpace; class RemoteClient; class Snapshot; class ObjectInformation; class Member; class Type; class Variable; class Frame; class ActorInformation; class Session; class Checker; } } // TODO, try to get rid of the global ModelChecker variable extern simgrid::mc::ModelChecker* mc_model_checker; #endif SimGrid-3.18/src/mc/mc_replay.hpp0000644000175000017500000000117113217757315017174 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REPLAY_H #define SIMGRID_MC_REPLAY_H #include "xbt/base.h" #include /** Replay path (if any) in string representation * * This is using the format generated by traceToString(). */ XBT_PUBLIC_DATA(std::string) MC_record_path; /** Whether the replay mode is enabled */ static inline int MC_record_replay_is_active() { return not MC_record_path.empty(); } #endif SimGrid-3.18/src/mc/ChunkedData.cpp0000644000175000017500000000327513217757316017377 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "xbt/asserts.h" #include "xbt/misc.h" #include "src/mc/AddressSpace.hpp" #include "src/mc/ChunkedData.hpp" #include "src/mc/PageStore.hpp" namespace simgrid { namespace mc { /** Take a per-page snapshot of a region * * @param addr The start of the region (must be at the beginning of a page) * @param page_count Number of pages of the region * @return Snapshot page numbers of this new snapshot */ ChunkedData::ChunkedData(PageStore& store, AddressSpace& as, RemotePtr addr, std::size_t page_count) { store_ = &store; this->pagenos_.resize(page_count); std::vector buffer(xbt_pagesize); for (size_t i = 0; i != page_count; ++i) { RemotePtr page = remote((void*) simgrid::mc::mmu::join(i, addr.address())); xbt_assert(simgrid::mc::mmu::split(page.address()).second == 0, "Not at the beginning of a page"); /* Adding another copy (and a syscall) will probably slow things a lot. TODO, optimize this somehow (at least by grouping the syscalls) if needed. Either: - reduce the number of syscalls - let the application snapshot itself - move the segments in shared memory (this will break `fork` however) */ as.read_bytes(buffer.data(), xbt_pagesize, page, simgrid::mc::ProcessIndexDisabled); pagenos_[i] = store_->store_page(buffer.data()); } } } } SimGrid-3.18/src/mc/Frame.hpp0000644000175000017500000000333413217757316016257 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_FRAME_HPP #define SIMGRID_MC_FRAME_HPP #include #include #include "xbt/base.h" #include "xbt/range.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" #include "src/mc/Variable.hpp" namespace simgrid { namespace mc { /** Debug information about a given function or scope within a function */ class Frame { public: Frame(); /** Kind of scope (DW_TAG_subprogram, DW_TAG_inlined_subroutine, etc.) */ int tag = DW_TAG_invalid; /** Name of the function (if it is a function) */ std::string name; /** Range of instruction addresses for which this scope is valid */ simgrid::xbt::Range range; simgrid::dwarf::LocationList frame_base_location; /** List of the variables (sorted by name) */ std::vector variables; /* Unique identifier for this scope (in the object_info) * * This is the global DWARF offset of the DIE. */ unsigned long int id = 0; std::vector scopes; /** Value of `DW_AT_abstract_origin` * * For inlined subprograms, this is the ID of the * parent function. */ unsigned long int abstract_origin_id = 0; simgrid::mc::ObjectInformation* object_info = nullptr; void* frame_base(unw_cursor_t& unw_cursor) const; void remove_variable(char* name); }; inline Frame::Frame() { this->tag = 0; this->range = {0, 0}; this->id = 0; this->abstract_origin_id = 0; this->object_info = nullptr; } } } #endif SimGrid-3.18/src/mc/mc_unw.cpp0000644000175000017500000002046113217757316016510 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file * Libunwind support for mc_address_space objects. */ // We need this for the register indices: // #define _GNU_SOURCE #include // On x86_64, libunwind unw_context_t has the same layout as ucontext_t: #include #include #ifdef __FreeBSD__ typedef register_t greg_t; #endif #include #include "src/mc/Frame.hpp" #include "src/mc/mc_unw.hpp" #include "src/mc/remote/RemoteClient.hpp" using simgrid::mc::remote; namespace simgrid { namespace mc { // ***** Implementation /** Get frame unwind information (libunwind method) * * Delegates to the local/ptrace implementation. */ int UnwindContext::find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t *pip, int need_unwind_info, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->find_proc_info( context->process_->unw_underlying_addr_space, ip, pip, need_unwind_info, context->process_->unw_underlying_context ); } /** Release frame unwind information (libunwind method) * * Delegates to the local/ptrace implementation. */ void UnwindContext::put_unwind_info(unw_addr_space_t as, unw_proc_info_t *pip, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->put_unwind_info( context->process_->unw_underlying_addr_space, pip, context->process_->unw_underlying_context ); } /** (libunwind method) * * Not implemented. */ int UnwindContext::get_dyn_info_list_addr(unw_addr_space_t as, unw_word_t *dilap, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->get_dyn_info_list_addr( context->process_->unw_underlying_addr_space, dilap, context->process_->unw_underlying_context ); } /** Read from the target address space memory (libunwind method) * * Delegates to the `simgrid::mc::Process*`. */ int UnwindContext::access_mem(unw_addr_space_t as, unw_word_t addr, unw_word_t *valp, int write, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; if (write) return - UNW_EREADONLYREG; context->addressSpace_->read_bytes(valp, sizeof(unw_word_t), remote(addr)); return 0; } void* UnwindContext::get_reg(unw_context_t* context, unw_regnum_t regnum) noexcept { #ifdef __x86_64 mcontext_t* mcontext = &context->uc_mcontext; switch (regnum) { # ifdef __linux__ case UNW_X86_64_RAX: return &mcontext->gregs[REG_RAX]; case UNW_X86_64_RDX: return &mcontext->gregs[REG_RDX]; case UNW_X86_64_RCX: return &mcontext->gregs[REG_RCX]; case UNW_X86_64_RBX: return &mcontext->gregs[REG_RBX]; case UNW_X86_64_RSI: return &mcontext->gregs[REG_RSI]; case UNW_X86_64_RDI: return &mcontext->gregs[REG_RDI]; case UNW_X86_64_RBP: return &mcontext->gregs[REG_RBP]; case UNW_X86_64_RSP: return &mcontext->gregs[REG_RSP]; case UNW_X86_64_R8: return &mcontext->gregs[REG_R8]; case UNW_X86_64_R9: return &mcontext->gregs[REG_R9]; case UNW_X86_64_R10: return &mcontext->gregs[REG_R10]; case UNW_X86_64_R11: return &mcontext->gregs[REG_R11]; case UNW_X86_64_R12: return &mcontext->gregs[REG_R12]; case UNW_X86_64_R13: return &mcontext->gregs[REG_R13]; case UNW_X86_64_R14: return &mcontext->gregs[REG_R14]; case UNW_X86_64_R15: return &mcontext->gregs[REG_R15]; case UNW_X86_64_RIP: return &mcontext->gregs[REG_RIP]; # elif defined __FreeBSD__ case UNW_X86_64_RAX: return &mcontext->mc_rax; case UNW_X86_64_RDX: return &mcontext->mc_rdx; case UNW_X86_64_RCX: return &mcontext->mc_rcx; case UNW_X86_64_RBX: return &mcontext->mc_rbx; case UNW_X86_64_RSI: return &mcontext->mc_rsi; case UNW_X86_64_RDI: return &mcontext->mc_rdi; case UNW_X86_64_RBP: return &mcontext->mc_rbp; case UNW_X86_64_RSP: return &mcontext->mc_rsp; case UNW_X86_64_R8: return &mcontext->mc_r8; case UNW_X86_64_R9: return &mcontext->mc_r9; case UNW_X86_64_R10: return &mcontext->mc_r10; case UNW_X86_64_R11: return &mcontext->mc_r11; case UNW_X86_64_R12: return &mcontext->mc_r12; case UNW_X86_64_R13: return &mcontext->mc_r13; case UNW_X86_64_R14: return &mcontext->mc_r14; case UNW_X86_64_R15: return &mcontext->mc_r15; case UNW_X86_64_RIP: return &mcontext->mc_rip; # else # error "Unable to get register from ucontext, please add your case" # endif default: return nullptr; } #else return nullptr; #endif } /** Read a standard register (libunwind method) */ int UnwindContext::access_reg(unw_addr_space_t as, unw_regnum_t regnum, unw_word_t *valp, int write, void* arg) noexcept { simgrid::mc::UnwindContext* as_context = (simgrid::mc::UnwindContext*) arg; unw_context_t* context = &as_context->unwindContext_; if (write) return -UNW_EREADONLYREG; greg_t* preg = (greg_t*) get_reg(context, regnum); if (not preg) return -UNW_EBADREG; *valp = *preg; return 0; } /** Read a floating-point register (libunwind method) * * FP registers are caller-saved. The values saved by functions such as * `getcontext()` is not relevant for the caller. It is not really necessary * to save and handle them. */ int UnwindContext::access_fpreg(unw_addr_space_t as, unw_regnum_t regnum, unw_fpreg_t *fpvalp, int write, void* arg) noexcept { return -UNW_EBADREG; } /** Resume the execution of the context (libunwind method) * * We don't use this. */ int UnwindContext::resume(unw_addr_space_t as, unw_cursor_t *cp, void* arg) noexcept { return -UNW_EUNSPEC; } /** Find informations about a function (libunwind method) */ int UnwindContext::get_proc_name(unw_addr_space_t as, unw_word_t addr, char *bufp, size_t buf_len, unw_word_t *offp, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; simgrid::mc::Frame* frame = context->process_->find_function(remote(addr)); if (not frame) return - UNW_ENOINFO; *offp = (unw_word_t) frame->range.begin() - addr; strncpy(bufp, frame->name.c_str(), buf_len); if (bufp[buf_len - 1]) { bufp[buf_len - 1] = 0; return -UNW_ENOMEM; } return 0; } // ***** Init /** Virtual table for our `libunwind` implementation * * Stack unwinding on a `simgrid::mc::Process*` (for memory, unwinding information) * and `ucontext_t` (for processor registers). * * It works with the `simgrid::mc::UnwindContext` context. */ unw_accessors_t UnwindContext::accessors = { &find_proc_info, &put_unwind_info, &get_dyn_info_list_addr, &access_mem, &access_reg, &access_fpreg, &resume, &get_proc_name }; unw_addr_space_t UnwindContext::createUnwindAddressSpace() { return unw_create_addr_space(&UnwindContext::accessors, BYTE_ORDER); } void UnwindContext::clear() { addressSpace_ = nullptr; process_ = nullptr; } void UnwindContext::initialize(simgrid::mc::RemoteClient* process, unw_context_t* c) { clear(); this->addressSpace_ = process; this->process_ = process; // Take a copy of the context for our own purpose: this->unwindContext_ = *c; #if SIMGRID_PROCESSOR_x86_64 || SIMGRID_PROCESSOR_i686 # ifdef __linux__ // On x86_64, ucontext_t contains a pointer to itself for FP registers. // We don't really need support for FR registers as they are caller saved // and probably never use those fields as libunwind-x86_64 does not read // FP registers from the unw_context_t // Let's ignore this and see what happens: this->unwindContext_.uc_mcontext.fpregs = nullptr; # endif #else // Do we need to do any fixup like this? # error Target CPU type is not handled. #endif } unw_cursor_t UnwindContext::cursor() { unw_cursor_t cursor; if (process_ == nullptr || addressSpace_ == nullptr || unw_init_remote(&cursor, process_->unw_addr_space, this) != 0) xbt_die("UnwindContext not initialized"); return cursor; } } } SimGrid-3.18/src/mc/RegionSnapshot.cpp0000644000175000017500000001005013217757316020154 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #ifdef __FreeBSD__ # define MAP_POPULATE MAP_PREFAULT_READ #endif #include "mc/mc.h" #include "src/mc/mc_snapshot.hpp" #include "src/mc/ChunkedData.hpp" #include "src/mc/RegionSnapshot.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_RegionSnaphot, mc, "Logging specific to region snapshots"); namespace simgrid { namespace mc { static inline const char* to_cstr(RegionType region) { switch (region) { case RegionType::Unknown: return "unknown"; case RegionType::Heap: return "Heap"; case RegionType::Data: return "Data"; default: return "?"; } } Buffer::Buffer(std::size_t size, Type type) : size_(size), type_(type) { switch(type_) { case Type::Malloc: data_ = ::operator new(size_); break; case Type::Mmap: data_ = ::mmap(nullptr, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (data_ == MAP_FAILED) { data_ = nullptr; size_ = 0; type_ = Type::Malloc; throw std::bad_alloc(); } break; default: abort(); } } void Buffer::clear() noexcept { switch(type_) { case Type::Malloc: ::operator delete(data_); break; case Type::Mmap: if (munmap(data_, size_) != 0) abort(); break; default: abort(); } data_ = nullptr; size_ = 0; type_ = Type::Malloc; } RegionSnapshot dense_region( RegionType region_type, void *start_addr, void* permanent_addr, size_t size) { // When KSM support is enables, we allocate memory using mmap: // * we don't want to advise bits of the heap as mergable // * mmap gives data aligned on page boundaries which is merge friendly simgrid::mc::Buffer data; if (_sg_mc_ksm) data = Buffer::mmap(size); else data = Buffer::malloc(size); mc_model_checker->process().read_bytes(data.get(), size, remote(permanent_addr), simgrid::mc::ProcessIndexDisabled); #ifdef __linux__ if (_sg_mc_ksm) // Mark the region as mergeable *after* we have written into it. // Trying to merge them before is useless/counterproductive. madvise(data.get(), size, MADV_MERGEABLE); #endif simgrid::mc::RegionSnapshot region( region_type, start_addr, permanent_addr, size); region.flat_data(std::move(data)); XBT_DEBUG("New region : type : %s, data : %p (real addr %p), size : %zu", to_cstr(region_type), region.flat_data().get(), permanent_addr, size); return region; } /** @brief Take a snapshot of a given region * * @param type * @param start_addr Address of the region in the simulated process * @param permanent_addr Permanent address of this data (for privatized variables, this is the virtual address of the privatized mapping) * @param size Size of the data* */ RegionSnapshot region( RegionType type, void *start_addr, void* permanent_addr, size_t size) { if (_sg_mc_sparse_checkpoint) return sparse_region(type, start_addr, permanent_addr, size); else return dense_region(type, start_addr, permanent_addr, size); } RegionSnapshot sparse_region(RegionType region_type, void *start_addr, void* permanent_addr, size_t size) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); assert(process != nullptr); xbt_assert((((uintptr_t)start_addr) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); xbt_assert((((uintptr_t)permanent_addr) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); size_t page_count = simgrid::mc::mmu::chunkCount(size); simgrid::mc::ChunkedData page_data(mc_model_checker->page_store(), *process, RemotePtr(permanent_addr), page_count); simgrid::mc::RegionSnapshot region( region_type, start_addr, permanent_addr, size); region.page_data(std::move(page_data)); return region; } } } SimGrid-3.18/src/mc/DwarfExpression.cpp0000644000175000017500000001544213217757316020346 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/AddressSpace.hpp" #include "src/mc/DwarfExpression.hpp" #include "src/mc/Frame.hpp" #include "src/mc/LocationList.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/mc_dwarf.hpp" #include "src/mc/mc_private.hpp" using simgrid::mc::remote; namespace simgrid { namespace dwarf { void execute( const Dwarf_Op* ops, std::size_t n, const ExpressionContext& context, ExpressionStack& stack) { for (size_t i = 0; i != n; ++i) { const Dwarf_Op *op = ops + i; std::uint8_t atom = op->atom; intptr_t first; intptr_t second; switch (atom) { // Registers: case DW_OP_breg0: case DW_OP_breg1: case DW_OP_breg2: case DW_OP_breg3: case DW_OP_breg4: case DW_OP_breg5: case DW_OP_breg6: case DW_OP_breg7: case DW_OP_breg8: case DW_OP_breg9: case DW_OP_breg10: case DW_OP_breg11: case DW_OP_breg12: case DW_OP_breg13: case DW_OP_breg14: case DW_OP_breg15: case DW_OP_breg16: case DW_OP_breg17: case DW_OP_breg18: case DW_OP_breg19: case DW_OP_breg20: case DW_OP_breg21: case DW_OP_breg22: case DW_OP_breg23: case DW_OP_breg24: case DW_OP_breg25: case DW_OP_breg26: case DW_OP_breg27: case DW_OP_breg28: case DW_OP_breg29: case DW_OP_breg30: case DW_OP_breg31:{ // Push register + constant: int register_id = simgrid::dwarf::dwarf_register_to_libunwind( op->atom - DW_OP_breg0); unw_word_t res; if (not context.cursor) throw evaluation_error("Missing stack context"); unw_get_reg(context.cursor, register_id, &res); stack.push(res + op->number); break; } // Push the CFA (Canonical Frame Address): case DW_OP_call_frame_cfa: { /* See 6.4 of DWARF4 (http://dwarfstd.org/doc/DWARF4.pdf#page=140): * * > Typically, the CFA is defined to be the value of the stack * > pointer at the call site in the previous frame (which may be * > different from its value on entry to the current frame). * * We need to unwind the frame in order to get the SP of the parent * frame. * * Warning: the CFA returned by libunwind (UNW_X86_64_RSP, etc.) * is the SP of the *current* frame. */ if (not context.cursor) throw evaluation_error("Missint cursor"); // Get frame: unw_cursor_t cursor = *(context.cursor); unw_step(&cursor); unw_word_t res; unw_get_reg(&cursor, UNW_REG_SP, &res); stack.push(res); break; } // Frame base: case DW_OP_fbreg: stack.push((std::uintptr_t) context.frame_base + op->number); break; // ***** Constants: // Short constant literals: case DW_OP_lit0: case DW_OP_lit1: case DW_OP_lit2: case DW_OP_lit3: case DW_OP_lit4: case DW_OP_lit5: case DW_OP_lit6: case DW_OP_lit7: case DW_OP_lit8: case DW_OP_lit9: case DW_OP_lit10: case DW_OP_lit11: case DW_OP_lit12: case DW_OP_lit13: case DW_OP_lit14: case DW_OP_lit15: case DW_OP_lit16: case DW_OP_lit17: case DW_OP_lit18: case DW_OP_lit19: case DW_OP_lit20: case DW_OP_lit21: case DW_OP_lit22: case DW_OP_lit23: case DW_OP_lit24: case DW_OP_lit25: case DW_OP_lit26: case DW_OP_lit27: case DW_OP_lit28: case DW_OP_lit29: case DW_OP_lit30: case DW_OP_lit31: // Push a literal/constant on the stack: stack.push(atom - DW_OP_lit0); break; // Address from the base address of this ELF object. // Push the address on the stack (base_address + argument). case DW_OP_addr: { if (not context.object_info) throw evaluation_error("No base address"); Dwarf_Off addr = (Dwarf_Off) (std::uintptr_t) context.object_info->base_address() + op->number; stack.push(addr); break; } // General constants: // Push the constant argument on the stack. case DW_OP_const1u: case DW_OP_const2u: case DW_OP_const4u: case DW_OP_const8u: case DW_OP_const1s: case DW_OP_const2s: case DW_OP_const4s: case DW_OP_const8s: case DW_OP_constu: case DW_OP_consts: stack.push(op->number); break; // ***** Stack manipulation: // Push another copy/duplicate the value at the top of the stack: case DW_OP_dup: stack.dup(); break; // Pop/drop the top of the stack: case DW_OP_drop: stack.pop(); break; case DW_OP_swap: stack.swap(); break; // Duplicate the value under the top of the stack: case DW_OP_over: stack.push(stack.top(1)); break; // ***** Operations: // Those usually take the top of the stack and the next value as argument // and replace the top of the stack with the computed value // (stack.top() += stack.before_top()). case DW_OP_plus: first = stack.pop(); second = stack.pop(); stack.push(first + second); break; case DW_OP_mul: first = stack.pop(); second = stack.pop(); stack.push(first * second); break; case DW_OP_plus_uconst: stack.top() += op->number; break; case DW_OP_not: stack.top() = ~stack.top(); break; case DW_OP_neg: stack.top() = - (intptr_t) stack.top(); break; case DW_OP_minus: first = stack.pop(); second = stack.pop(); stack.push(second - first); break; case DW_OP_and: first = stack.pop(); second = stack.pop(); stack.push(first & second); break; case DW_OP_or: first = stack.pop(); second = stack.pop(); stack.push(first | second); break; case DW_OP_xor: first = stack.pop(); second = stack.pop(); stack.push(first ^ second); break; case DW_OP_nop: break; // ***** Deference (memory fetch) case DW_OP_deref_size: throw evaluation_error("Unsupported operation"); case DW_OP_deref: // Computed address: if (not context.address_space) throw evaluation_error("Missing address space"); context.address_space->read_bytes( &stack.top(), sizeof(uintptr_t), remote(stack.top()), context.process_index); break; // Not handled: default: throw evaluation_error("Unsupported operation"); } } } } } SimGrid-3.18/src/mc/mc_record.cpp0000644000175000017500000000633113217757316017155 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "xbt/log.h" #include "xbt/sysdep.h" #include "simgrid/simix.h" #include "src/kernel/context/Context.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/mc_replay.hpp" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_private.hpp" #include "src/mc/mc_base.h" #include "src/mc/Transition.hpp" #if SIMGRID_HAVE_MC #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_state.hpp" #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_record, mc, " Logging specific to MC record/replay facility"); std::string MC_record_path; namespace simgrid { namespace mc { void replay(RecordTrace const& trace) { simgrid::mc::wait_for_requests(); for (simgrid::mc::Transition const& transition : trace) { XBT_DEBUG("Executing %i$%i", transition.pid, transition.argument); // Choose a request: smx_actor_t process = SIMIX_process_from_PID(transition.pid); if (not process) xbt_die("Unexpected process (pid:%d).", transition.pid); smx_simcall_t simcall = &(process->simcall); if (not simcall || simcall->call == SIMCALL_NONE) xbt_die("No simcall for process %d.", transition.pid); if (not simgrid::mc::request_is_visible(simcall) || not simgrid::mc::actor_is_enabled(process)) xbt_die("Unexpected simcall."); // Execute the request: SIMIX_simcall_handle(simcall, transition.argument); simgrid::mc::wait_for_requests(); } } void replay(std::string path_string) { simgrid::mc::processes_time.resize(SIMIX_process_get_maxpid()); simgrid::mc::RecordTrace trace = simgrid::mc::parseRecordTrace(path_string.c_str()); simgrid::mc::replay(trace); simgrid::mc::processes_time.clear(); } RecordTrace parseRecordTrace(const char* data) { RecordTrace res; XBT_INFO("path=%s", data); if (data == nullptr || data[0] == '\0') throw std::invalid_argument("Could not parse record path"); const char* current = data; while (*current) { simgrid::mc::Transition item; int count = sscanf(current, "%d/%d", &item.pid, &item.argument); if(count != 2 && count != 1) throw std::invalid_argument("Could not parse record path"); res.push_back(item); // Find next chunk: const char* end = std::strchr(current, ';'); if(end == nullptr) break; else current = end + 1; } return res; } #if SIMGRID_HAVE_MC std::string traceToString(simgrid::mc::RecordTrace const& trace) { std::ostringstream stream; for (auto i = trace.begin(); i != trace.end(); ++i) { if (i != trace.begin()) stream << ';'; stream << i->pid; if (i->argument) stream << '/' << i->argument; } return stream.str(); } void dumpRecordPath() { if (MC_record_is_active()) { RecordTrace trace = mc_model_checker->getChecker()->getRecordTrace(); XBT_INFO("Path = %s", traceToString(trace).c_str()); } } #endif } } SimGrid-3.18/src/mc/mc_checkpoint.cpp0000644000175000017500000004732313217757316020034 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #ifndef WIN32 #include #endif #include "src/internal_config.h" #include "src/mc/mc_private.hpp" #include "src/smpi/include/private.hpp" #include "xbt/file.hpp" #include "xbt/mmalloc.h" #include "xbt/module.h" #include "src/xbt/mmalloc/mmprivate.h" #include "src/simix/smx_private.hpp" #include #include #include "src/mc/mc_private.hpp" #include #include "src/mc/mc_hash.hpp" #include "src/mc/mc_mmu.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_snapshot.hpp" #include "src/mc/mc_unw.hpp" #include "src/mc/remote/mc_protocol.h" #include "src/mc/RegionSnapshot.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Frame.hpp" #include "src/mc/Variable.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_checkpoint, mc, "Logging specific to mc_checkpoint"); #define PROT_RWX (PROT_READ | PROT_WRITE | PROT_EXEC) #define PROT_RW (PROT_READ | PROT_WRITE) #define PROT_RX (PROT_READ | PROT_EXEC) extern std::string _sg_mc_property_file; extern std::string _sg_mc_dot_output_file; namespace simgrid { namespace mc { /************************************ Free functions **************************************/ /*****************************************************************************************/ /** @brief Restore a region from a snapshot * * @param region Target region */ static void restore(mc_mem_region_t region) { switch(region->storage_type()) { case simgrid::mc::StorageType::Flat: mc_model_checker->process().write_bytes(region->flat_data().get(), region->size(), region->permanent_address()); break; case simgrid::mc::StorageType::Chunked: mc_region_restore_sparse(&mc_model_checker->process(), region); break; case simgrid::mc::StorageType::Privatized: for (auto& p : region->privatized_data()) restore(&p); break; default: // includes StorageType::NoData xbt_die("Storage type not supported"); break; } } #if HAVE_SMPI RegionSnapshot privatized_region( RegionType region_type, void *start_addr, void* permanent_addr, std::size_t size ) { size_t process_count = MC_smpi_process_count(); // Read smpi_privatization_regions from MCed: smpi_privatization_region_t remote_smpi_privatization_regions; mc_model_checker->process().read_variable( "smpi_privatization_regions", &remote_smpi_privatization_regions, sizeof(remote_smpi_privatization_regions)); s_smpi_privatization_region_t privatization_regions[process_count]; mc_model_checker->process().read_bytes( &privatization_regions, sizeof(privatization_regions), remote(remote_smpi_privatization_regions)); std::vector data; data.reserve(process_count); for (size_t i = 0; i < process_count; i++) data.push_back(simgrid::mc::region(region_type, start_addr, privatization_regions[i].address, size)); simgrid::mc::RegionSnapshot region = simgrid::mc::RegionSnapshot( region_type, start_addr, permanent_addr, size); region.privatized_data(std::move(data)); return region; } #endif static void add_region(int index, simgrid::mc::Snapshot* snapshot, simgrid::mc::RegionType type, simgrid::mc::ObjectInformation* object_info, void *start_addr, void* permanent_addr, std::size_t size) { if (type == simgrid::mc::RegionType::Data) xbt_assert(object_info, "Missing object info for object."); else if (type == simgrid::mc::RegionType::Heap) xbt_assert(not object_info, "Unexpected object info for heap region."); simgrid::mc::RegionSnapshot region; #if HAVE_SMPI const bool privatization_aware = object_info && mc_model_checker->process().privatized(*object_info); if (privatization_aware && MC_smpi_process_count()) region = simgrid::mc::privatized_region( type, start_addr, permanent_addr, size); else #endif region = simgrid::mc::region(type, start_addr, permanent_addr, size); region.object_info(object_info); snapshot->snapshot_regions[index] = std::unique_ptr( new simgrid::mc::RegionSnapshot(std::move(region))); } static void get_memory_regions(simgrid::mc::RemoteClient* process, simgrid::mc::Snapshot* snapshot) { const size_t n = process->object_infos.size(); snapshot->snapshot_regions.resize(n + 1); int i = 0; for (auto const& object_info : process->object_infos) add_region(i++, snapshot, simgrid::mc::RegionType::Data, object_info.get(), object_info->start_rw, object_info->start_rw, object_info->end_rw - object_info->start_rw); xbt_mheap_t heap = process->get_heap(); void *start_heap = heap->base; void *end_heap = heap->breakval; add_region(n, snapshot, simgrid::mc::RegionType::Heap, nullptr, start_heap, start_heap, (char *) end_heap - (char *) start_heap); snapshot->heap_bytes_used = mmalloc_get_bytes_used_remote( heap->heaplimit, process->get_malloc_info()); #if HAVE_SMPI if (mc_model_checker->process().privatized() && MC_smpi_process_count()) // snapshot->privatization_index = smpi_loaded_page mc_model_checker->process().read_variable( "smpi_loaded_page", &snapshot->privatization_index, sizeof(snapshot->privatization_index)); else #endif snapshot->privatization_index = simgrid::mc::ProcessIndexMissing; } /** \brief Fills the position of the segments (executable, read-only, read/write). * */ // TODO, use the ELF segment information for more robustness void find_object_address( std::vector const& maps, simgrid::mc::ObjectInformation* result) { std::string name = simgrid::xbt::Path(result->file_name).getBasename(); for (size_t i = 0; i < maps.size(); ++i) { simgrid::xbt::VmMap const& reg = maps[i]; if (maps[i].pathname.empty()) continue; std::string map_basename = simgrid::xbt::Path(maps[i].pathname).getBasename(); if (map_basename != name) continue; // This is the non-GNU_RELRO-part of the data segment: if (reg.prot == PROT_RW) { xbt_assert(not result->start_rw, "Multiple read-write segments for %s, not supported", maps[i].pathname.c_str()); result->start_rw = (char*) reg.start_addr; result->end_rw = (char*) reg.end_addr; // The next VMA might be end of the data segment: if (i + 1 < maps.size() && maps[i + 1].pathname.empty() && maps[i + 1].prot == PROT_RW && maps[i + 1].start_addr == reg.end_addr) result->end_rw = (char*) maps[i + 1].end_addr; } // This is the text segment: else if (reg.prot == PROT_RX) { xbt_assert(not result->start_exec, "Multiple executable segments for %s, not supported", maps[i].pathname.c_str()); result->start_exec = (char*) reg.start_addr; result->end_exec = (char*) reg.end_addr; // The next VMA might be end of the data segment: if (i + 1 < maps.size() && maps[i + 1].pathname.empty() && maps[i + 1].prot == PROT_RW && maps[i + 1].start_addr == reg.end_addr) { result->start_rw = (char*) maps[i + 1].start_addr; result->end_rw = (char*) maps[i + 1].end_addr; } } // This is the GNU_RELRO-part of the data segment: else if (reg.prot == PROT_READ) { xbt_assert(not result->start_ro, "Multiple read only segments for %s, not supported", maps[i].pathname.c_str()); result->start_ro = (char*) reg.start_addr; result->end_ro = (char*) reg.end_addr; } } result->start = result->start_rw; if ((const void*) result->start_ro < result->start) result->start = result->start_ro; if ((const void*) result->start_exec < result->start) result->start = result->start_exec; result->end = result->end_rw; if (result->end_ro && (const void*) result->end_ro > result->end) result->end = result->end_ro; if (result->end_exec && (const void*) result->end_exec > result->end) result->end = result->end_exec; xbt_assert(result->start_exec || result->start_rw || result->start_ro); } /************************************* Take Snapshot ************************************/ /****************************************************************************************/ /** \brief Checks whether the variable is in scope for a given IP. * * A variable may be defined only from a given value of IP. * * \param var Variable description * \param scope Scope description * \param ip Instruction pointer * \return true if the variable is valid * */ static bool valid_variable(simgrid::mc::Variable* var, simgrid::mc::Frame* scope, const void *ip) { // The variable is not yet valid: if (scope->range.begin() + var->start_scope > (std::uint64_t) ip) return false; else return true; } static void fill_local_variables_values(mc_stack_frame_t stack_frame, simgrid::mc::Frame* scope, int process_index, std::vector& result) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); if (not scope || not scope->range.contain(stack_frame->ip)) return; for (simgrid::mc::Variable& current_variable : scope->variables) { if (not valid_variable(¤t_variable, scope, (void*)stack_frame->ip)) continue; int region_type; // FIXME, get rid of `region_type` if ((long) stack_frame->ip > (long) process->libsimgrid_info->start_exec) region_type = 1; else region_type = 2; s_local_variable_t new_var; new_var.subprogram = stack_frame->frame; new_var.ip = stack_frame->ip; new_var.name = current_variable.name; new_var.type = current_variable.type; new_var.region = region_type; new_var.address = nullptr; if (current_variable.address != nullptr) new_var.address = current_variable.address; else if (not current_variable.location_list.empty()) { simgrid::dwarf::Location location = simgrid::dwarf::resolve( current_variable.location_list, current_variable.object_info, &(stack_frame->unw_cursor), (void *) stack_frame->frame_base, &mc_model_checker->process(), process_index); if (not location.in_memory()) xbt_die("Cannot handle non-address variable"); new_var.address = location.address(); } else xbt_die("No address"); result.push_back(std::move(new_var)); } // Recursive processing of nested scopes: for (simgrid::mc::Frame& nested_scope : scope->scopes) fill_local_variables_values( stack_frame, &nested_scope, process_index, result); } static std::vector get_local_variables_values(std::vector& stack_frames, int process_index) { std::vector variables; for (s_mc_stack_frame_t& stack_frame : stack_frames) fill_local_variables_values(&stack_frame, stack_frame.frame, process_index, variables); return variables; } static std::vector unwind_stack_frames(simgrid::mc::UnwindContext* stack_context) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); std::vector result; unw_cursor_t c = stack_context->cursor(); // TODO, check condition check (unw_init_local==0 means end of frame) while (1) { s_mc_stack_frame_t stack_frame; stack_frame.unw_cursor = c; unw_word_t ip; unw_word_t sp; unw_get_reg(&c, UNW_REG_IP, &ip); unw_get_reg(&c, UNW_REG_SP, &sp); stack_frame.ip = ip; stack_frame.sp = sp; // TODO, use real addresses in frame_t instead of fixing it here simgrid::mc::Frame* frame = process->find_function(remote(ip)); stack_frame.frame = frame; if (frame) { stack_frame.frame_name = frame->name; stack_frame.frame_base = (unw_word_t) frame->frame_base(c); } else { stack_frame.frame_base = 0; stack_frame.frame_name = std::string(); } result.push_back(std::move(stack_frame)); /* Stop before context switch with maestro */ if (frame != nullptr && frame->name == "smx_ctx_sysv_wrapper") break; int ret = unw_step(&c); if (ret == 0) xbt_die("Unexpected end of stack."); else if (ret < 0) xbt_die("Error while unwinding stack"); } if (result.empty()) { XBT_INFO("unw_init_local failed"); xbt_abort(); } return result; } static std::vector take_snapshot_stacks(simgrid::mc::Snapshot* snapshot) { std::vector res; for (auto const& stack : mc_model_checker->process().stack_areas()) { s_mc_snapshot_stack_t st; // Read the context from remote process: unw_context_t context; mc_model_checker->process().read_bytes( &context, sizeof(context), remote(stack.context)); st.context.initialize(&mc_model_checker->process(), &context); st.stack_frames = unwind_stack_frames(&st.context); st.local_variables = get_local_variables_values(st.stack_frames, stack.process_index); st.process_index = stack.process_index; unw_word_t sp = st.stack_frames[0].sp; res.push_back(std::move(st)); size_t stack_size = (char*) stack.address + stack.size - (char*) sp; snapshot->stack_sizes.push_back(stack_size); } return res; } static void snapshot_handle_ignore(simgrid::mc::Snapshot* snapshot) { xbt_assert(snapshot->process()); // Copy the memory: for (auto const& region : mc_model_checker->process().ignored_regions()) { s_mc_snapshot_ignored_data_t ignored_data; ignored_data.start = (void*)region.addr; ignored_data.data.resize(region.size); // TODO, we should do this once per privatization segment: snapshot->process()->read_bytes( ignored_data.data.data(), region.size, remote(region.addr), simgrid::mc::ProcessIndexDisabled); snapshot->ignored_data.push_back(std::move(ignored_data)); } // Zero the memory: for (auto const& region : mc_model_checker->process().ignored_regions()) snapshot->process()->clear_bytes(remote(region.addr), region.size); } static void snapshot_ignore_restore(simgrid::mc::Snapshot* snapshot) { for (auto const& ignored_data : snapshot->ignored_data) snapshot->process()->write_bytes( ignored_data.data.data(), ignored_data.data.size(), remote(ignored_data.start)); } static std::vector get_current_fds(pid_t pid) { const size_t fd_dir_path_size = 20; char fd_dir_path[fd_dir_path_size]; int res = snprintf(fd_dir_path, fd_dir_path_size, "/proc/%lli/fd", (long long int) pid); xbt_assert(res >= 0); if ((size_t) res > fd_dir_path_size) xbt_die("Unexpected buffer is too small for fd_dir_path"); DIR* fd_dir = opendir(fd_dir_path); if (fd_dir == nullptr) xbt_die("Cannot open directory '/proc/self/fd'\n"); std::vector fds; struct dirent* fd_number; while ((fd_number = readdir(fd_dir))) { int fd_value = xbt_str_parse_int(fd_number->d_name, "Found a non-numerical FD: %s. Freaking out!"); if(fd_value < 3) continue; const size_t source_size = 25; char source[25]; int res = snprintf(source, source_size, "/proc/%lli/fd/%s", (long long int) pid, fd_number->d_name); xbt_assert(res >= 0); if ((size_t) res > source_size) xbt_die("Unexpected buffer is too small for fd %s", fd_number->d_name); const size_t link_size = 200; char link[200]; res = readlink(source, link, link_size); if (res<0) xbt_die("Could not read link for %s", source); if (res==200) xbt_die("Buffer to small for link of %s", source); link[res] = '\0'; #if HAVE_SMPI if(smpi_is_privatization_file(link)) continue; #endif // This is (probably) the DIR* we are reading: // TODO, read all the file entries at once and close the DIR.* if(strcmp(fd_dir_path, link) == 0) continue; // We don't handle them. // It does not mean we should silently ignore them however. if (strncmp(link, "pipe:", std::strlen("pipe:")) == 0 || strncmp(link, "socket:", std::strlen("socket:")) == 0) continue; // If dot_output enabled, do not handle the corresponding file if (dot_output != nullptr) { std::string link_basename = simgrid::xbt::Path(link).getBasename(); if (link_basename == _sg_mc_dot_output_file) continue; } // This is probably a shared memory used by lttng-ust: if(strncmp("/dev/shm/ust-shm-tmp-", link, std::strlen("/dev/shm/ust-shm-tmp-"))==0) continue; // Add an entry for this FD in the snapshot: s_fd_infos_t fd; fd.filename = std::string(link); fd.number = fd_value; fd.flags = fcntl(fd_value, F_GETFL) | fcntl(fd_value, F_GETFD) ; fd.current_position = lseek(fd_value, 0, SEEK_CUR); fds.push_back(std::move(fd)); } closedir (fd_dir); return fds; } std::shared_ptr take_snapshot(int num_state) { XBT_DEBUG("Taking snapshot %i", num_state); simgrid::mc::RemoteClient* mc_process = &mc_model_checker->process(); std::shared_ptr snapshot = std::make_shared(mc_process, num_state); for (auto const& p : mc_model_checker->process().actors()) snapshot->enabled_processes.insert(p.copy.getBuffer()->pid); snapshot_handle_ignore(snapshot.get()); if (_sg_mc_snapshot_fds) snapshot->current_fds = get_current_fds(mc_model_checker->process().pid()); /* Save the std heap and the writable mapped pages of libsimgrid and binary */ get_memory_regions(mc_process, snapshot.get()); snapshot->to_ignore = mc_model_checker->process().ignored_heap(); if (_sg_mc_max_visited_states > 0 || not _sg_mc_property_file.empty()) { snapshot->stacks = take_snapshot_stacks(snapshot.get()); if (_sg_mc_hash) snapshot->hash = simgrid::mc::hash(*snapshot); else snapshot->hash = 0; } else snapshot->hash = 0; snapshot_ignore_restore(snapshot.get()); return snapshot; } static inline void restore_snapshot_regions(simgrid::mc::Snapshot* snapshot) { for (std::unique_ptr const& region : snapshot->snapshot_regions) { // For privatized, variables we decided it was not necessary to take the snapshot: if (region) restore(region.get()); } #if HAVE_SMPI if(snapshot->privatization_index >= 0) { // Fix the privatization mmap: s_mc_message_restore_t message{MC_MESSAGE_RESTORE, snapshot->privatization_index}; mc_model_checker->process().getChannel().send(message); } #endif } static inline void restore_snapshot_fds(simgrid::mc::Snapshot* snapshot) { xbt_die("FD snapshot not implemented in client/server mode."); for (auto const& fd : snapshot->current_fds) { int new_fd = open(fd.filename.c_str(), fd.flags); if (new_fd < 0) xbt_die("Could not reopen the file %s fo restoring the file descriptor", fd.filename.c_str()); if (new_fd != fd.number) { dup2(new_fd, fd.number); close(new_fd); } lseek(fd.number, fd.current_position, SEEK_SET); } } void restore_snapshot(std::shared_ptr snapshot) { XBT_DEBUG("Restore snapshot %i", snapshot->num_state); restore_snapshot_regions(snapshot.get()); if (_sg_mc_snapshot_fds) restore_snapshot_fds(snapshot.get()); snapshot_ignore_restore(snapshot.get()); mc_model_checker->process().clear_cache(); } } } SimGrid-3.18/src/mc/mc_state.hpp0000644000175000017500000001023013217757316017015 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_STATE_HPP #define SIMGRID_MC_STATE_HPP #include #include #include "src/mc/Transition.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/mc_snapshot.hpp" namespace simgrid { namespace mc { enum class PatternCommunicationType { none = 0, send = 1, receive = 2, }; class PatternCommunication { public: int num = 0; simgrid::kernel::activity::CommImpl* comm_addr; PatternCommunicationType type = PatternCommunicationType::send; unsigned long src_proc = 0; unsigned long dst_proc = 0; const char* src_host = nullptr; const char* dst_host = nullptr; std::string rdv; std::vector data; int tag = 0; int index = 0; PatternCommunication() { std::memset(&comm_addr, 0, sizeof(comm_addr)); } PatternCommunication dup() const { simgrid::mc::PatternCommunication res; // num? res.comm_addr = this->comm_addr; res.type = this->type; // src_proc? // dst_proc? res.dst_proc = this->dst_proc; res.dst_host = this->dst_host; res.rdv = this->rdv; res.data = this->data; // tag? res.index = this->index; return res; } }; /* On every state, each process has an entry of the following type. * This represents both the process and its transition because * a process cannot have more than one enabled transition at a given time. */ class ProcessState { /* Possible exploration status of a process transition in a state. * Either the checker did not consider the transition, or it was considered and to do, or considered and done. */ enum class InterleavingType { /** This process transition is not considered by the checker (yet?) */ disabled = 0, /** The checker algorithm decided that this process transitions should be done at some point */ todo, /** The checker algorithm decided that this should be done, but it was done in the meanwhile */ done, }; /** Exploration control information */ InterleavingType state = InterleavingType::disabled; public: /** Number of times that the process was considered to be executed */ // TODO, make this private unsigned int times_considered = 0; bool isDisabled() const { return this->state == InterleavingType::disabled; } bool isDone() const { return this->state == InterleavingType::done; } bool isTodo() const { return this->state == InterleavingType::todo; } /** Mark that we should try executing this process at some point in the future of the checker algorithm */ void consider() { this->state = InterleavingType::todo; this->times_considered = 0; } void setDone() { this->state = InterleavingType::done; } }; /* A node in the exploration graph (kind-of) */ class XBT_PRIVATE State { public: /** Sequential state number (used for debugging) */ int num = 0; /** State's exploration status by process */ std::vector actorStates; Transition transition; /** The simcall which was executed, going out of that state */ s_smx_simcall_t executed_req; /* Internal translation of the executed_req simcall * * SIMCALL_COMM_TESTANY is translated to a SIMCALL_COMM_TEST * and SIMCALL_COMM_WAITANY to a SIMCALL_COMM_WAIT. */ s_smx_simcall_t internal_req; /* Can be used as a copy of the remote synchro object */ simgrid::mc::Remote internal_comm; /** Snapshot of system state (if needed) */ std::shared_ptr system_state; // For CommunicationDeterminismChecker std::vector> incomplete_comm_pattern; std::vector communicationIndices; explicit State(unsigned long state_number); std::size_t interleaveSize() const; void addInterleavingSet(smx_actor_t actor) { this->actorStates[actor->pid].consider(); } Transition getTransition() const; }; } } XBT_PRIVATE smx_simcall_t MC_state_get_request(simgrid::mc::State* state); #endif SimGrid-3.18/src/mc/mc_snapshot.cpp0000644000175000017500000002415213217757316017537 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "xbt/asserts.h" #include "xbt/sysdep.h" #include "src/internal_config.h" #include "src/smpi/include/private.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/mc_mmu.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_snapshot.hpp" extern "C" { /** @brief Find the snapshoted region from a pointer * * @param addr Pointer * @param snapshot Snapshot * @param Snapshot region in the snapshot this pointer belongs to * (or nullptr if it does not belong to any snapshot region) * */ mc_mem_region_t mc_get_snapshot_region( const void* addr, const simgrid::mc::Snapshot* snapshot, int process_index) { size_t n = snapshot->snapshot_regions.size(); for (size_t i = 0; i != n; ++i) { mc_mem_region_t region = snapshot->snapshot_regions[i].get(); if (not(region && region->contain(simgrid::mc::remote(addr)))) continue; if (region->storage_type() == simgrid::mc::StorageType::Privatized) { #if HAVE_SMPI // Use the current process index of the snapshot: if (process_index == simgrid::mc::ProcessIndexDisabled) process_index = snapshot->privatization_index; if (process_index < 0) xbt_die("Missing process index"); if (process_index >= (int) region->privatized_data().size()) xbt_die("Invalid process index"); simgrid::mc::RegionSnapshot& priv_region = region->privatized_data()[process_index]; xbt_assert(priv_region.contain(simgrid::mc::remote(addr))); return &priv_region; #else xbt_die("Privatized region in a non SMPI build (this should not happen)"); #endif } return region; } return nullptr; } /** @brief Read memory from a snapshot region broken across fragmented pages * * @param addr Process (non-snapshot) address of the data * @param region Snapshot memory region where the data is located * @param target Buffer to store the value * @param size Size of the data to read in bytes * @return Pointer where the data is located (target buffer of original location) */ const void* MC_region_read_fragmented(mc_mem_region_t region, void* target, const void* addr, size_t size) { // Last byte of the memory area: void* end = (char*) addr + size - 1; // TODO, we assume the chunks are aligned to natural chunk boundaries. // We should remove this assumption. // Page of the last byte of the memory area: size_t page_end = simgrid::mc::mmu::split((std::uintptr_t) end).first; void* dest = target; if (dest==nullptr) xbt_die("Missing destination buffer for fragmented memory access"); // Read each page: while (simgrid::mc::mmu::split((std::uintptr_t) addr).first != page_end) { void* snapshot_addr = mc_translate_address_region_chunked((uintptr_t) addr, region); void* next_page = (void*) simgrid::mc::mmu::join( simgrid::mc::mmu::split((std::uintptr_t) addr).first + 1, 0); size_t readable = (char*) next_page - (char*) addr; memcpy(dest, snapshot_addr, readable); addr = (char*) addr + readable; dest = (char*) dest + readable; size -= readable; } // Read the end: void* snapshot_addr = mc_translate_address_region_chunked((uintptr_t)addr, region); memcpy(dest, snapshot_addr, size); return target; } /** Compare memory between snapshots (with known regions) * * @param addr1 Address in the first snapshot * @param snapshot2 Region of the address in the first snapshot * @param addr2 Address in the second snapshot * @param snapshot2 Region of the address in the second snapshot * @return same as memcmp * */ int MC_snapshot_region_memcmp( const void* addr1, mc_mem_region_t region1, const void* addr2, mc_mem_region_t region2, size_t size) { // Using alloca() for large allocations may trigger stack overflow: // use malloc if the buffer is too big. bool stack_alloc = size < 64; const bool region1_need_buffer = region1==nullptr || region1->storage_type()==simgrid::mc::StorageType::Flat; const bool region2_need_buffer = region2==nullptr || region2->storage_type()==simgrid::mc::StorageType::Flat; void* buffer1a = region1_need_buffer ? nullptr : stack_alloc ? alloca(size) : ::operator new(size); void* buffer2a = region2_need_buffer ? nullptr : stack_alloc ? alloca(size) : ::operator new(size); const void* buffer1 = MC_region_read(region1, buffer1a, addr1, size); const void* buffer2 = MC_region_read(region2, buffer2a, addr2, size); int res; if (buffer1 == buffer2) res = 0; else res = memcmp(buffer1, buffer2, size); if (not stack_alloc) { ::operator delete(buffer1a); ::operator delete(buffer2a); } return res; } /** Compare memory between snapshots * * @param addr1 Address in the first snapshot * @param snapshot1 First snapshot * @param addr2 Address in the second snapshot * @param snapshot2 Second snapshot * @return same as memcmp * */ int MC_snapshot_memcmp( const void* addr1, simgrid::mc::Snapshot* snapshot1, const void* addr2, simgrid::mc::Snapshot* snapshot2, int process_index, size_t size) { mc_mem_region_t region1 = mc_get_snapshot_region(addr1, snapshot1, process_index); mc_mem_region_t region2 = mc_get_snapshot_region(addr2, snapshot2, process_index); return MC_snapshot_region_memcmp(addr1, region1, addr2, region2, size); } } // extern "C" namespace simgrid { namespace mc { Snapshot::Snapshot(RemoteClient* process, int _num_state) : AddressSpace(process) , num_state(_num_state) , heap_bytes_used(0) , enabled_processes() , privatization_index(0) , hash(0) { } const void* Snapshot::read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index, ReadOptions options) const { mc_mem_region_t region = mc_get_snapshot_region((void*)address.address(), this, process_index); if (region) { const void* res = MC_region_read(region, buffer, (void*)address.address(), size); if (buffer == res || options & ReadOptions::lazy()) return res; else { memcpy(buffer, res, size); return buffer; } } else return this->process()->read_bytes( buffer, size, address, process_index, options); } } } #ifdef SIMGRID_TEST #include #include #include #include "src/mc/mc_mmu.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_snapshot.hpp" XBT_TEST_SUITE("mc_snapshot", "Snapshots"); static inline void init_memory(void* mem, size_t size) { char* dest = (char*) mem; for (size_t i = 0; i < size; ++i) { dest[i] = rand() & 255; } } static void test_snapshot(bool sparse_checkpoint); XBT_TEST_UNIT("flat_snapshot", test_flat_snapshots, "Test flat snapshots") { test_snapshot(0); } XBT_TEST_UNIT("page_snapshots", test_per_snpashots, "Test per-page snapshots") { test_snapshot(1); } static void test_snapshot(bool sparse_checkpoint) { xbt_test_add("Initialization"); _sg_mc_sparse_checkpoint = sparse_checkpoint; xbt_assert(xbt_pagesize == getpagesize()); xbt_assert(1 << xbt_pagebits == xbt_pagesize); std::unique_ptr process(new simgrid::mc::RemoteClient(getpid(), -1)); process->init(); mc_model_checker = new ::simgrid::mc::ModelChecker(std::move(process)); for(int n=1; n!=256; ++n) { // Store region page(s): size_t byte_size = n * xbt_pagesize; void* source = mmap(nullptr, byte_size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); xbt_assert(source!=MAP_FAILED, "Could not allocate source memory"); // Init memory and take snapshots: init_memory(source, byte_size); simgrid::mc::RegionSnapshot region0 = simgrid::mc::sparse_region( simgrid::mc::RegionType::Unknown, source, source, byte_size); for(int i=0; i #include #include #include #include #include "src/mc/mc_unw.hpp" #include "src/mc/remote/RemoteClient.hpp" /** \file * Libunwind namespace implementation using process_vm_readv. *. * This implem */ /** Partial structure of libunwind-ptrace context in order to get the PID * * HACK, The context type for libunwind-race is an opaque type. * We need to get the PID which is the first field. This is a hack * which might break if the libunwind-ptrace structure changes. */ struct _UPT_info { pid_t pid; // Other things... }; /** Get the PID of a `libunwind-ptrace` context */ static inline pid_t _UPT_getpid(void* arg) { _UPT_info* info = static_cast<_UPT_info*>(arg); return info->pid; } /** Read from the memory, avoid using `ptrace` (libunwind method) */ static int access_mem(const unw_addr_space_t as, const unw_word_t addr, unw_word_t* const valp, const int write, void* const arg) { if (write) return - UNW_EINVAL; pid_t pid = _UPT_getpid(arg); size_t size = sizeof(unw_word_t); #if HAVE_PROCESS_VM_READV // process_vm_read implementation. // This is only available since Linux 3.2. struct iovec local = { valp, size }; struct iovec remote = { (void*) addr, size }; ssize_t s = process_vm_readv(pid, &local, 1, &remote, 1, 0); if (s >= 0) { if ((size_t) s != size) return - UNW_EINVAL; else return 0; } if (s < 0 && errno != ENOSYS) return - UNW_EINVAL; #endif // /proc/${pid}/mem implementation. // On recent kernels, we do not need to ptrace the target process. // On older kernels, it is necessary to ptrace the target process. size_t count = size; off_t off = (off_t) addr; char* buf = (char*) valp; int fd = simgrid::mc::open_vm(pid, O_RDONLY); if (fd < 0) return - UNW_EINVAL; while (1) { ssize_t s = pread(fd, buf, count, off); if (s == 0) { close(fd); return - UNW_EINVAL; } if (s == -1) break; count -= s; buf += s; off += s; if (count == 0) { close(fd); return 0; } } close(fd); // ptrace implementation. // We need to have PTRACE_ATTACH-ed it before. return _UPT_access_mem(as, addr, valp, write, arg); } namespace simgrid { namespace unw { /** Virtual table for our `libunwind-process_vm_readv` implementation. * * This implementation reuse most the code of `libunwind-ptrace` but * does not use ptrace() to read the target process memory by * `process_vm_readv()` or `/dev/${pid}/mem` if possible. * * Does not support any MC-specific behaviour (privatization, snapshots) * and `ucontext_t`. * * It works with `void*` contexts allocated with `_UPT_create(pid)`. */ // TODO, we could get rid of this if we properly stop the model-checked // process before reading the memory. static unw_accessors_t accessors = { &_UPT_find_proc_info, &_UPT_put_unwind_info, &_UPT_get_dyn_info_list_addr, &access_mem, &_UPT_access_reg, &_UPT_access_fpreg, &_UPT_resume, &_UPT_get_proc_name }; unw_addr_space_t create_addr_space() { return unw_create_addr_space(&accessors, BYTE_ORDER); } void* create_context(unw_addr_space_t as, pid_t pid) { return _UPT_create(pid); } } } SimGrid-3.18/src/mc/Session.hpp0000644000175000017500000000437713217757316016660 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SESSION_HPP #define SIMGRID_MC_SESSION_HPP #ifdef __linux__ #include #endif #include "xbt/sysdep.h" #include "xbt/system_error.hpp" #include #include #include #include "xbt/log.h" #include "src/mc/mc_forward.hpp" #include "src/mc/ModelChecker.hpp" namespace simgrid { namespace mc { /** A model-checking session * * This is expected to become the interface used by model-checking * algorithms to control the execution of the model-checked process * and the exploration of the execution graph. Model-checking * algorithms should be able to be written in high-level languages * (e.g. Python) using bindings on this interface. */ class Session { private: std::unique_ptr modelChecker_; std::shared_ptr initialSnapshot_; Session(pid_t pid, int socket); // No copy: Session(Session const&) = delete; Session& operator=(Session const&) = delete; public: ~Session(); void close(); void initialize(); void execute(Transition const& transition); void logState(); void restoreInitialState(); // static constructors /** Create a new session by forking * * This sets up the environment for the model-checked process * (environoment variables, sockets, etc.). * * The code is expected to `exec` the model-checker program. */ static Session* fork(std::function code); /** Spawn a model-checked process * * @param path full path of the executable * @param argv arguments for the model-checked process (NULL-terminated) */ static Session* spawnv(const char *path, char *const argv[]); /** Spawn a model-checked process (using PATH) * * @param file file name of the executable (found using `PATH`) * @param argv arguments for the model-checked process (NULL-terminated) */ static Session* spawnvp(const char *file, char *const argv[]); }; // Temporary extern simgrid::mc::Session* session; } } #endif SimGrid-3.18/src/mc/LocationList.cpp0000644000175000017500000000650513217757316017627 0ustar mquinsonmquinson/* Copyright (c) 2004-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "xbt/asserts.h" #include "xbt/sysdep.h" #include #include "src/mc/mc_dwarf.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace dwarf { /** Resolve a location expression */ Location resolve( simgrid::dwarf::DwarfExpression const& expression, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index) { simgrid::dwarf::ExpressionContext context; context.frame_base = frame_pointer_address; context.cursor = c; context.address_space = address_space; context.object_info = object_info; context.process_index = process_index; if (not expression.empty() && expression[0].atom >= DW_OP_reg0 && expression[0].atom <= DW_OP_reg31) { int dwarf_register = expression[0].atom - DW_OP_reg0; xbt_assert(c, "Missing frame context for register operation DW_OP_reg%i", dwarf_register); return Location(dwarf_register_to_libunwind(dwarf_register)); } simgrid::dwarf::ExpressionStack stack; simgrid::dwarf::execute(expression, context, stack); return Location((void*) stack.top()); } // TODO, move this in a method of LocationList static simgrid::dwarf::DwarfExpression const* find_expression( simgrid::dwarf::LocationList const& locations, unw_word_t ip) { for (simgrid::dwarf::LocationListEntry const& entry : locations) if (entry.valid_for_ip(ip)) return &entry.expression(); return nullptr; } Location resolve( simgrid::dwarf::LocationList const& locations, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index) { unw_word_t ip = 0; if (c && unw_get_reg(c, UNW_REG_IP, &ip)) xbt_die("Could not resolve IP"); simgrid::dwarf::DwarfExpression const* expression = find_expression(locations, ip); if (not expression) xbt_die("Could not resolve location"); return simgrid::dwarf::resolve( *expression, object_info, c, frame_pointer_address, address_space, process_index); } LocationList location_list( simgrid::mc::ObjectInformation& info, Dwarf_Attribute& attr) { LocationList locations; std::ptrdiff_t offset = 0; while (1) { Dwarf_Addr base; Dwarf_Addr start; Dwarf_Addr end; Dwarf_Op *ops; std::size_t len; offset = dwarf_getlocations(&attr, offset, &base, &start, &end, &ops, &len); if (offset == 0) break; else if (offset == -1) xbt_die("Error while loading location list"); std::uint64_t base_address = (std::uint64_t) info.base_address(); LocationListEntry::range_type range; if (start == 0) // If start == 0, this is not a location list: range = { 0, UINT64_MAX }; else range = { base_address + start, base_address + end }; locations.push_back({ DwarfExpression(ops, ops+len), range }); } return locations; } } } SimGrid-3.18/src/mc/mc_dwarf.hpp0000644000175000017500000000147113217757316017007 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_DWARF_HPP #define SIMGRID_MC_DWARF_HPP #include "xbt/base.h" #define DW_LANG_Objc DW_LANG_ObjC /* fix spelling error in older dwarf.h */ #include #include "src/mc/mc_forward.hpp" namespace simgrid { namespace dwarf { XBT_PRIVATE const char* attrname(int attr); XBT_PRIVATE const char* tagname(int tag); XBT_PRIVATE void* resolve_member( const void* base, simgrid::mc::Type* type, simgrid::mc::Member* member, simgrid::mc::AddressSpace* snapshot, int process_index); XBT_PRIVATE int dwarf_register_to_libunwind(int dwarf_register); } } #endif SimGrid-3.18/src/mc/mc_client_api.cpp0000644000175000017500000000620613217757316020007 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "xbt/sysdep.h" #include #include "src/mc/ModelChecker.hpp" #include "src/mc/mc_ignore.h" #include "src/mc/mc_private.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/remote/Client.hpp" #include "src/mc/remote/mc_protocol.h" /** @file mc_client_api.cpp * * This is the implementation of the API used by the user simulated program to * communicate with the MC (declared in modelchecker.h). */ XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_client_api, mc, "Public API for the model-checked application"); // MC_random() is in mc_base.cpp void MC_assert(int prop) { xbt_assert(mc_model_checker == nullptr); if (MC_is_active() && not prop) simgrid::mc::Client::get()->reportAssertionFailure(); } void MC_cut() { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; // FIXME, We want to do this in the model-checker: xbt_die("MC_cut() not implemented"); } void MC_ignore(void* addr, size_t size) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; simgrid::mc::Client::get()->ignoreMemory(addr, size); } void MC_automaton_new_propositional_symbol(const char* id, int (*fct)()) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; xbt_die("Support for client-side function proposition is not implemented: " "use MC_automaton_new_propositional_symbol_pointer instead."); } void MC_automaton_new_propositional_symbol_pointer(const char *name, int* value) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; simgrid::mc::Client::get()->declareSymbol(name, value); } /** @brief Register a stack in the model checker * * The stacks are allocated in the heap. The MC handle them specifically * when we analyze/compare the content of the heap so it must be told where * they are with this function. * * @param stack Where the stack is * @param actor Actor owning the stack * @param context The context associated to that stack * @param size Size of the stack */ void MC_register_stack_area(void* stack, smx_actor_t actor, ucontext_t* context, size_t size) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; simgrid::mc::Client::get()->declareStack(stack, size, actor, context); } void MC_ignore_global_variable(const char *name) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; // TODO, send a message to the model_checker xbt_die("Unimplemented"); } void MC_ignore_heap(void *address, size_t size) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; simgrid::mc::Client::get()->ignoreHeap(address, size); } void MC_unignore_heap(void* address, size_t size) { xbt_assert(mc_model_checker == nullptr); if (not MC_is_active()) return; simgrid::mc::Client::get()->unignoreHeap(address, size); } SimGrid-3.18/src/mc/mc_unw.hpp0000644000175000017500000000570613217757316016522 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_UNW_HPP #define SIMGRID_MC_UNW_HPP /** @file * Libunwind implementation for the model-checker * * Libunwind provides an pluggable stack unwinding API: the way the current * registers and memory is accessed, the way unwinding informations is found * is pluggable. * * This component implements the libunwind API for he model-checker: * * * reading memory from a simgrid::mc::AddressSpace*; * * * reading stack registers from a saved snapshot (context). * * Parts of the libunwind information fetching is currently handled by the * standard `libunwind` implementations (either the local one or the ptrace one) * because parsing `.eh_frame` section is not fun and `libdw` does not help * much here. */ #include "src/mc/mc_forward.hpp" #include "xbt/base.h" #include #include #include namespace simgrid { namespace unw { XBT_PRIVATE unw_addr_space_t create_addr_space(); XBT_PRIVATE void* create_context(unw_addr_space_t as, pid_t pid); } } namespace simgrid { namespace mc { class UnwindContext { simgrid::mc::AddressSpace* addressSpace_ = nullptr; simgrid::mc::RemoteClient* process_ = nullptr; unw_context_t unwindContext_; public: UnwindContext() = default; ~UnwindContext() { clear(); } void initialize(simgrid::mc::RemoteClient* process, unw_context_t* c); void clear(); unw_cursor_t cursor(); private: // Methods and virtual table for libunwind static int find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t* pip, int need_unwind_info, void* arg) noexcept; static void put_unwind_info(unw_addr_space_t as, unw_proc_info_t* pip, void* arg) noexcept; static int get_dyn_info_list_addr(unw_addr_space_t as, unw_word_t* dilap, void* arg) noexcept; static int access_mem(unw_addr_space_t as, unw_word_t addr, unw_word_t* valp, int write, void* arg) noexcept; static void* get_reg(unw_context_t* context, unw_regnum_t regnum) noexcept; static int access_reg(unw_addr_space_t as, unw_regnum_t regnum, unw_word_t* valp, int write, void* arg) noexcept; static int access_fpreg(unw_addr_space_t as, unw_regnum_t regnum, unw_fpreg_t* fpvalp, int write, void* arg) noexcept; static int resume(unw_addr_space_t as, unw_cursor_t* cp, void* arg) noexcept; static int get_proc_name(unw_addr_space_t as, unw_word_t addr, char* bufp, size_t buf_len, unw_word_t* offp, void* arg) noexcept; static unw_accessors_t accessors; public: // Create a libunwind address space: static unw_addr_space_t createUnwindAddressSpace(); }; void MC_dump_stack_unw(FILE* file, unw_cursor_t cursor); void dumpStack(FILE* file, unw_cursor_t cursor); void dumpStack(FILE* file, pid_t pid); } } #endif SimGrid-3.18/src/mc/mc_base.cpp0000644000175000017500000001364213217757315016613 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "mc/mc.h" #include "src/mc/mc_base.h" #include "src/mc/mc_replay.hpp" #include "src/simix/smx_private.hpp" #if SIMGRID_HAVE_MC #include "src/mc/ModelChecker.hpp" using simgrid::mc::remote; #endif XBT_LOG_NEW_DEFAULT_CATEGORY(mc, "All MC categories"); int MC_random(int min, int max) { #if SIMGRID_HAVE_MC xbt_assert(mc_model_checker == nullptr); #endif /* TODO, if the MC is disabled we do not really need to make a simcall for this :) */ return simcall_mc_random(min, max); } namespace simgrid { namespace mc { void wait_for_requests() { #if SIMGRID_HAVE_MC xbt_assert(mc_model_checker == nullptr, "This must be called from the client"); #endif while (not simix_global->process_to_run.empty()) { SIMIX_process_runall(); for (smx_actor_t const& process : simix_global->process_that_ran) { smx_simcall_t req = &process->simcall; if (req->call != SIMCALL_NONE && not simgrid::mc::request_is_visible(req)) SIMIX_simcall_handle(req, 0); } } #if SIMGRID_HAVE_MC xbt_dynar_reset(simix_global->actors_vector); for (std::pair const& kv : simix_global->process_list) { xbt_dynar_push_as(simix_global->actors_vector, smx_actor_t, kv.second); } #endif } /** @brief returns if there this transition can proceed in a finite amount of time * * It is used in the model-checker to not get into self-deadlock where it would execute a never ending transition. * * Only WAIT operations (on comm, on mutex, etc) can ever return false because they could lock the MC exploration. * Wait operations are OK and return true in only two situations: * - if the wait will succeed immediately (if both peer of the comm are there already or if the mutex is available) * - if a timeout is provided, because we can fire the timeout if the transition is not ready without blocking in this * transition for ever. * */ // Called from both MCer and MCed: bool actor_is_enabled(smx_actor_t actor) { #if SIMGRID_HAVE_MC // If in the MCer, ask the client app since it has all the data if (mc_model_checker != nullptr) { return mc_model_checker->process().actor_is_enabled(actor->pid); } #endif // Now, we are in the client app, no need for remote memory reading. smx_simcall_t req = &actor->simcall; switch (req->call) { case SIMCALL_NONE: return false; case SIMCALL_COMM_WAIT: { /* FIXME: check also that src and dst processes are not suspended */ simgrid::kernel::activity::CommImpl* act = static_cast(simcall_comm_wait__getraw__comm(req)); if (act->src_timeout || act->dst_timeout) { /* If it has a timeout it will be always be enabled (regardless of who declared the timeout), * because even if the communication is not ready, it can timeout and won't block. */ if (_sg_mc_timeout == 1) return true; } /* On the other hand if it hasn't a timeout, check if the comm is ready.*/ else if (act->detached && act->src_proc == nullptr && act->type == SIMIX_COMM_READY) return (act->dst_proc != nullptr); return (act->src_proc && act->dst_proc); } case SIMCALL_COMM_WAITANY: { xbt_dynar_t comms = simcall_comm_waitany__get__comms(req); for (unsigned int index = 0; index < comms->used; ++index) { simgrid::kernel::activity::CommImpl* act = xbt_dynar_get_as(comms, index, simgrid::kernel::activity::CommImpl*); if (act->src_proc && act->dst_proc) return true; } return false; } case SIMCALL_MUTEX_LOCK: { smx_mutex_t mutex = simcall_mutex_lock__get__mutex(req); if (mutex->owner == nullptr) return true; return mutex->owner->pid == req->issuer->pid; } case SIMCALL_SEM_ACQUIRE: { static int warned = 0; if (not warned) XBT_INFO("Using semaphore in model-checked code is still experimental. Use at your own risk"); warned = 1; return true; } case SIMCALL_COND_WAIT: { static int warned = 0; if (not warned) XBT_INFO("Using condition variables in model-checked code is still experimental. Use at your own risk"); warned = 1; return true; } default: /* The rest of the requests are always enabled */ return true; } } /* This is the list of requests that are visible from the checker algorithm. * Any other requests are handled right away on the application side. */ bool request_is_visible(smx_simcall_t req) { #if SIMGRID_HAVE_MC xbt_assert(mc_model_checker == nullptr, "This should be called from the client side"); #endif return req->call == SIMCALL_COMM_ISEND || req->call == SIMCALL_COMM_IRECV || req->call == SIMCALL_COMM_WAIT || req->call == SIMCALL_COMM_WAITANY || req->call == SIMCALL_COMM_TEST || req->call == SIMCALL_COMM_TESTANY || req->call == SIMCALL_MC_RANDOM || req->call == SIMCALL_MUTEX_LOCK || req->call == SIMCALL_MUTEX_TRYLOCK || req->call == SIMCALL_MUTEX_UNLOCK; } } } static int prng_random(int min, int max) { unsigned long output_size = ((unsigned long) max - (unsigned long) min) + 1; unsigned long input_size = (unsigned long) RAND_MAX + 1; unsigned long reject_size = input_size % output_size; unsigned long accept_size = input_size - reject_size; // module*accept_size // Use rejection in order to avoid skew unsigned long x; do { #ifndef _WIN32 x = (unsigned long) random(); #else x = (unsigned long) rand(); #endif } while( x >= accept_size ); return min + (x % output_size); } int simcall_HANDLER_mc_random(smx_simcall_t simcall, int min, int max) { if (not MC_is_active() && MC_record_path.empty()) return prng_random(min, max); return simcall->mc_value; } SimGrid-3.18/src/mc/checker/0000755000175000017500000000000013217757316016115 5ustar mquinsonmquinsonSimGrid-3.18/src/mc/checker/LivenessChecker.cpp0000644000175000017500000004147413217757316021710 0ustar mquinsonmquinson/* Copyright (c) 2011-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include #include "src/mc/Session.hpp" #include "src/mc/Transition.hpp" #include "src/mc/checker/LivenessChecker.hpp" #include "src/mc/mc_exit.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/mc_replay.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/remote/Client.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_liveness, mc, "Logging specific to algorithms for liveness properties verification"); extern std::string _sg_mc_property_file; /********* Static functions *********/ namespace simgrid { namespace mc { VisitedPair::VisitedPair(int pair_num, xbt_automaton_state_t automaton_state, std::shared_ptr> atomic_propositions, std::shared_ptr graph_state) : num(pair_num), automaton_state(automaton_state) { simgrid::mc::RemoteClient* process = &(mc_model_checker->process()); this->graph_state = std::move(graph_state); if(this->graph_state->system_state == nullptr) this->graph_state->system_state = simgrid::mc::take_snapshot(pair_num); this->heap_bytes_used = mmalloc_get_bytes_used_remote(process->get_heap()->heaplimit, process->get_malloc_info()); this->actors_count = mc_model_checker->process().actors().size(); this->other_num = -1; this->atomic_propositions = std::move(atomic_propositions); } static bool evaluate_label(xbt_automaton_exp_label_t l, std::vector const& values) { switch (l->type) { case xbt_automaton_exp_label::AUT_OR: return evaluate_label(l->u.or_and.left_exp, values) || evaluate_label(l->u.or_and.right_exp, values); case xbt_automaton_exp_label::AUT_AND: return evaluate_label(l->u.or_and.left_exp, values) && evaluate_label(l->u.or_and.right_exp, values); case xbt_automaton_exp_label::AUT_NOT: return not evaluate_label(l->u.exp_not, values); case xbt_automaton_exp_label::AUT_PREDICAT:{ unsigned int cursor = 0; xbt_automaton_propositional_symbol_t p = nullptr; xbt_dynar_foreach(simgrid::mc::property_automaton->propositional_symbols, cursor, p) { if (std::strcmp(xbt_automaton_propositional_symbol_get_name(p), l->u.predicat) == 0) return values[cursor] != 0; } xbt_die("Missing predicate"); break; } case xbt_automaton_exp_label::AUT_ONE: return true; default: xbt_die("Unexpected vaue for automaton"); } } Pair::Pair(unsigned long expanded_pairs) : num(expanded_pairs) {} std::shared_ptr> LivenessChecker::getPropositionValues() { std::vector values; unsigned int cursor = 0; xbt_automaton_propositional_symbol_t ps = nullptr; xbt_dynar_foreach(simgrid::mc::property_automaton->propositional_symbols, cursor, ps) values.push_back(xbt_automaton_propositional_symbol_evaluate(ps)); return std::make_shared>(std::move(values)); } int LivenessChecker::compare(simgrid::mc::VisitedPair* state1, simgrid::mc::VisitedPair* state2) { simgrid::mc::Snapshot* s1 = state1->graph_state->system_state.get(); simgrid::mc::Snapshot* s2 = state2->graph_state->system_state.get(); int num1 = state1->num; int num2 = state2->num; return simgrid::mc::snapshot_compare(num1, s1, num2, s2); } std::shared_ptr LivenessChecker::insertAcceptancePair(simgrid::mc::Pair* pair) { std::shared_ptr new_pair = std::make_shared( pair->num, pair->automaton_state, pair->atomic_propositions, pair->graph_state); auto res = boost::range::equal_range(acceptancePairs_, new_pair.get(), simgrid::mc::DerefAndCompareByActorsCountAndUsedHeap()); if (pair->search_cycle) for (auto i = res.first; i != res.second; ++i) { std::shared_ptr const& pair_test = *i; if (xbt_automaton_state_compare( pair_test->automaton_state, new_pair->automaton_state) != 0 || *(pair_test->atomic_propositions) != *(new_pair->atomic_propositions) || this->compare(pair_test.get(), new_pair.get()) != 0) continue; XBT_INFO("Pair %d already reached (equal to pair %d) !", new_pair->num, pair_test->num); explorationStack_.pop_back(); if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, pair_test->num, this->previousRequest_.c_str()); return nullptr; } acceptancePairs_.insert(res.first, new_pair); return new_pair; } void LivenessChecker::removeAcceptancePair(int pair_num) { for (auto i = acceptancePairs_.begin(); i != acceptancePairs_.end(); ++i) if ((*i)->num == pair_num) { acceptancePairs_.erase(i); break; } } void LivenessChecker::replay() { XBT_DEBUG("**** Begin Replay ****"); /* Intermediate backtracking */ if(_sg_mc_checkpoint > 0) { simgrid::mc::Pair* pair = explorationStack_.back().get(); if(pair->graph_state->system_state){ simgrid::mc::restore_snapshot(pair->graph_state->system_state); return; } } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); /* Traverse the stack from the initial state and re-execute the transitions */ int depth = 1; for (std::shared_ptr const& pair : explorationStack_) { if (pair == explorationStack_.back()) break; std::shared_ptr state = pair->graph_state; if (pair->exploration_started) { int req_num = state->transition.argument; smx_simcall_t saved_req = &state->executed_req; smx_simcall_t req = nullptr; if (saved_req != nullptr) { /* because we got a copy of the executed request, we have to fetch the real one, pointed by the request field of the issuer process */ const smx_actor_t issuer = MC_smx_simcall_get_issuer(saved_req); req = &issuer->simcall; /* Debug information */ XBT_DEBUG("Replay (depth = %d) : %s (%p)", depth, simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::simix).c_str(), state.get()); } this->getSession().execute(state->transition); } /* Update statistics */ visitedPairsCount_++; mc_model_checker->executed_transitions++; depth++; } XBT_DEBUG("**** End Replay ****"); } /** * \brief Checks whether a given pair has already been visited by the algorithm. */ int LivenessChecker::insertVisitedPair(std::shared_ptr visited_pair, simgrid::mc::Pair* pair) { if (_sg_mc_max_visited_states == 0) return -1; if (visited_pair == nullptr) visited_pair = std::make_shared(pair->num, pair->automaton_state, pair->atomic_propositions, pair->graph_state); auto range = boost::range::equal_range(visitedPairs_, visited_pair.get(), simgrid::mc::DerefAndCompareByActorsCountAndUsedHeap()); for (auto i = range.first; i != range.second; ++i) { VisitedPair* pair_test = i->get(); if (xbt_automaton_state_compare( pair_test->automaton_state, visited_pair->automaton_state) != 0 || *(pair_test->atomic_propositions) != *(visited_pair->atomic_propositions) || this->compare(pair_test, visited_pair.get()) != 0) continue; if (pair_test->other_num == -1) visited_pair->other_num = pair_test->num; else visited_pair->other_num = pair_test->other_num; if (dot_output == nullptr) XBT_DEBUG("Pair %d already visited ! (equal to pair %d)", visited_pair->num, pair_test->num); else XBT_DEBUG("Pair %d already visited ! (equal to pair %d (pair %d in dot_output))", visited_pair->num, pair_test->num, visited_pair->other_num); (*i) = std::move(visited_pair); return (*i)->other_num; } visitedPairs_.insert(range.first, std::move(visited_pair)); this->purgeVisitedPairs(); return -1; } void LivenessChecker::purgeVisitedPairs() { if (_sg_mc_max_visited_states != 0 && visitedPairs_.size() > (std::size_t)_sg_mc_max_visited_states) { // Remove the oldest entry with a linear search: visitedPairs_.erase(boost::min_element(visitedPairs_, [](std::shared_ptr const a, std::shared_ptr const& b) { return a->num < b->num; } )); } } LivenessChecker::LivenessChecker(Session& session) : Checker(session) { } RecordTrace LivenessChecker::getRecordTrace() // override { RecordTrace res; for (std::shared_ptr const& pair : explorationStack_) res.push_back(pair->graph_state->getTransition()); return res; } void LivenessChecker::logState() // override { XBT_INFO("Expanded pairs = %lu", expandedPairsCount_); XBT_INFO("Visited pairs = %lu", visitedPairsCount_); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); } void LivenessChecker::showAcceptanceCycle(std::size_t depth) { XBT_INFO("*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*"); XBT_INFO("| ACCEPTANCE CYCLE |"); XBT_INFO("*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*"); XBT_INFO("Counter-example that violates formula :"); simgrid::mc::dumpRecordPath(); for (auto const& s : this->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); XBT_INFO("Counter-example depth : %zu", depth); } std::vector LivenessChecker::getTextualTrace() // override { std::vector trace; for (std::shared_ptr const& pair : explorationStack_) { int req_num = pair->graph_state->transition.argument; smx_simcall_t req = &pair->graph_state->executed_req; if (req && req->call != SIMCALL_NONE) trace.push_back(simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::executed)); } return trace; } std::shared_ptr LivenessChecker::newPair(Pair* current_pair, xbt_automaton_state_t state, std::shared_ptr> propositions) { expandedPairsCount_++; std::shared_ptr next_pair = std::make_shared(expandedPairsCount_); next_pair->automaton_state = state; next_pair->graph_state = std::shared_ptr(new simgrid::mc::State(++expandedStatesCount_)); next_pair->atomic_propositions = std::move(propositions); if (current_pair) next_pair->depth = current_pair->depth + 1; else next_pair->depth = 1; /* Get enabled actors and insert them in the interleave set of the next graph_state */ for (auto& actor : mc_model_checker->process().actors()) if (simgrid::mc::actor_is_enabled(actor.copy.getBuffer())) next_pair->graph_state->addInterleavingSet(actor.copy.getBuffer()); next_pair->requests = next_pair->graph_state->interleaveSize(); /* FIXME : get search_cycle value for each accepting state */ if (next_pair->automaton_state->type == 1 || (current_pair && current_pair->search_cycle)) next_pair->search_cycle = true; else next_pair->search_cycle = false; return next_pair; } void LivenessChecker::backtrack() { /* Traverse the stack backwards until a pair with a non empty interleave set is found, deleting all the pairs that have it empty in the way. */ while (not explorationStack_.empty()) { std::shared_ptr current_pair = explorationStack_.back(); explorationStack_.pop_back(); if (current_pair->requests > 0) { /* We found a backtracking point */ XBT_DEBUG("Backtracking to depth %d", current_pair->depth); explorationStack_.push_back(std::move(current_pair)); this->replay(); XBT_DEBUG("Backtracking done"); break; } else { XBT_DEBUG("Delete pair %d at depth %d", current_pair->num, current_pair->depth); if (current_pair->automaton_state->type == 1) this->removeAcceptancePair(current_pair->num); } } } void LivenessChecker::run() { XBT_INFO("Check the liveness property %s", _sg_mc_property_file.c_str()); MC_automaton_load(_sg_mc_property_file.c_str()); XBT_DEBUG("Starting the liveness algorithm"); simgrid::mc::session->initialize(); /* Initialize */ this->previousPair_ = 0; std::shared_ptr> propos = this->getPropositionValues(); // For each initial state of the property automaton, push a // (application_state, automaton_state) pair to the exploration stack: unsigned int cursor = 0; xbt_automaton_state_t automaton_state; xbt_dynar_foreach (simgrid::mc::property_automaton->states, cursor, automaton_state) if (automaton_state->type == -1) explorationStack_.push_back(this->newPair(nullptr, automaton_state, propos)); /* Actually run the double DFS search for counter-examples */ while (not explorationStack_.empty()) { std::shared_ptr current_pair = explorationStack_.back(); /* Update current state in buchi automaton */ simgrid::mc::property_automaton->current_state = current_pair->automaton_state; XBT_DEBUG( "********************* ( Depth = %d, search_cycle = %d, interleave size = %zu, pair_num = %d, requests = %d)", current_pair->depth, current_pair->search_cycle, current_pair->graph_state->interleaveSize(), current_pair->num, current_pair->requests); if (current_pair->requests == 0) { this->backtrack(); continue; } std::shared_ptr reached_pair; if (current_pair->automaton_state->type == 1 && not current_pair->exploration_started) { reached_pair = this->insertAcceptancePair(current_pair.get()); if (reached_pair == nullptr) { this->showAcceptanceCycle(current_pair->depth); throw simgrid::mc::LivenessError(); } } /* Pair already visited ? stop the exploration on the current path */ if (not current_pair->exploration_started) { int visited_num = this->insertVisitedPair(reached_pair, current_pair.get()); if (visited_num != -1) { if (dot_output != nullptr) { fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, visited_num, this->previousRequest_.c_str()); fflush(dot_output); } XBT_DEBUG("Pair already visited (equal to pair %d), exploration on the current path stopped.", visited_num); current_pair->requests = 0; this->backtrack(); continue; } } smx_simcall_t req = MC_state_get_request(current_pair->graph_state.get()); int req_num = current_pair->graph_state->transition.argument; if (dot_output != nullptr) { if (this->previousPair_ != 0 && this->previousPair_ != current_pair->num) { fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, current_pair->num, this->previousRequest_.c_str()); this->previousRequest_.clear(); } this->previousPair_ = current_pair->num; this->previousRequest_ = simgrid::mc::request_get_dot_output(req, req_num); if (current_pair->search_cycle) fprintf(dot_output, "%d [shape=doublecircle];\n", current_pair->num); fflush(dot_output); } XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::simix).c_str()); /* Update stats */ mc_model_checker->executed_transitions++; if (not current_pair->exploration_started) visitedPairsCount_++; /* Answer the request */ mc_model_checker->handle_simcall(current_pair->graph_state->transition); /* Wait for requests (schedules processes) */ mc_model_checker->wait_for_requests(); current_pair->requests--; current_pair->exploration_started = true; /* Get values of atomic propositions (variables used in the property formula) */ std::shared_ptr> prop_values = this->getPropositionValues(); // For each enabled transition in the property automaton, push a // (application_state, automaton_state) pair to the exploration stack: for (int cursor = xbt_dynar_length(current_pair->automaton_state->out) - 1; cursor >= 0; cursor--) { xbt_automaton_transition_t transition_succ = (xbt_automaton_transition_t)xbt_dynar_get_as(current_pair->automaton_state->out, cursor, xbt_automaton_transition_t); if (evaluate_label(transition_succ->label, *prop_values)) explorationStack_.push_back(this->newPair( current_pair.get(), transition_succ->dst, prop_values)); } } XBT_INFO("No property violation found."); simgrid::mc::session->logState(); } Checker* createLivenessChecker(Session& session) { return new LivenessChecker(session); } } } SimGrid-3.18/src/mc/checker/Checker.cpp0000644000175000017500000000116413217757316020167 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "src/mc/checker/Checker.hpp" #include "src/mc/ModelChecker.hpp" namespace simgrid { namespace mc { Checker::Checker(Session& session) : session_(&session) { xbt_assert(mc_model_checker); xbt_assert(mc_model_checker->getChecker() == nullptr); mc_model_checker->setChecker(this); } } } SimGrid-3.18/src/mc/checker/SafetyChecker.hpp0000644000175000017500000000252713217757316021354 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SAFETY_CHECKER_HPP #define SIMGRID_MC_SAFETY_CHECKER_HPP #include #include #include #include #include "src/mc/VisitedState.hpp" #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_safety.hpp" namespace simgrid { namespace mc { class XBT_PRIVATE SafetyChecker : public Checker { simgrid::mc::ReductionMode reductionMode_ = simgrid::mc::ReductionMode::unset; public: explicit SafetyChecker(Session& session); ~SafetyChecker() = default; void run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; void logState() override; private: void checkNonTermination(simgrid::mc::State* current_state); void backtrack(); void restoreState(); /** Stack representing the position in the exploration graph */ std::list> stack_; simgrid::mc::VisitedStates visitedStates_; std::unique_ptr visitedState_; unsigned long expandedStatesCount_ = 0; }; } } #endif SimGrid-3.18/src/mc/checker/CommunicationDeterminismChecker.cpp0000644000175000017500000005616513217757316025131 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/Transition.hpp" #include "src/mc/VisitedState.hpp" #include "src/mc/checker/CommunicationDeterminismChecker.hpp" #include "src/mc/mc_exit.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_state.hpp" #include "src/mc/remote/Client.hpp" #include "smpi_request.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_comm_determinism, mc, "Logging specific to MC communication determinism detection"); /********** Global variables **********/ xbt_dynar_t initial_communications_pattern; xbt_dynar_t incomplete_communications_pattern; /********** Static functions ***********/ static e_mc_comm_pattern_difference_t compare_comm_pattern(simgrid::mc::PatternCommunication* comm1, simgrid::mc::PatternCommunication* comm2) { if(comm1->type != comm2->type) return TYPE_DIFF; if (comm1->rdv != comm2->rdv) return RDV_DIFF; if (comm1->src_proc != comm2->src_proc) return SRC_PROC_DIFF; if (comm1->dst_proc != comm2->dst_proc) return DST_PROC_DIFF; if (comm1->tag != comm2->tag) return TAG_DIFF; if (comm1->data.size() != comm2->data.size()) return DATA_SIZE_DIFF; if (comm1->data != comm2->data) return DATA_DIFF; return NONE_DIFF; } static char* print_determinism_result(e_mc_comm_pattern_difference_t diff, int process, simgrid::mc::PatternCommunication* comm, unsigned int cursor) { char* type; char* res; if (comm->type == simgrid::mc::PatternCommunicationType::send) type = bprintf("The send communications pattern of the process %d is different!", process - 1); else type = bprintf("The recv communications pattern of the process %d is different!", process - 1); switch(diff) { case TYPE_DIFF: res = bprintf("%s Different type for communication #%u", type, cursor); break; case RDV_DIFF: res = bprintf("%s Different rdv for communication #%u", type, cursor); break; case TAG_DIFF: res = bprintf("%s Different tag for communication #%u", type, cursor); break; case SRC_PROC_DIFF: res = bprintf("%s Different source for communication #%u", type, cursor); break; case DST_PROC_DIFF: res = bprintf("%s Different destination for communication #%u", type, cursor); break; case DATA_SIZE_DIFF: res = bprintf("%s\n Different data size for communication #%u", type, cursor); break; case DATA_DIFF: res = bprintf("%s\n Different data for communication #%u", type, cursor); break; default: res = nullptr; break; } return res; } static void update_comm_pattern(simgrid::mc::PatternCommunication* comm_pattern, simgrid::mc::RemotePtr comm_addr) { // HACK, type punning simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, comm_addr); simgrid::kernel::activity::CommImpl* comm = temp_comm.getBuffer(); smx_actor_t src_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(comm->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(comm->dst_proc)); comm_pattern->src_proc = src_proc->pid; comm_pattern->dst_proc = dst_proc->pid; comm_pattern->src_host = MC_smx_actor_get_host_name(src_proc); comm_pattern->dst_host = MC_smx_actor_get_host_name(dst_proc); if (comm_pattern->data.size() == 0 && comm->src_buff != nullptr) { size_t buff_size; mc_model_checker->process().read(&buff_size, remote(comm->dst_buff_size)); comm_pattern->data.resize(buff_size); mc_model_checker->process().read_bytes(comm_pattern->data.data(), comm_pattern->data.size(), remote(comm->src_buff)); } } namespace simgrid { namespace mc { void CommunicationDeterminismChecker::deterministic_comm_pattern(int process, simgrid::mc::PatternCommunication* comm, int backtracking) { simgrid::mc::PatternCommunicationList* list = xbt_dynar_get_as(initial_communications_pattern, process, simgrid::mc::PatternCommunicationList*); if (not backtracking) { e_mc_comm_pattern_difference_t diff = compare_comm_pattern(list->list[list->index_comm].get(), comm); if (diff != NONE_DIFF) { if (comm->type == simgrid::mc::PatternCommunicationType::send) { this->send_deterministic = 0; if (this->send_diff != nullptr) xbt_free(this->send_diff); this->send_diff = print_determinism_result(diff, process, comm, list->index_comm + 1); } else { this->recv_deterministic = 0; if (this->recv_diff != nullptr) xbt_free(this->recv_diff); this->recv_diff = print_determinism_result(diff, process, comm, list->index_comm + 1); } if (_sg_mc_send_determinism && not this->send_deterministic) { XBT_INFO("*********************************************************"); XBT_INFO("***** Non-send-deterministic communications pattern *****"); XBT_INFO("*********************************************************"); XBT_INFO("%s", this->send_diff); xbt_free(this->send_diff); this->send_diff = nullptr; simgrid::mc::session->logState(); mc_model_checker->exit(SIMGRID_MC_EXIT_NON_DETERMINISM); } else if (_sg_mc_comms_determinism && (not this->send_deterministic && not this->recv_deterministic)) { XBT_INFO("****************************************************"); XBT_INFO("***** Non-deterministic communications pattern *****"); XBT_INFO("****************************************************"); XBT_INFO("%s", this->send_diff); XBT_INFO("%s", this->recv_diff); xbt_free(this->send_diff); this->send_diff = nullptr; xbt_free(this->recv_diff); this->recv_diff = nullptr; simgrid::mc::session->logState(); mc_model_checker->exit(SIMGRID_MC_EXIT_NON_DETERMINISM); } } } } /********** Non Static functions ***********/ void CommunicationDeterminismChecker::get_comm_pattern(xbt_dynar_t list, smx_simcall_t request, e_mc_call_type_t call_type, int backtracking) { const smx_actor_t issuer = MC_smx_simcall_get_issuer(request); simgrid::mc::PatternCommunicationList* initial_pattern = xbt_dynar_get_as(initial_communications_pattern, issuer->pid, simgrid::mc::PatternCommunicationList*); xbt_dynar_t incomplete_pattern = xbt_dynar_get_as(incomplete_communications_pattern, issuer->pid, xbt_dynar_t); std::unique_ptr pattern = std::unique_ptr(new simgrid::mc::PatternCommunication()); pattern->index = initial_pattern->index_comm + xbt_dynar_length(incomplete_pattern); if (call_type == MC_CALL_TYPE_SEND) { /* Create comm pattern */ pattern->type = simgrid::mc::PatternCommunicationType::send; pattern->comm_addr = static_cast(simcall_comm_isend__getraw__result(request)); simgrid::mc::Remote temp_synchro; mc_model_checker->process().read(temp_synchro, remote(static_cast(pattern->comm_addr))); simgrid::kernel::activity::CommImpl* synchro = static_cast(temp_synchro.getBuffer()); char* remote_name = mc_model_checker->process().read( RemotePtr((uint64_t)(synchro->mbox ? &synchro->mbox->name_ : &synchro->mbox_cpy->name_))); pattern->rdv = mc_model_checker->process().read_string(RemotePtr(remote_name)); pattern->src_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(synchro->src_proc))->pid; pattern->src_host = MC_smx_actor_get_host_name(issuer); simgrid::smpi::Request mpi_request = mc_model_checker->process().read( RemotePtr((std::uint64_t)simcall_comm_isend__get__data(request))); pattern->tag = mpi_request.tag(); if (synchro->src_buff != nullptr) { pattern->data.resize(synchro->src_buff_size); mc_model_checker->process().read_bytes(pattern->data.data(), pattern->data.size(), remote(synchro->src_buff)); } if(mpi_request.detached()){ if (not this->initial_communications_pattern_done) { /* Store comm pattern */ simgrid::mc::PatternCommunicationList* list = xbt_dynar_get_as(initial_communications_pattern, pattern->src_proc, simgrid::mc::PatternCommunicationList*); list->list.push_back(std::move(pattern)); } else { /* Evaluate comm determinism */ this->deterministic_comm_pattern(pattern->src_proc, pattern.get(), backtracking); xbt_dynar_get_as(initial_communications_pattern, pattern->src_proc, simgrid::mc::PatternCommunicationList*) ->index_comm++; } return; } } else if (call_type == MC_CALL_TYPE_RECV) { pattern->type = simgrid::mc::PatternCommunicationType::receive; pattern->comm_addr = static_cast(simcall_comm_irecv__getraw__result(request)); simgrid::smpi::Request mpi_request; mc_model_checker->process().read(&mpi_request, remote((simgrid::smpi::Request*)simcall_comm_irecv__get__data(request))); pattern->tag = mpi_request.tag(); simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote(static_cast(pattern->comm_addr))); simgrid::kernel::activity::CommImpl* comm = temp_comm.getBuffer(); char* remote_name; mc_model_checker->process().read( &remote_name, remote(comm->mbox ? &simgrid::xbt::string::to_string_data(comm->mbox->name_).data : &simgrid::xbt::string::to_string_data(comm->mbox_cpy->name_).data)); pattern->rdv = mc_model_checker->process().read_string(RemotePtr(remote_name)); pattern->dst_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(comm->dst_proc))->pid; pattern->dst_host = MC_smx_actor_get_host_name(issuer); } else xbt_die("Unexpected call_type %i", (int) call_type); XBT_DEBUG("Insert incomplete comm pattern %p for process %lu", pattern.get(), issuer->pid); xbt_dynar_t dynar = xbt_dynar_get_as(incomplete_communications_pattern, issuer->pid, xbt_dynar_t); simgrid::mc::PatternCommunication* pattern2 = pattern.release(); xbt_dynar_push(dynar, &pattern2); } void CommunicationDeterminismChecker::complete_comm_pattern( xbt_dynar_t list, simgrid::mc::RemotePtr comm_addr, unsigned int issuer, int backtracking) { simgrid::mc::PatternCommunication* current_comm_pattern; unsigned int cursor = 0; std::unique_ptr comm_pattern; int completed = 0; /* Complete comm pattern */ xbt_dynar_foreach(xbt_dynar_get_as(incomplete_communications_pattern, issuer, xbt_dynar_t), cursor, current_comm_pattern) if (remote(current_comm_pattern->comm_addr) == comm_addr) { update_comm_pattern(current_comm_pattern, comm_addr); completed = 1; simgrid::mc::PatternCommunication* temp; xbt_dynar_remove_at(xbt_dynar_get_as(incomplete_communications_pattern, issuer, xbt_dynar_t), cursor, &temp); comm_pattern = std::unique_ptr(temp); XBT_DEBUG("Remove incomplete comm pattern for process %u at cursor %u", issuer, cursor); break; } if (not completed) xbt_die("Corresponding communication not found!"); simgrid::mc::PatternCommunicationList* pattern = xbt_dynar_get_as(initial_communications_pattern, issuer, simgrid::mc::PatternCommunicationList*); if (not this->initial_communications_pattern_done) /* Store comm pattern */ pattern->list.push_back(std::move(comm_pattern)); else { /* Evaluate comm determinism */ this->deterministic_comm_pattern(issuer, comm_pattern.get(), backtracking); pattern->index_comm++; } } CommunicationDeterminismChecker::CommunicationDeterminismChecker(Session& session) : Checker(session) { } CommunicationDeterminismChecker::~CommunicationDeterminismChecker() = default; RecordTrace CommunicationDeterminismChecker::getRecordTrace() // override { RecordTrace res; for (auto const& state : stack_) res.push_back(state->getTransition()); return res; } std::vector CommunicationDeterminismChecker::getTextualTrace() // override { std::vector trace; for (auto const& state : stack_) { smx_simcall_t req = &state->executed_req; if (req) trace.push_back( simgrid::mc::request_to_string(req, state->transition.argument, simgrid::mc::RequestType::executed)); } return trace; } void CommunicationDeterminismChecker::logState() // override { if (_sg_mc_comms_determinism && not this->recv_deterministic && this->send_deterministic) { XBT_INFO("******************************************************"); XBT_INFO("**** Only-send-deterministic communication pattern ****"); XBT_INFO("******************************************************"); XBT_INFO("%s", this->recv_diff); } else if (_sg_mc_comms_determinism && not this->send_deterministic && this->recv_deterministic) { XBT_INFO("******************************************************"); XBT_INFO("**** Only-recv-deterministic communication pattern ****"); XBT_INFO("******************************************************"); XBT_INFO("%s", this->send_diff); } XBT_INFO("Expanded states = %lu", expandedStatesCount_); XBT_INFO("Visited states = %lu", mc_model_checker->visited_states); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); XBT_INFO("Send-deterministic : %s", not this->send_deterministic ? "No" : "Yes"); if (_sg_mc_comms_determinism) XBT_INFO("Recv-deterministic : %s", not this->recv_deterministic ? "No" : "Yes"); } void CommunicationDeterminismChecker::prepare() { const int maxpid = MC_smx_get_maxpid(); // Create initial_communications_pattern elements: initial_communications_pattern = simgrid::xbt::newDeleteDynar(); for (int i = 0; i < maxpid; i++) { simgrid::mc::PatternCommunicationList* process_list_pattern = new simgrid::mc::PatternCommunicationList(); xbt_dynar_insert_at(initial_communications_pattern, i, &process_list_pattern); } // Create incomplete_communications_pattern elements: incomplete_communications_pattern = xbt_dynar_new(sizeof(xbt_dynar_t), xbt_dynar_free_voidp); for (int i = 0; i < maxpid; i++) { xbt_dynar_t process_pattern = xbt_dynar_new(sizeof(simgrid::mc::PatternCommunication*), nullptr); xbt_dynar_insert_at(incomplete_communications_pattern, i, &process_pattern); } std::unique_ptr initial_state = std::unique_ptr(new simgrid::mc::State(++expandedStatesCount_)); XBT_DEBUG("********* Start communication determinism verification *********"); /* Get an enabled actor and insert it in the interleave set of the initial state */ for (auto& actor : mc_model_checker->process().actors()) if (simgrid::mc::actor_is_enabled(actor.copy.getBuffer())) initial_state->addInterleavingSet(actor.copy.getBuffer()); stack_.push_back(std::move(initial_state)); } static inline bool all_communications_are_finished() { for (size_t current_actor = 1; current_actor < MC_smx_get_maxpid(); current_actor++) { xbt_dynar_t pattern = xbt_dynar_get_as(incomplete_communications_pattern, current_actor, xbt_dynar_t); if (not xbt_dynar_is_empty(pattern)) { XBT_DEBUG("Some communications are not finished, cannot stop the exploration! State not visited."); return false; } } return true; } void CommunicationDeterminismChecker::restoreState() { /* Intermediate backtracking */ simgrid::mc::State* state = stack_.back().get(); if (state->system_state) { simgrid::mc::restore_snapshot(state->system_state); MC_restore_communications_pattern(state); return; } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); unsigned n = MC_smx_get_maxpid(); assert(n == xbt_dynar_length(incomplete_communications_pattern)); assert(n == xbt_dynar_length(initial_communications_pattern)); for (unsigned j=0; j < n ; j++) { xbt_dynar_reset((xbt_dynar_t)xbt_dynar_get_as(incomplete_communications_pattern, j, xbt_dynar_t)); xbt_dynar_get_as(initial_communications_pattern, j, simgrid::mc::PatternCommunicationList*)->index_comm = 0; } /* Traverse the stack from the state at position start and re-execute the transitions */ for (std::unique_ptr const& state : stack_) { if (state == stack_.back()) break; int req_num = state->transition.argument; smx_simcall_t saved_req = &state->executed_req; xbt_assert(saved_req); /* because we got a copy of the executed request, we have to fetch the real one, pointed by the request field of the issuer process */ const smx_actor_t issuer = MC_smx_simcall_get_issuer(saved_req); smx_simcall_t req = &issuer->simcall; /* TODO : handle test and testany simcalls */ e_mc_call_type_t call = MC_get_call_type(req); mc_model_checker->handle_simcall(state->transition); MC_handle_comm_pattern(call, req, req_num, nullptr, 1); mc_model_checker->wait_for_requests(); /* Update statistics */ mc_model_checker->visited_states++; mc_model_checker->executed_transitions++; } } void CommunicationDeterminismChecker::main() { std::unique_ptr visited_state = nullptr; smx_simcall_t req = nullptr; while (not stack_.empty()) { /* Get current state */ simgrid::mc::State* state = stack_.back().get(); XBT_DEBUG("**************************************************"); XBT_DEBUG("Exploration depth = %zu (state = %d, interleaved processes = %zu)", stack_.size(), state->num, state->interleaveSize()); /* Update statistics */ mc_model_checker->visited_states++; if (stack_.size() <= (std::size_t)_sg_mc_max_depth) req = MC_state_get_request(state); else req = nullptr; if (req != nullptr && visited_state == nullptr) { int req_num = state->transition.argument; XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string(req, req_num, simgrid::mc::RequestType::simix).c_str()); std::string req_str; if (dot_output != nullptr) req_str = simgrid::mc::request_get_dot_output(req, req_num); mc_model_checker->executed_transitions++; /* TODO : handle test and testany simcalls */ e_mc_call_type_t call = MC_CALL_TYPE_NONE; if (_sg_mc_comms_determinism || _sg_mc_send_determinism) call = MC_get_call_type(req); /* Answer the request */ mc_model_checker->handle_simcall(state->transition); /* After this call req is no longer useful */ if (not this->initial_communications_pattern_done) MC_handle_comm_pattern(call, req, req_num, initial_communications_pattern, 0); else MC_handle_comm_pattern(call, req, req_num, nullptr, 0); /* Wait for requests (schedules processes) */ mc_model_checker->wait_for_requests(); /* Create the new expanded state */ std::unique_ptr next_state = std::unique_ptr(new simgrid::mc::State(++expandedStatesCount_)); /* If comm determinism verification, we cannot stop the exploration if some communications are not finished (at * least, data are transferred). These communications are incomplete and they cannot be analyzed and compared * with the initial pattern. */ bool compare_snapshots = all_communications_are_finished() && this->initial_communications_pattern_done; if (_sg_mc_max_visited_states != 0) visited_state = visitedStates_.addVisitedState(expandedStatesCount_, next_state.get(), compare_snapshots); else visited_state = nullptr; if (visited_state == nullptr) { /* Get enabled actors and insert them in the interleave set of the next state */ for (auto& actor : mc_model_checker->process().actors()) if (simgrid::mc::actor_is_enabled(actor.copy.getBuffer())) next_state->addInterleavingSet(actor.copy.getBuffer()); if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, next_state->num, req_str.c_str()); } else if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, visited_state->original_num == -1 ? visited_state->num : visited_state->original_num, req_str.c_str()); stack_.push_back(std::move(next_state)); } else { if (stack_.size() > (std::size_t) _sg_mc_max_depth) XBT_WARN("/!\\ Max depth reached! /!\\ "); else if (visited_state != nullptr) XBT_DEBUG("State already visited (equal to state %d), exploration stopped on this path.", visited_state->original_num == -1 ? visited_state->num : visited_state->original_num); else XBT_DEBUG("There are no more processes to interleave. (depth %zu)", stack_.size()); if (not this->initial_communications_pattern_done) this->initial_communications_pattern_done = 1; /* Trash the current state, no longer needed */ XBT_DEBUG("Delete state %d at depth %zu", state->num, stack_.size()); stack_.pop_back(); visited_state = nullptr; /* Check for deadlocks */ if (mc_model_checker->checkDeadlock()) { MC_show_deadlock(); throw simgrid::mc::DeadlockError(); } while (not stack_.empty()) { std::unique_ptr state = std::move(stack_.back()); stack_.pop_back(); if (state->interleaveSize() && stack_.size() < (std::size_t)_sg_mc_max_depth) { /* We found a back-tracking point, let's loop */ XBT_DEBUG("Back-tracking to state %d at depth %zu", state->num, stack_.size() + 1); stack_.push_back(std::move(state)); this->restoreState(); XBT_DEBUG("Back-tracking to state %d at depth %zu done", stack_.back()->num, stack_.size()); break; } else { XBT_DEBUG("Delete state %d at depth %zu", state->num, stack_.size() + 1); } } } } simgrid::mc::session->logState(); } void CommunicationDeterminismChecker::run() { XBT_INFO("Check communication determinism"); simgrid::mc::session->initialize(); this->prepare(); this->main(); } Checker* createCommunicationDeterminismChecker(Session& session) { return new CommunicationDeterminismChecker(session); } } } SimGrid-3.18/src/mc/checker/Checker.hpp0000644000175000017500000000420113217757316020167 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHECKER_HPP #define SIMGRID_MC_CHECKER_HPP #include #include #include #include "src/mc/Session.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_record.hpp" namespace simgrid { namespace mc { /** A model-checking algorithm * * This is an abstract base class used to group the data, state, configuration * of a model-checking algorithm. * * Implementing this interface will probably not be really mandatory, * you might be able to write your model-checking algorithm as plain * imperative code instead. * * It is expected to interact with the model-checking core through the * `Session` interface (but currently the `Session` interface does not * have all the necessary features). */ // abstract class Checker { Session* session_; public: explicit Checker(Session& session); // No copy: Checker(Checker const&) = delete; Checker& operator=(Checker const&) = delete; virtual ~Checker() = default; /** Main function of this algorithm */ virtual void run() = 0; /* These methods are callbacks called by the model-checking engine * to get and display information about the current state of the * model-checking algorithm: */ /** Show the current trace/stack * * Could this be handled in the Session/ModelChecker instead? */ virtual RecordTrace getRecordTrace() = 0; /** Generate a textual execution trace of the simulated application */ virtual std::vector getTextualTrace() = 0; /** Log additional information about the state of the model-checker */ virtual void logState() = 0; protected: Session& getSession() { return *session_; } }; XBT_PUBLIC() Checker* createLivenessChecker(Session& session); XBT_PUBLIC() Checker* createSafetyChecker(Session& session); XBT_PUBLIC() Checker* createCommunicationDeterminismChecker(Session& session); } } #endif SimGrid-3.18/src/mc/checker/LivenessChecker.hpp0000644000175000017500000000576413217757316021717 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_LIVENESS_CHECKER_HPP #define SIMGRID_MC_LIVENESS_CHECKER_HPP #include #include #include #include #include #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_state.hpp" #include #include #include namespace simgrid { namespace mc { class XBT_PRIVATE Pair { public: int num = 0; bool search_cycle = false; std::shared_ptr graph_state = nullptr; /* System state included */ xbt_automaton_state_t automaton_state = nullptr; std::shared_ptr> atomic_propositions; int requests = 0; int depth = 0; bool exploration_started = false; explicit Pair(unsigned long expanded_pairs); ~Pair() = default; Pair(Pair const&) = delete; Pair& operator=(Pair const&) = delete; }; class XBT_PRIVATE VisitedPair { public: int num; int other_num = 0; /* Dot output for */ std::shared_ptr graph_state = nullptr; /* System state included */ xbt_automaton_state_t automaton_state; std::shared_ptr> atomic_propositions; std::size_t heap_bytes_used = 0; int actors_count = 0; VisitedPair( int pair_num, xbt_automaton_state_t automaton_state, std::shared_ptr> atomic_propositions, std::shared_ptr graph_state); ~VisitedPair() = default; }; class XBT_PRIVATE LivenessChecker : public Checker { public: explicit LivenessChecker(Session& session); ~LivenessChecker() = default; void run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; void logState() override; private: int compare(simgrid::mc::VisitedPair* state1, simgrid::mc::VisitedPair* state2); std::shared_ptr> getPropositionValues(); std::shared_ptr insertAcceptancePair(simgrid::mc::Pair* pair); int insertVisitedPair(std::shared_ptr visited_pair, simgrid::mc::Pair* pair); void showAcceptanceCycle(std::size_t depth); void replay(); void removeAcceptancePair(int pair_num); void purgeVisitedPairs(); void backtrack(); std::shared_ptr newPair(Pair* pair, xbt_automaton_state_t state, std::shared_ptr> propositions); // A stack of (application_state, automaton_state) pairs for DFS exploration: std::list> explorationStack_; std::list> acceptancePairs_; std::list> visitedPairs_; unsigned long visitedPairsCount_ = 0; unsigned long expandedPairsCount_ = 0; unsigned long expandedStatesCount_ = 0; int previousPair_ = 0; std::string previousRequest_; }; } } #endif SimGrid-3.18/src/mc/checker/CommunicationDeterminismChecker.hpp0000644000175000017500000000363113217757316025124 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/mc/VisitedState.hpp" #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_forward.hpp" #ifndef SIMGRID_MC_COMMUNICATION_DETERMINISM_CHECKER_HPP #define SIMGRID_MC_COMMUNICATION_DETERMINISM_CHECKER_HPP namespace simgrid { namespace mc { class XBT_PRIVATE CommunicationDeterminismChecker : public Checker { public: explicit CommunicationDeterminismChecker(Session& session); ~CommunicationDeterminismChecker(); void run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; private: void prepare(); void main(); void logState() override; void deterministic_comm_pattern(int process, simgrid::mc::PatternCommunication* comm, int backtracking); void restoreState(); public: // These are used by functions which should be moved in CommunicationDeterminismChecker: void get_comm_pattern(xbt_dynar_t list, smx_simcall_t request, e_mc_call_type_t call_type, int backtracking); void complete_comm_pattern(xbt_dynar_t list, simgrid::mc::RemotePtr comm_addr, unsigned int issuer, int backtracking); private: /** Stack representing the position in the exploration graph */ std::list> stack_; simgrid::mc::VisitedStates visitedStates_; unsigned long expandedStatesCount_ = 0; bool initial_communications_pattern_done = false; bool recv_deterministic = true; bool send_deterministic = true; char *send_diff = nullptr; char *recv_diff = nullptr; }; #endif } } SimGrid-3.18/src/mc/checker/SafetyChecker.cpp0000644000175000017500000002776513217757316021362 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include "src/mc/Session.hpp" #include "src/mc/Transition.hpp" #include "src/mc/VisitedState.hpp" #include "src/mc/checker/SafetyChecker.hpp" #include "src/mc/mc_exit.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/xbt/mmalloc/mmprivate.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_safety, mc, "Logging specific to MC safety verification "); namespace simgrid { namespace mc { static int snapshot_compare(simgrid::mc::State* state1, simgrid::mc::State* state2) { simgrid::mc::Snapshot* s1 = state1->system_state.get(); simgrid::mc::Snapshot* s2 = state2->system_state.get(); int num1 = state1->num; int num2 = state2->num; return snapshot_compare(num1, s1, num2, s2); } void SafetyChecker::checkNonTermination(simgrid::mc::State* current_state) { for (auto state = stack_.rbegin(); state != stack_.rend(); ++state) if (snapshot_compare(state->get(), current_state) == 0) { XBT_INFO("Non-progressive cycle: state %d -> state %d", (*state)->num, current_state->num); XBT_INFO("******************************************"); XBT_INFO("*** NON-PROGRESSIVE CYCLE DETECTED ***"); XBT_INFO("******************************************"); XBT_INFO("Counter-example execution trace:"); for (auto const& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); throw simgrid::mc::TerminationError(); } } RecordTrace SafetyChecker::getRecordTrace() // override { RecordTrace res; for (auto const& state : stack_) res.push_back(state->getTransition()); return res; } std::vector SafetyChecker::getTextualTrace() // override { std::vector trace; for (auto const& state : stack_) { int value = state->transition.argument; smx_simcall_t req = &state->executed_req; if (req) trace.push_back(simgrid::mc::request_to_string( req, value, simgrid::mc::RequestType::executed)); } return trace; } void SafetyChecker::logState() // override { XBT_INFO("Expanded states = %lu", expandedStatesCount_); XBT_INFO("Visited states = %lu", mc_model_checker->visited_states); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); } void SafetyChecker::run() { /* This function runs the DFS algorithm the state space. * We do so iteratively instead of recursively, dealing with the call stack manually. * This allows to explore the call stack at wish. */ while (not stack_.empty()) { /* Get current state */ simgrid::mc::State* state = stack_.back().get(); XBT_DEBUG("**************************************************"); XBT_DEBUG("Exploration depth=%zu (state=%p, num %d)(%zu interleave)", stack_.size(), state, state->num, state->interleaveSize()); mc_model_checker->visited_states++; // Backtrack if we reached the maximum depth if (stack_.size() > (std::size_t)_sg_mc_max_depth) { XBT_WARN("/!\\ Max depth reached ! /!\\ "); this->backtrack(); continue; } // Backtrack if we are revisiting a state we saw previously if (visitedState_ != nullptr) { XBT_DEBUG("State already visited (equal to state %d), exploration stopped on this path.", visitedState_->original_num == -1 ? visitedState_->num : visitedState_->original_num); visitedState_ = nullptr; this->backtrack(); continue; } // Search an enabled transition in the current state; backtrack if the interleave set is empty // get_request also sets state.transition to be the one corresponding to the returned req smx_simcall_t req = MC_state_get_request(state); // req is now the transition of the process that was selected to be executed if (req == nullptr) { XBT_DEBUG("There are no more processes to interleave. (depth %zu)", stack_.size() + 1); this->backtrack(); continue; } // If there are processes to interleave and the maximum depth has not been // reached then perform one step of the exploration algorithm. XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string( req, state->transition.argument, simgrid::mc::RequestType::simix).c_str()); std::string req_str; if (dot_output != nullptr) req_str = simgrid::mc::request_get_dot_output(req, state->transition.argument); mc_model_checker->executed_transitions++; /* Actually answer the request: let execute the selected request (MCed does one step) */ this->getSession().execute(state->transition); /* Create the new expanded state (copy the state of MCed into our MCer data) */ std::unique_ptr next_state = std::unique_ptr(new simgrid::mc::State(++expandedStatesCount_)); if (_sg_mc_termination) this->checkNonTermination(next_state.get()); /* Check whether we already explored next_state in the past (but only if interested in state-equality reduction) */ if (_sg_mc_max_visited_states > 0) visitedState_ = visitedStates_.addVisitedState(expandedStatesCount_, next_state.get(), true); /* If this is a new state (or if we don't care about state-equality reduction) */ if (visitedState_ == nullptr) { /* Get an enabled process and insert it in the interleave set of the next state */ for (auto& remoteActor : mc_model_checker->process().actors()) { auto actor = remoteActor.copy.getBuffer(); if (simgrid::mc::actor_is_enabled(actor)) { next_state->addInterleavingSet(actor); if (reductionMode_ == simgrid::mc::ReductionMode::dpor) break; // With DPOR, we take the first enabled transition } } if (dot_output != nullptr) std::fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, next_state->num, req_str.c_str()); } else if (dot_output != nullptr) std::fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, visitedState_->original_num == -1 ? visitedState_->num : visitedState_->original_num, req_str.c_str()); stack_.push_back(std::move(next_state)); } XBT_INFO("No property violation found."); simgrid::mc::session->logState(); } void SafetyChecker::backtrack() { stack_.pop_back(); /* Check for deadlocks */ if (mc_model_checker->checkDeadlock()) { MC_show_deadlock(); throw simgrid::mc::DeadlockError(); } /* Traverse the stack backwards until a state with a non empty interleave set is found, deleting all the states that have it empty in the way. For each deleted state, check if the request that has generated it (from it's predecessor state), depends on any other previous request executed before it. If it does then add it to the interleave set of the state that executed that previous request. */ while (not stack_.empty()) { std::unique_ptr state = std::move(stack_.back()); stack_.pop_back(); if (reductionMode_ == simgrid::mc::ReductionMode::dpor) { smx_simcall_t req = &state->internal_req; if (req->call == SIMCALL_MUTEX_LOCK || req->call == SIMCALL_MUTEX_TRYLOCK) xbt_die("Mutex is currently not supported with DPOR, use --cfg=model-check/reduction:none"); const smx_actor_t issuer = MC_smx_simcall_get_issuer(req); for (auto i = stack_.rbegin(); i != stack_.rend(); ++i) { simgrid::mc::State* prev_state = i->get(); if (simgrid::mc::request_depend(req, &prev_state->internal_req)) { if (XBT_LOG_ISENABLED(mc_safety, xbt_log_priority_debug)) { XBT_DEBUG("Dependent Transitions:"); int value = prev_state->transition.argument; smx_simcall_t prev_req = &prev_state->executed_req; XBT_DEBUG("%s (state=%d)", simgrid::mc::request_to_string( prev_req, value, simgrid::mc::RequestType::internal).c_str(), prev_state->num); value = state->transition.argument; prev_req = &state->executed_req; XBT_DEBUG("%s (state=%d)", simgrid::mc::request_to_string( prev_req, value, simgrid::mc::RequestType::executed).c_str(), state->num); } if (not prev_state->actorStates[issuer->pid].isDone()) prev_state->addInterleavingSet(issuer); else XBT_DEBUG("Process %p is in done set", req->issuer); break; } else if (req->issuer == prev_state->internal_req.issuer) { XBT_DEBUG("Simcall %d and %d with same issuer", req->call, prev_state->internal_req.call); break; } else { const smx_actor_t previous_issuer = MC_smx_simcall_get_issuer(&prev_state->internal_req); XBT_DEBUG("Simcall %d, process %lu (state %d) and simcall %d, process %lu (state %d) are independent", req->call, issuer->pid, state->num, prev_state->internal_req.call, previous_issuer->pid, prev_state->num); } } } if (state->interleaveSize() && stack_.size() < (std::size_t) _sg_mc_max_depth) { /* We found a back-tracking point, let's loop */ XBT_DEBUG("Back-tracking to state %d at depth %zu", state->num, stack_.size() + 1); stack_.push_back(std::move(state)); this->restoreState(); XBT_DEBUG("Back-tracking to state %d at depth %zu done", stack_.back()->num, stack_.size()); break; } else { XBT_DEBUG("Delete state %d at depth %zu", state->num, stack_.size() + 1); } } } void SafetyChecker::restoreState() { /* Intermediate backtracking */ simgrid::mc::State* state = stack_.back().get(); if (state->system_state) { simgrid::mc::restore_snapshot(state->system_state); return; } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); /* Traverse the stack from the state at position start and re-execute the transitions */ for (std::unique_ptr const& state : stack_) { if (state == stack_.back()) break; session->execute(state->transition); /* Update statistics */ mc_model_checker->visited_states++; mc_model_checker->executed_transitions++; } } SafetyChecker::SafetyChecker(Session& session) : Checker(session) { reductionMode_ = simgrid::mc::reduction_mode; if (_sg_mc_termination) reductionMode_ = simgrid::mc::ReductionMode::none; else if (reductionMode_ == simgrid::mc::ReductionMode::unset) reductionMode_ = simgrid::mc::ReductionMode::dpor; if (_sg_mc_termination) XBT_INFO("Check non progressive cycles"); else XBT_INFO("Check a safety property. Reduction is: %s.", (reductionMode_ == simgrid::mc::ReductionMode::none ? "none": (reductionMode_ == simgrid::mc::ReductionMode::dpor ? "dpor": "unknown"))); simgrid::mc::session->initialize(); XBT_DEBUG("Starting the safety algorithm"); std::unique_ptr initial_state = std::unique_ptr(new simgrid::mc::State(++expandedStatesCount_)); XBT_DEBUG("**************************************************"); XBT_DEBUG("Initial state"); /* Get an enabled actor and insert it in the interleave set of the initial state */ for (auto& actor : mc_model_checker->process().actors()) if (simgrid::mc::actor_is_enabled(actor.copy.getBuffer())) { initial_state->addInterleavingSet(actor.copy.getBuffer()); if (reductionMode_ != simgrid::mc::ReductionMode::none) break; } stack_.push_back(std::move(initial_state)); } Checker* createSafetyChecker(Session& session) { return new SafetyChecker(session); } } } SimGrid-3.18/src/mc/checker/simgrid_mc.cpp0000644000175000017500000000561713217757316020747 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include "simgrid/sg_config.h" #include "src/xbt_modinter.h" #include "src/mc/Session.hpp" #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_base.h" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_exit.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_safety.hpp" #include "src/mc/remote/mc_protocol.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_main, mc, "Entry point for simgrid-mc"); extern std::string _sg_mc_property_file; static inline char** argvdup(int argc, char** argv) { char** argv_copy = new char*[argc + 1]; std::memcpy(argv_copy, argv, sizeof(char*) * argc); argv_copy[argc] = nullptr; return argv_copy; } static std::unique_ptr createChecker(simgrid::mc::Session& session) { if (_sg_mc_comms_determinism || _sg_mc_send_determinism) return std::unique_ptr(simgrid::mc::createCommunicationDeterminismChecker(session)); else if (_sg_mc_property_file.empty()) return std::unique_ptr(simgrid::mc::createSafetyChecker(session)); else return std::unique_ptr(simgrid::mc::createLivenessChecker(session)); } int main(int argc, char** argv) { using simgrid::mc::Session; XBT_LOG_CONNECT(mc_main); try { if (argc < 2) xbt_die("Missing arguments.\n"); // Currently, we need this before sg_config_init: _sg_do_model_check = 1; // The initialization function can touch argv. // We make a copy of argv before modifying it in order to pass the original // value to the model-checked: char** argv_copy = argvdup(argc, argv); xbt_log_init(&argc, argv); sg_config_init(&argc, argv); std::unique_ptr session = std::unique_ptr(Session::spawnvp(argv_copy[1], argv_copy+1)); delete[] argv_copy; simgrid::mc::session = session.get(); std::unique_ptr checker = createChecker(*session); int res = SIMGRID_MC_EXIT_SUCCESS; try { checker->run(); } catch (simgrid::mc::DeadlockError& de) { res = SIMGRID_MC_EXIT_DEADLOCK; } catch (simgrid::mc::TerminationError& te) { res = SIMGRID_MC_EXIT_NON_TERMINATION; } catch (simgrid::mc::LivenessError& le) { res = SIMGRID_MC_EXIT_LIVENESS; } checker = nullptr; session->close(); return res; } catch(std::exception& e) { XBT_ERROR("Exception: %s", e.what()); return SIMGRID_MC_EXIT_ERROR; } catch(...) { XBT_ERROR("Unknown exception"); return SIMGRID_MC_EXIT_ERROR; } } SimGrid-3.18/src/mc/mc_xbt.hpp0000644000175000017500000000131013217757316016471 0ustar mquinsonmquinson/* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_XBT_HPP #define SIMGRID_MC_XBT_HPP #include "xbt/dynar.h" #include "src/mc/AddressSpace.hpp" #include "src/mc/remote/RemotePtr.hpp" namespace simgrid { namespace mc { XBT_PRIVATE void read_element(AddressSpace const& as, void* local, RemotePtr addr, std::size_t i, std::size_t len); XBT_PRIVATE std::size_t read_length( AddressSpace const& as, RemotePtr addr); } } #endif SimGrid-3.18/src/mc/mc_dwarf.cpp0000644000175000017500000013005613217757316017004 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #define DW_LANG_Objc DW_LANG_ObjC /* fix spelling error in older dwarf.h */ #include #include #include #include "src/simgrid/util.hpp" #include "xbt/log.h" #include "xbt/string.hpp" #include "xbt/sysdep.h" #include #include "src/mc/mc_dwarf.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" #include "src/mc/remote/RemoteClient.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_dwarf, mc, "DWARF processing"); /** \brief The default DW_TAG_lower_bound for a given DW_AT_language. * * The default for a given language is defined in the DWARF spec. * * \param language constant as defined by the DWARf spec */ static uint64_t MC_dwarf_default_lower_bound(int lang); /** \brief Computes the the element_count of a DW_TAG_enumeration_type DIE * * This is the number of elements in a given array dimension. * * A reference of the compilation unit (DW_TAG_compile_unit) is * needed because the default lower bound (when there is no DW_AT_lower_bound) * depends of the language of the compilation unit (DW_AT_language). * * \param die DIE for the DW_TAG_enumeration_type or DW_TAG_subrange_type * \param unit DIE of the DW_TAG_compile_unit */ static uint64_t MC_dwarf_subrange_element_count(Dwarf_Die* die, Dwarf_Die* unit); /** \brief Computes the number of elements of a given DW_TAG_array_type. * * \param die DIE for the DW_TAG_array_type */ static uint64_t MC_dwarf_array_element_count(Dwarf_Die * die, Dwarf_Die * unit); /** \brief Process a DIE * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Process a type DIE */ static void MC_dwarf_handle_type_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Calls MC_dwarf_handle_die on all children of the given die * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_children(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Handle a variable (DW_TAG_variable or other) * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_variable_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Get the DW_TAG_type of the DIE * * \param die DIE * \return DW_TAG_type attribute as a new string (nullptr if none) */ static std::uint64_t MC_dwarf_at_type(Dwarf_Die * die); namespace simgrid { namespace dwarf { enum class TagClass { Unknown, Type, Subprogram, Variable, Scope, Namespace }; /*** Class of forms defined in the DWARF standard */ enum class FormClass { Unknown, Address, // Location in the program's address space Block, // Arbitrary block of bytes Constant, String, Flag, // Boolean value Reference, // Reference to another DIE ExprLoc, // DWARF expression/location description LinePtr, LocListPtr, MacPtr, RangeListPtr }; static TagClass classify_tag(int tag) { switch (tag) { case DW_TAG_array_type: case DW_TAG_class_type: case DW_TAG_enumeration_type: case DW_TAG_typedef: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_string_type: case DW_TAG_structure_type: case DW_TAG_subroutine_type: case DW_TAG_union_type: case DW_TAG_ptr_to_member_type: case DW_TAG_set_type: case DW_TAG_subrange_type: case DW_TAG_base_type: case DW_TAG_const_type: case DW_TAG_file_type: case DW_TAG_packed_type: case DW_TAG_volatile_type: case DW_TAG_restrict_type: case DW_TAG_interface_type: case DW_TAG_unspecified_type: case DW_TAG_shared_type: return TagClass::Type; case DW_TAG_subprogram: return TagClass::Subprogram; case DW_TAG_variable: case DW_TAG_formal_parameter: return TagClass::Variable; case DW_TAG_lexical_block: case DW_TAG_try_block: case DW_TAG_catch_block: case DW_TAG_inlined_subroutine: case DW_TAG_with_stmt: return TagClass::Scope; case DW_TAG_namespace: return TagClass::Namespace; default: return TagClass::Unknown; } } /** \brief Find the DWARF data class for a given DWARF data form * * This mapping is defined in the DWARF spec. * * \param form The form (values taken from the DWARF spec) * \return An internal representation for the corresponding class * */ static FormClass classify_form(int form) { switch (form) { case DW_FORM_addr: return FormClass::Address; case DW_FORM_block2: case DW_FORM_block4: case DW_FORM_block: case DW_FORM_block1: return FormClass::Block; case DW_FORM_data1: case DW_FORM_data2: case DW_FORM_data4: case DW_FORM_data8: case DW_FORM_udata: case DW_FORM_sdata: return FormClass::Constant; case DW_FORM_string: case DW_FORM_strp: return FormClass::String; case DW_FORM_ref_addr: case DW_FORM_ref1: case DW_FORM_ref2: case DW_FORM_ref4: case DW_FORM_ref8: case DW_FORM_ref_udata: return FormClass::Reference; case DW_FORM_flag: case DW_FORM_flag_present: return FormClass::Flag; case DW_FORM_exprloc: return FormClass::ExprLoc; // TODO sec offset // TODO indirect default: return FormClass::Unknown; } } /** \brief Get the name of the tag of a given DIE * * \param die DIE * \return name of the tag of this DIE */ inline XBT_PRIVATE const char *tagname(Dwarf_Die * die) { return simgrid::dwarf::tagname(dwarf_tag(die)); } } } // ***** Attributes /** \brief Get an attribute of a given DIE as a string * * \param die the DIE * \param attribute attribute * \return value of the given attribute of the given DIE */ static const char *MC_dwarf_attr_integrate_string(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (not dwarf_attr_integrate(die, attribute, &attr)) return nullptr; else return dwarf_formstring(&attr); } /** \brief Get the linkage name of a DIE. * * Use either DW_AT_linkage_name or DW_AT_MIPS_linkage_name. * DW_AT_linkage_name is standardized since DWARF 4. * Before this version of DWARF, the MIPS extensions * DW_AT_MIPS_linkage_name is used (at least by GCC). * * \param die the DIE * \return linkage name of the given DIE (or nullptr) * */ static const char *MC_dwarf_at_linkage_name(Dwarf_Die * die) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_linkage_name); if (not name) name = MC_dwarf_attr_integrate_string(die, DW_AT_MIPS_linkage_name); return name; } static Dwarf_Off MC_dwarf_attr_dieoffset(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_hasattr_integrate(die, attribute) == 0) return 0; dwarf_attr_integrate(die, attribute, &attr); Dwarf_Die subtype_die; if (dwarf_formref_die(&attr, &subtype_die) == nullptr) xbt_die("Could not find DIE"); return dwarf_dieoffset(&subtype_die); } static Dwarf_Off MC_dwarf_attr_integrate_dieoffset(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_hasattr_integrate(die, attribute) == 0) return 0; dwarf_attr_integrate(die, DW_AT_type, &attr); Dwarf_Die subtype_die; if (dwarf_formref_die(&attr, &subtype_die) == nullptr) xbt_die("Could not find DIE"); return dwarf_dieoffset(&subtype_die); } /** \brief Find the type/subtype (DW_AT_type) for a DIE * * \param die the DIE * \return DW_AT_type reference as a global offset in hexadecimal (or nullptr) */ static std::uint64_t MC_dwarf_at_type(Dwarf_Die * die) { return MC_dwarf_attr_integrate_dieoffset(die, DW_AT_type); } static uint64_t MC_dwarf_attr_integrate_addr(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_attr_integrate(die, attribute, &attr) == nullptr) return 0; Dwarf_Addr value; if (dwarf_formaddr(&attr, &value) == 0) return (uint64_t) value; else return 0; } static uint64_t MC_dwarf_attr_integrate_uint(Dwarf_Die * die, int attribute, uint64_t default_value) { Dwarf_Attribute attr; if (dwarf_attr_integrate(die, attribute, &attr) == nullptr) return default_value; Dwarf_Word value; return dwarf_formudata(dwarf_attr_integrate(die, attribute, &attr), &value) == 0 ? (uint64_t) value : default_value; } static bool MC_dwarf_attr_flag(Dwarf_Die * die, int attribute, bool integrate) { Dwarf_Attribute attr; if ((integrate ? dwarf_attr_integrate(die, attribute, &attr) : dwarf_attr(die, attribute, &attr)) == 0) return false; bool result; if (dwarf_formflag(&attr, &result)) xbt_die("Unexpected form for attribute %s", simgrid::dwarf::attrname(attribute)); return result; } /** @brief Find the default lower bound for a given language * * The default lower bound of an array (when DW_TAG_lower_bound * is missing) depends on the language of the compilation unit. * * @param lang Language of the compilation unit (values defined in the DWARF spec) * @return Default lower bound of an array in this compilation unit * */ static uint64_t MC_dwarf_default_lower_bound(int lang) { switch (lang) { case DW_LANG_C: case DW_LANG_C89: case DW_LANG_C99: case DW_LANG_C_plus_plus: case DW_LANG_D: case DW_LANG_Java: case DW_LANG_ObjC: case DW_LANG_ObjC_plus_plus: case DW_LANG_Python: case DW_LANG_UPC: return 0; case DW_LANG_Ada83: case DW_LANG_Ada95: case DW_LANG_Fortran77: case DW_LANG_Fortran90: case DW_LANG_Fortran95: case DW_LANG_Modula2: case DW_LANG_Pascal83: case DW_LANG_PL1: case DW_LANG_Cobol74: case DW_LANG_Cobol85: return 1; default: xbt_die("No default DW_TAG_lower_bound for language %i and none given", lang); return 0; } } /** \brief Finds the number of elements in a DW_TAG_subrange_type or DW_TAG_enumeration_type DIE * * \param die the DIE * \param unit DIE of the compilation unit * \return number of elements in the range * */ static uint64_t MC_dwarf_subrange_element_count(Dwarf_Die * die, Dwarf_Die * unit) { xbt_assert(dwarf_tag(die) == DW_TAG_enumeration_type || dwarf_tag(die) == DW_TAG_subrange_type, "MC_dwarf_subrange_element_count called with DIE of type %s", simgrid::dwarf::tagname(die)); // Use DW_TAG_count if present: if (dwarf_hasattr_integrate(die, DW_AT_count)) return MC_dwarf_attr_integrate_uint(die, DW_AT_count, 0); // Otherwise compute DW_TAG_upper_bound-DW_TAG_lower_bound + 1: if (not dwarf_hasattr_integrate(die, DW_AT_upper_bound)) // This is not really 0, but the code expects this (we do not know): return 0; uint64_t upper_bound = MC_dwarf_attr_integrate_uint(die, DW_AT_upper_bound, -1); uint64_t lower_bound = 0; if (dwarf_hasattr_integrate(die, DW_AT_lower_bound)) lower_bound = MC_dwarf_attr_integrate_uint(die, DW_AT_lower_bound, -1); else lower_bound = MC_dwarf_default_lower_bound(dwarf_srclang(unit)); return upper_bound - lower_bound + 1; } /** \brief Finds the number of elements in a array type (DW_TAG_array_type) * * The compilation unit might be needed because the default lower * bound depends on the language of the compilation unit. * * \param die the DIE of the DW_TAG_array_type * \param unit the DIE of the compilation unit * \return number of elements in this array type * */ static uint64_t MC_dwarf_array_element_count(Dwarf_Die * die, Dwarf_Die * unit) { xbt_assert(dwarf_tag(die) == DW_TAG_array_type, "MC_dwarf_array_element_count called with DIE of type %s", simgrid::dwarf::tagname(die)); int result = 1; Dwarf_Die child; int res; for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) { int child_tag = dwarf_tag(&child); if (child_tag == DW_TAG_subrange_type || child_tag == DW_TAG_enumeration_type) result *= MC_dwarf_subrange_element_count(&child, unit); } return result; } // ***** Variable /** Sort the variable by name and address. * * We could use boost::container::flat_set instead. */ static bool MC_compare_variable( simgrid::mc::Variable const& a, simgrid::mc::Variable const& b) { int cmp = strcmp(a.name.c_str(), b.name.c_str()); if (cmp < 0) return true; else if (cmp > 0) return false; else return a.address < b.address; } // ***** simgrid::mc::Type* /** \brief Initialize the location of a member of a type * (DW_AT_data_member_location of a DW_TAG_member). * * \param type a type (struct, class) * \param member the member of the type * \param child DIE of the member (DW_TAG_member) */ static void MC_dwarf_fill_member_location( simgrid::mc::Type* type, simgrid::mc::Member* member, Dwarf_Die * child) { if (dwarf_hasattr(child, DW_AT_data_bit_offset)) xbt_die("Can't groke DW_AT_data_bit_offset."); if (not dwarf_hasattr_integrate(child, DW_AT_data_member_location)) { if (type->type == DW_TAG_union_type) return; xbt_die ("Missing DW_AT_data_member_location field in DW_TAG_member %s of type <%" PRIx64 ">%s", member->name.c_str(), (uint64_t) type->id, type->name.c_str()); } Dwarf_Attribute attr; dwarf_attr_integrate(child, DW_AT_data_member_location, &attr); int form = dwarf_whatform(&attr); simgrid::dwarf::FormClass form_class = simgrid::dwarf::classify_form(form); switch (form_class) { case simgrid::dwarf::FormClass::ExprLoc: case simgrid::dwarf::FormClass::Block: // Location expression: { Dwarf_Op *expr; size_t len; if (dwarf_getlocation(&attr, &expr, &len)) xbt_die ("Could not read location expression DW_AT_data_member_location in DW_TAG_member %s of type <%" PRIx64 ">%s", MC_dwarf_attr_integrate_string(child, DW_AT_name), (uint64_t) type->id, type->name.c_str()); member->location_expression = simgrid::dwarf::DwarfExpression(expr, expr+len); break; } case simgrid::dwarf::FormClass::Constant: // Offset from the base address of the object: { Dwarf_Word offset; if (not dwarf_formudata(&attr, &offset)) member->offset(offset); else xbt_die("Cannot get %s location <%" PRIx64 ">%s", MC_dwarf_attr_integrate_string(child, DW_AT_name), (uint64_t) type->id, type->name.c_str()); break; } default: // includes FormClass::LocListPtr (reference to a location list: TODO) and FormClass::Reference (it's supposed to be // possible in DWARF2 but I couldn't find its semantic in the spec) xbt_die("Can't handle form class (%d) / form 0x%x as DW_AT_member_location", (int)form_class, (unsigned)form); } } /** \brief Populate the list of members of a type * * \param info ELF object containing the type DIE * \param die DIE of the type * \param unit DIE of the compilation unit containing the type DIE * \param type the type */ static void MC_dwarf_add_members(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Type* type) { int res; Dwarf_Die child; xbt_assert(type->members.empty()); for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) { int tag = dwarf_tag(&child); if (tag == DW_TAG_member || tag == DW_TAG_inheritance) { // Skip declarations: if (MC_dwarf_attr_flag(&child, DW_AT_declaration, false)) continue; // Skip compile time constants: if (dwarf_hasattr(&child, DW_AT_const_value)) continue; // TODO, we should use another type (because is is not a type but a member) simgrid::mc::Member member; if (tag == DW_TAG_inheritance) member.flags |= simgrid::mc::Member::INHERITANCE_FLAG; const char *name = MC_dwarf_attr_integrate_string(&child, DW_AT_name); if (name) member.name = name; // Those base names are used by GCC and clang for virtual table pointers // respectively ("__vptr$ClassName", "__vptr.ClassName"): if (boost::algorithm::starts_with(member.name, "__vptr$") || boost::algorithm::starts_with(member.name, "__vptr.")) member.flags |= simgrid::mc::Member::VIRTUAL_POINTER_FLAG; // A cleaner solution would be to check against the type: // --- // tag: DW_TAG_member // name: "_vptr$Foo" // type: // # Type for a pointer to a vtable // tag: DW_TAG_pointer_type // type: // # Type for a vtable: // tag: DW_TAG_pointer_type // name: "__vtbl_ptr_type" // type: // tag: DW_TAG_subroutine_type // type: // tag: DW_TAG_base_type // name: "int" // --- member.byte_size = MC_dwarf_attr_integrate_uint(&child, DW_AT_byte_size, 0); member.type_id = MC_dwarf_at_type(&child); if (dwarf_hasattr(&child, DW_AT_data_bit_offset)) xbt_die("Can't groke DW_AT_data_bit_offset."); MC_dwarf_fill_member_location(type, &member, &child); if (not member.type_id) xbt_die("Missing type for member %s of <%" PRIx64 ">%s", member.name.c_str(), (uint64_t) type->id, type->name.c_str()); type->members.push_back(std::move(member)); } } } /** \brief Create a MC type object from a DIE * * \param info current object info object * \param die DIE (for a given type) * \param unit compilation unit of the current DIE * \return MC representation of the type */ static simgrid::mc::Type MC_dwarf_die_to_type( simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { simgrid::mc::Type type; type.type = dwarf_tag(die); type.name = std::string(); type.element_count = -1; // Global Offset type.id = dwarf_dieoffset(die); const char *prefix = ""; switch (type.type) { case DW_TAG_structure_type: prefix = "struct "; break; case DW_TAG_union_type: prefix = "union "; break; case DW_TAG_class_type: prefix = "class "; break; default: prefix = ""; } const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name != nullptr) { if (ns) type.name = simgrid::xbt::string_printf("%s%s::%s", prefix, ns, name); else type.name = simgrid::xbt::string_printf("%s%s", prefix, name); } type.type_id = MC_dwarf_at_type(die); // Some compilers do not emit DW_AT_byte_size for pointer_type, // so we fill this. We currently assume that the model-checked process is in // the same architecture.. if (type.type == DW_TAG_pointer_type) type.byte_size = sizeof(void*); // Computation of the byte_size if (dwarf_hasattr_integrate(die, DW_AT_byte_size)) type.byte_size = MC_dwarf_attr_integrate_uint(die, DW_AT_byte_size, 0); else if (type.type == DW_TAG_array_type || type.type == DW_TAG_structure_type || type.type == DW_TAG_class_type) { Dwarf_Word size; if (dwarf_aggregate_size(die, &size) == 0) type.byte_size = size; } switch (type.type) { case DW_TAG_array_type: type.element_count = MC_dwarf_array_element_count(die, unit); // TODO, handle DW_byte_stride and (not) DW_bit_stride break; case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: break; case DW_TAG_structure_type: case DW_TAG_union_type: case DW_TAG_class_type: MC_dwarf_add_members(info, die, unit, &type); MC_dwarf_handle_children(info, die, unit, frame, ns ? simgrid::xbt::string_printf("%s::%s", ns, name).c_str() : type.name.c_str()); break; default: XBT_DEBUG("Unhandled type: %d (%s)", type.type, simgrid::dwarf::tagname(type.type)); break; } return type; } static void MC_dwarf_handle_type_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { simgrid::mc::Type type = MC_dwarf_die_to_type(info, die, unit, frame, ns); auto& t = (info->types[type.id] = std::move(type)); if (not t.name.empty() && type.byte_size != 0) info->full_types_by_name[t.name] = &t; } static int mc_anonymous_variable_index = 0; static std::unique_ptr MC_die_to_variable( simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { // Skip declarations: if (MC_dwarf_attr_flag(die, DW_AT_declaration, false)) return nullptr; // Skip compile time constants: if (dwarf_hasattr(die, DW_AT_const_value)) return nullptr; Dwarf_Attribute attr_location; if (dwarf_attr(die, DW_AT_location, &attr_location) == nullptr) // No location: do not add it ? return nullptr; std::unique_ptr variable = std::unique_ptr(new simgrid::mc::Variable()); variable->id = dwarf_dieoffset(die); variable->global = frame == nullptr; // Can be override base on DW_AT_location variable->object_info = info; const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name) variable->name = name; variable->type_id = MC_dwarf_at_type(die); int form = dwarf_whatform(&attr_location); simgrid::dwarf::FormClass form_class; if (form == DW_FORM_sec_offset) form_class = simgrid::dwarf::FormClass::Constant; else form_class = simgrid::dwarf::classify_form(form); switch (form_class) { case simgrid::dwarf::FormClass::ExprLoc: case simgrid::dwarf::FormClass::Block: // Location expression: { Dwarf_Op *expr; size_t len; if (dwarf_getlocation(&attr_location, &expr, &len)) { xbt_die( "Could not read location expression in DW_AT_location " "of variable <%" PRIx64 ">%s", (uint64_t) variable->id, variable->name.c_str()); } if (len == 1 && expr[0].atom == DW_OP_addr) { variable->global = true; uintptr_t offset = (uintptr_t) expr[0].number; uintptr_t base = (uintptr_t) info->base_address(); variable->address = (void *) (base + offset); } else variable->location_list = { simgrid::dwarf::LocationListEntry(simgrid::dwarf::DwarfExpression(expr, expr + len))}; break; } case simgrid::dwarf::FormClass::LocListPtr: case simgrid::dwarf::FormClass::Constant: // Reference to location list: variable->location_list = simgrid::dwarf::location_list( *info, attr_location); break; default: xbt_die("Unexpected form 0x%x (%i), class 0x%x (%i) list for location in <%" PRIx64 ">%s", (unsigned)form, form, (unsigned)form_class, (int)form_class, (uint64_t)variable->id, variable->name.c_str()); } // Handle start_scope: if (dwarf_hasattr(die, DW_AT_start_scope)) { Dwarf_Attribute attr; dwarf_attr(die, DW_AT_start_scope, &attr); int form = dwarf_whatform(&attr); simgrid::dwarf::FormClass form_class = simgrid::dwarf::classify_form(form); if (form_class == simgrid::dwarf::FormClass::Constant) { Dwarf_Word value; variable->start_scope = dwarf_formudata(&attr, &value) == 0 ? (size_t)value : 0; } else { // TODO: FormClass::RangeListPtr xbt_die("Unhandled form 0x%x, class 0x%X for DW_AT_start_scope of variable %s", (unsigned)form, (unsigned)form_class, name == nullptr ? "?" : name); } } if (ns && variable->global) variable->name = std::string(ns) + "::" + variable->name; // The current code needs a variable name, // generate a fake one: if (variable->name.empty()) { variable->name = "@anonymous#" + std::to_string(mc_anonymous_variable_index); mc_anonymous_variable_index++; } return variable; } static void MC_dwarf_handle_variable_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { std::unique_ptr variable = MC_die_to_variable(info, die, unit, frame, ns); if (not variable) return; // Those arrays are sorted later: if (variable->global) info->global_variables.push_back(std::move(*variable)); else if (frame != nullptr) frame->variables.push_back(std::move(*variable)); else xbt_die("No frame for this local variable"); } static void MC_dwarf_handle_scope_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* parent_frame, const char *ns) { // TODO, handle DW_TAG_type/DW_TAG_location for DW_TAG_with_stmt int tag = dwarf_tag(die); simgrid::dwarf::TagClass klass = simgrid::dwarf::classify_tag(tag); // (Template) Subprogram declaration: if (klass == simgrid::dwarf::TagClass::Subprogram && MC_dwarf_attr_flag(die, DW_AT_declaration, false)) return; if (klass == simgrid::dwarf::TagClass::Scope) xbt_assert(parent_frame, "No parent scope for this scope"); simgrid::mc::Frame frame; frame.tag = tag; frame.id = dwarf_dieoffset(die); frame.object_info = info; if (klass == simgrid::dwarf::TagClass::Subprogram) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name && ns) frame.name = std::string(ns) + "::" + name; else if (name) frame.name = name; } frame.abstract_origin_id = MC_dwarf_attr_dieoffset(die, DW_AT_abstract_origin); // This is the base address for DWARF addresses. // Relocated addresses are offset from this base address. // See DWARF4 spec 7.5 std::uint64_t base = (std::uint64_t) info->base_address(); // TODO, support DW_AT_ranges uint64_t low_pc = MC_dwarf_attr_integrate_addr(die, DW_AT_low_pc); frame.range.begin() = low_pc ? (std::uint64_t) base + low_pc : 0; if (low_pc) { // DW_AT_high_pc: Dwarf_Attribute attr; if (not dwarf_attr_integrate(die, DW_AT_high_pc, &attr)) xbt_die("Missing DW_AT_high_pc matching with DW_AT_low_pc"); Dwarf_Sword offset; Dwarf_Addr high_pc; switch (simgrid::dwarf::classify_form(dwarf_whatform(&attr))) { // DW_AT_high_pc if an offset from the low_pc: case simgrid::dwarf::FormClass::Constant: if (dwarf_formsdata(&attr, &offset) != 0) xbt_die("Could not read constant"); frame.range.end() = frame.range.begin() + offset; break; // DW_AT_high_pc is a relocatable address: case simgrid::dwarf::FormClass::Address: if (dwarf_formaddr(&attr, &high_pc) != 0) xbt_die("Could not read address"); frame.range.end() = base + high_pc; break; default: xbt_die("Unexpected class for DW_AT_high_pc"); } } if (klass == simgrid::dwarf::TagClass::Subprogram) { Dwarf_Attribute attr_frame_base; if (dwarf_attr_integrate(die, DW_AT_frame_base, &attr_frame_base)) frame.frame_base_location = simgrid::dwarf::location_list(*info, attr_frame_base); } // Handle children: MC_dwarf_handle_children(info, die, unit, &frame, ns); // We sort them in order to have an (somewhat) efficient by name // lookup: boost::range::sort(frame.variables, MC_compare_variable); // Register it: if (klass == simgrid::dwarf::TagClass::Subprogram) info->subprograms[frame.id] = std::move(frame); else if (klass == simgrid::dwarf::TagClass::Scope) parent_frame->scopes.push_back(std::move(frame)); } static void mc_dwarf_handle_namespace_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (frame) xbt_die("Unexpected namespace in a subprogram"); char *new_ns = ns == nullptr ? xbt_strdup(name) : bprintf("%s::%s", ns, name); MC_dwarf_handle_children(info, die, unit, frame, new_ns); xbt_free(new_ns); } static void MC_dwarf_handle_children(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { // For each child DIE: Dwarf_Die child; int res; for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) MC_dwarf_handle_die(info, &child, unit, frame, ns); } static void MC_dwarf_handle_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { int tag = dwarf_tag(die); simgrid::dwarf::TagClass klass = simgrid::dwarf::classify_tag(tag); switch (klass) { // Type: case simgrid::dwarf::TagClass::Type: MC_dwarf_handle_type_die(info, die, unit, frame, ns); break; // Subprogram or scope: case simgrid::dwarf::TagClass::Subprogram: case simgrid::dwarf::TagClass::Scope: MC_dwarf_handle_scope_die(info, die, unit, frame, ns); return; // Variable: case simgrid::dwarf::TagClass::Variable: MC_dwarf_handle_variable_die(info, die, unit, frame, ns); break; case simgrid::dwarf::TagClass::Namespace: mc_dwarf_handle_namespace_die(info, die, unit, frame, ns); break; default: break; } } static Elf64_Half get_type(Elf* elf) { Elf64_Ehdr* ehdr64 = elf64_getehdr(elf); if (ehdr64) return ehdr64->e_type; Elf32_Ehdr* ehdr32 = elf32_getehdr(elf); if (ehdr32) return ehdr32->e_type; xbt_die("Could not get ELF heeader"); } static void read_dwarf_info(simgrid::mc::ObjectInformation* info, Dwarf* dwarf) { // For each compilation unit: Dwarf_Off offset = 0; Dwarf_Off next_offset = 0; size_t length; while (dwarf_nextcu(dwarf, offset, &next_offset, &length, nullptr, nullptr, nullptr) == 0) { Dwarf_Die unit_die; if (dwarf_offdie(dwarf, offset + length, &unit_die) != nullptr) MC_dwarf_handle_children(info, &unit_die, &unit_die, nullptr, nullptr); offset = next_offset; } } /** Get the build-id (NT_GNU_BUILD_ID) from the ELF file * * This build-id may is used to locate an external debug (DWARF) file * for this ELF file. * * @param elf libelf handle for an ELF file * @return build-id for this ELF file (or an empty vector if none is found) */ static std::vector get_build_id(Elf* elf) { #ifdef __linux // Summary: the GNU build ID is stored in a ("GNU, NT_GNU_BUILD_ID) note // found in a PT_NOTE entry in the program header table. size_t phnum; if (elf_getphdrnum (elf, &phnum) != 0) xbt_die("Could not read program headers"); // Iterate over the program headers and find the PT_NOTE ones: for (size_t i = 0; i < phnum; ++i) { GElf_Phdr phdr_temp; GElf_Phdr *phdr = gelf_getphdr(elf, i, &phdr_temp); if (phdr->p_type != PT_NOTE) continue; Elf_Data* data = elf_getdata_rawchunk(elf, phdr->p_offset, phdr->p_filesz, ELF_T_NHDR); // Iterate over the notes and find the NT_GNU_BUILD_ID one: size_t pos = 0; while (pos < data->d_size) { GElf_Nhdr nhdr; // Location of the name within Elf_Data: size_t name_pos; size_t desc_pos; pos = gelf_getnote(data, pos, &nhdr, &name_pos, &desc_pos); // A build ID note is identified by the pair ("GNU", NT_GNU_BUILD_ID) // (a namespace and a type within this namespace): if (nhdr.n_type == NT_GNU_BUILD_ID && nhdr.n_namesz == sizeof("GNU") && memcmp((char*) data->d_buf + name_pos, "GNU", sizeof("GNU")) == 0) { XBT_DEBUG("Found GNU/NT_GNU_BUILD_ID note"); char* start = (char*) data->d_buf + desc_pos; char* end = (char*) start + nhdr.n_descsz; return std::vector(start, end); } } } #endif return std::vector(); } static char hexdigits[16] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' }; /** Binary data to hexadecimal */ static inline std::array to_hex(std::uint8_t byte) { // Horrid double braces! // Apparently, this is needed in C++11 (not in C++14). return { { hexdigits[byte >> 4], hexdigits[byte & 0xF] } }; } /** Binary data to hexadecimal */ static std::string to_hex(const char* data, std::size_t count) { std::string res; res.resize(2*count); for (std::size_t i = 0; i < count; i++) { std::array hex_byte = to_hex(data[i]); for (int j = 0; j < 2; ++j) res[2 * i + j] = hex_byte[j]; } return res; } /** Binary data to hexadecimal */ static std::string to_hex(std::vector const& data) { return to_hex(data.data(), data.size()); } /** Base directories for external debug files */ static const char* debug_paths[] = { "/usr/lib/debug/", "/usr/local/lib/debug/", }; /** Locate an external debug file from the NT_GNU_BUILD_ID * * This is one of the mechanisms used for * [separate debug files](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html). */ // Example: // /usr/lib/debug/.build-id/0b/dc77f1c29aea2b14ff5acd9a19ab3175ffdeae.debug static std::string find_by_build_id(std::vector id) { std::string filename; std::string hex = to_hex(id); for (const char* const& debug_path : debug_paths) { // Example: filename = std::string(debug_path) + ".build-id/" + to_hex(id.data(), 1) + '/' + to_hex(id.data() + 1, id.size() - 1) + ".debug"; XBT_DEBUG("Checking debug file: %s", filename.c_str()); if (access(filename.c_str(), F_OK) == 0) { XBT_DEBUG("Found debug file: %s\n", hex.c_str()); return filename; } } XBT_DEBUG("Not debuf info found for build ID %s\n", hex.data()); return std::string(); } /** \brief Populate the debugging informations of the given ELF object * * Read the DWARf information of the EFFL object and populate the * lists of types, variables, functions. */ static void MC_load_dwarf(simgrid::mc::ObjectInformation* info) { if (elf_version(EV_CURRENT) == EV_NONE) xbt_die("libelf initialization error"); // Open the ELF file: int fd = open(info->file_name.c_str(), O_RDONLY); if (fd < 0) xbt_die("Could not open file %s", info->file_name.c_str()); Elf* elf = elf_begin(fd, ELF_C_READ, nullptr); if (elf == nullptr) xbt_die("Not an ELF file"); Elf_Kind kind = elf_kind(elf); if (kind != ELF_K_ELF) xbt_die("Not an ELF file"); // Remember if this is a `ET_EXEC` (fixed location) or `ET_DYN`: Elf64_Half type = get_type(elf); if (type == ET_EXEC) info->flags |= simgrid::mc::ObjectInformation::Executable; // Read DWARF debug information in the file: Dwarf* dwarf = dwarf_begin_elf (elf, DWARF_C_READ, nullptr); if (dwarf != nullptr) { read_dwarf_info(info, dwarf); dwarf_end(dwarf); elf_end(elf); close(fd); return; } dwarf_end(dwarf); // If there was no DWARF in the file, try to find it in a separate file. // Different methods might be used to store the DWARF informations: // * GNU NT_GNU_BUILD_ID // * .gnu_debuglink // See https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html // for reference of what we are doing. // Try with NT_GNU_BUILD_ID: we find the build ID in the ELF file and then // use this ID to find the file in some known locations in the filesystem. std::vector build_id = get_build_id(elf); if (not build_id.empty()) { elf_end(elf); close(fd); // Find the debug file using the build id: std::string debug_file = find_by_build_id(build_id); if (debug_file.empty()) { std::string hex = to_hex(build_id); xbt_die("Missing debug info for %s with build-id %s\n" "You might want to install the suitable debugging package.\n", info->file_name.c_str(), hex.c_str()); } // Load the DWARF info from this file: XBT_DEBUG("Load DWARF for %s from %s", info->file_name.c_str(), debug_file.c_str()); fd = open(debug_file.c_str(), O_RDONLY); if (fd < 0) xbt_die("Could not open file %s", debug_file.c_str()); Dwarf* dwarf = dwarf_begin(fd, DWARF_C_READ); if (dwarf == nullptr) xbt_die("No DWARF info in %s for %s", debug_file.c_str(), info->file_name.c_str()); read_dwarf_info(info, dwarf); dwarf_end(dwarf); close(fd); return; } // TODO, try to find DWARF info using .gnu_debuglink. elf_end(elf); close(fd); xbt_die("Debugging information not found for %s\n" "Try recompiling with -g\n", info->file_name.c_str()); } // ***** Functions index static int MC_compare_frame_index_items(simgrid::mc::FunctionIndexEntry* a, simgrid::mc::FunctionIndexEntry* b) { if (a->low_pc < b->low_pc) return -1; else if (a->low_pc == b->low_pc) return 0; else return 1; } static void MC_make_functions_index(simgrid::mc::ObjectInformation* info) { info->functions_index.clear(); for (auto& e : info->subprograms) { if (e.second.range.begin() == 0) continue; simgrid::mc::FunctionIndexEntry entry; entry.low_pc = (void*) e.second.range.begin(); entry.function = &e.second; info->functions_index.push_back(entry); } info->functions_index.shrink_to_fit(); // Sort the array by low_pc: boost::range::sort(info->functions_index, [](simgrid::mc::FunctionIndexEntry const& a, simgrid::mc::FunctionIndexEntry const& b) { return a.low_pc < b.low_pc; }); } static void MC_post_process_variables(simgrid::mc::ObjectInformation* info) { // Someone needs this to be sorted but who? boost::range::sort(info->global_variables, MC_compare_variable); for (simgrid::mc::Variable& variable : info->global_variables) if (variable.type_id) variable.type = simgrid::util::find_map_ptr( info->types, variable.type_id); } static void mc_post_process_scope(simgrid::mc::ObjectInformation* info, simgrid::mc::Frame* scope) { if (scope->tag == DW_TAG_inlined_subroutine) { // Attach correct namespaced name in inlined subroutine: auto i = info->subprograms.find(scope->abstract_origin_id); xbt_assert(i != info->subprograms.end(), "Could not lookup abstract origin %" PRIx64, (std::uint64_t) scope->abstract_origin_id); scope->name = i->second.name; } // Direct: for (simgrid::mc::Variable& variable : scope->variables) if (variable.type_id) variable.type = simgrid::util::find_map_ptr( info->types, variable.type_id); // Recursive post-processing of nested-scopes: for (simgrid::mc::Frame& nested_scope : scope->scopes) mc_post_process_scope(info, &nested_scope); } static simgrid::mc::Type* MC_resolve_type( simgrid::mc::ObjectInformation* info, unsigned type_id) { if (not type_id) return nullptr; simgrid::mc::Type* type = simgrid::util::find_map_ptr(info->types, type_id); if (type == nullptr) return nullptr; // We already have the information on the type: if (type->byte_size != 0) return type; // Don't have a name, we can't find a more complete version: if (type->name.empty()) return type; // Try to find a more complete description of the type: // We need to fix in order to support C++. simgrid::mc::Type** subtype = simgrid::util::find_map_ptr( info->full_types_by_name, type->name); if (subtype) type = *subtype; return type; } static void MC_post_process_types(simgrid::mc::ObjectInformation* info) { // Lookup "subtype" field: for (auto& i : info->types) { i.second.subtype = MC_resolve_type(info, i.second.type_id); for (simgrid::mc::Member& member : i.second.members) member.type = MC_resolve_type(info, member.type_id); } } namespace simgrid { namespace mc { /** \brief Finds informations about a given shared object/executable */ std::shared_ptr createObjectInformation( std::vector const& maps, const char *name) { std::shared_ptr result = std::make_shared(); result->file_name = name; simgrid::mc::find_object_address(maps, result.get()); MC_load_dwarf(result.get()); MC_post_process_variables(result.get()); MC_post_process_types(result.get()); for (auto& entry : result.get()->subprograms) mc_post_process_scope(result.get(), &entry.second); MC_make_functions_index(result.get()); return result; } /*************************************************************************/ void postProcessObjectInformation(simgrid::mc::RemoteClient* process, simgrid::mc::ObjectInformation* info) { for (auto& i : info->types) { simgrid::mc::Type* type = &(i.second); simgrid::mc::Type* subtype = type; while (subtype->type == DW_TAG_typedef || subtype->type == DW_TAG_volatile_type || subtype->type == DW_TAG_const_type) if (subtype->subtype) subtype = subtype->subtype; else break; // Resolve full_type: if (not subtype->name.empty() && subtype->byte_size == 0) for (auto const& object_info : process->object_infos) { auto i = object_info->full_types_by_name.find(subtype->name); if (i != object_info->full_types_by_name.end() && not i->second->name.empty() && i->second->byte_size) { type->full_type = i->second; break; } } else type->full_type = subtype; } } } } namespace simgrid { namespace dwarf { /** Convert a DWARF register into a libunwind register * * DWARF and libunwind does not use the same convention for numbering the * registers on some architectures. The function makes the necessary * conversion. */ int dwarf_register_to_libunwind(int dwarf_register) { #if defined(__x86_64__) // It seems for this arch, DWARF and libunwind agree in the numbering: return dwarf_register; #elif defined(__i386__) // Couldn't find the authoritative source of information for this. // This is inspired from http://source.winehq.org/source/dlls/dbghelp/cpu_i386.c#L517. switch (dwarf_register) { case 0: return UNW_X86_EAX; case 1: return UNW_X86_ECX; case 2: return UNW_X86_EDX; case 3: return UNW_X86_EBX; case 4: return UNW_X86_ESP; case 5: return UNW_X86_EBP; case 6: return UNW_X86_ESI; case 7: return UNW_X86_EDI; case 8: return UNW_X86_EIP; case 9: return UNW_X86_EFLAGS; case 10: return UNW_X86_CS; case 11: return UNW_X86_SS; case 12: return UNW_X86_DS; case 13: return UNW_X86_ES; case 14: return UNW_X86_FS; case 15: return UNW_X86_GS; case 16: return UNW_X86_ST0; case 17: return UNW_X86_ST1; case 18: return UNW_X86_ST2; case 19: return UNW_X86_ST3; case 20: return UNW_X86_ST4; case 21: return UNW_X86_ST5; case 22: return UNW_X86_ST6; case 23: return UNW_X86_ST7; default: xbt_die("Bad/unknown register number."); } #else #error This architecture is not supported yet for DWARF expression evaluation. #endif } } } SimGrid-3.18/src/mc/mc_comm_pattern.cpp0000644000175000017500000000737213217757316020375 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "xbt/dynar.h" #include "xbt/sysdep.h" #include #include "src/mc/checker/CommunicationDeterminismChecker.hpp" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_xbt.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_comm_pattern, mc, "Logging specific to MC communication patterns"); static void MC_patterns_copy(xbt_dynar_t dest, std::vector const& source) { xbt_dynar_reset(dest); for (simgrid::mc::PatternCommunication const& comm : source) { simgrid::mc::PatternCommunication* copy_comm = new simgrid::mc::PatternCommunication(comm.dup()); xbt_dynar_push(dest, ©_comm); } } void MC_restore_communications_pattern(simgrid::mc::State* state) { simgrid::mc::PatternCommunicationList* list_process_comm; unsigned int cursor; xbt_dynar_foreach(initial_communications_pattern, cursor, list_process_comm) list_process_comm->index_comm = state->communicationIndices[cursor]; for (unsigned i = 0; i < MC_smx_get_maxpid(); i++) MC_patterns_copy( xbt_dynar_get_as(incomplete_communications_pattern, i, xbt_dynar_t), state->incomplete_comm_pattern[i] ); } void MC_state_copy_incomplete_communications_pattern(simgrid::mc::State* state) { state->incomplete_comm_pattern.clear(); for (unsigned i=0; i < MC_smx_get_maxpid(); i++) { xbt_dynar_t patterns = xbt_dynar_get_as(incomplete_communications_pattern, i, xbt_dynar_t); std::vector res; simgrid::mc::PatternCommunication* comm; unsigned int cursor; xbt_dynar_foreach(patterns, cursor, comm) res.push_back(comm->dup()); state->incomplete_comm_pattern.push_back(std::move(res)); } } void MC_state_copy_index_communications_pattern(simgrid::mc::State* state) { state->communicationIndices.clear(); simgrid::mc::PatternCommunicationList* list_process_comm; unsigned int cursor; xbt_dynar_foreach(initial_communications_pattern, cursor, list_process_comm) state->communicationIndices.push_back(list_process_comm->index_comm); } void MC_handle_comm_pattern( e_mc_call_type_t call_type, smx_simcall_t req, int value, xbt_dynar_t pattern, int backtracking) { // HACK, do not rely on the Checker implementation outside of it simgrid::mc::CommunicationDeterminismChecker* checker = (simgrid::mc::CommunicationDeterminismChecker*) mc_model_checker->getChecker(); switch(call_type) { case MC_CALL_TYPE_NONE: break; case MC_CALL_TYPE_SEND: case MC_CALL_TYPE_RECV: checker->get_comm_pattern(pattern, req, call_type, backtracking); break; case MC_CALL_TYPE_WAIT: case MC_CALL_TYPE_WAITANY: { simgrid::mc::RemotePtr comm_addr{nullptr}; if (call_type == MC_CALL_TYPE_WAIT) comm_addr = remote(static_cast(simcall_comm_wait__getraw__comm(req))); else { simgrid::kernel::activity::CommImpl* addr; // comm_addr = REMOTE(xbt_dynar_get_as(simcall_comm_waitany__get__comms(req), value, smx_synchro_t)): simgrid::mc::read_element(mc_model_checker->process(), &addr, remote(simcall_comm_waitany__get__comms(req)), value, sizeof(comm_addr)); comm_addr = remote(addr); } checker->complete_comm_pattern(pattern, comm_addr, MC_smx_simcall_get_issuer(req)->pid, backtracking); } break; default: xbt_die("Unexpected call type %i", (int)call_type); } } SimGrid-3.18/src/mc/mc_page_snapshot.cpp0000644000175000017500000000341013217757316020525 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* MC interface: definitions that non-MC modules must see, but not the user */ #include // pread, pwrite #include "src/mc/PageStore.hpp" #include "src/mc/mc_mmu.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_snapshot.hpp" #include #include "src/mc/ChunkedData.hpp" using simgrid::mc::remote; extern "C" { /** @brief Restore a snapshot of a region * * If possible, the restoration will be incremental * (the modified pages will not be touched). * * @param start_addr * @param page_count Number of pages of the region * @param pagenos */ void mc_restore_page_snapshot_region(simgrid::mc::RemoteClient* process, void* start_addr, simgrid::mc::ChunkedData const& pages_copy) { for (size_t i = 0; i != pages_copy.page_count(); ++i) { // Otherwise, copy the page: void* target_page = (void*) simgrid::mc::mmu::join(i, (std::uintptr_t) start_addr); const void* source_page = pages_copy.page(i); process->write_bytes(source_page, xbt_pagesize, remote(target_page)); } } // ***** High level API void mc_region_restore_sparse(simgrid::mc::RemoteClient* process, mc_mem_region_t reg) { xbt_assert(((reg->permanent_address().address()) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); xbt_assert(simgrid::mc::mmu::chunkCount(reg->size()) == reg->page_data().page_count()); mc_restore_page_snapshot_region(process, (void*) reg->permanent_address().address(), reg->page_data()); } } SimGrid-3.18/src/mc/VisitedState.hpp0000644000175000017500000000241113217757316017630 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_VISITED_STATE_HPP #define SIMGRID_MC_VISITED_STATE_HPP #include #include #include "src/mc/mc_snapshot.hpp" #include "src/mc/mc_state.hpp" namespace simgrid { namespace mc { class XBT_PRIVATE VisitedState { public: std::shared_ptr system_state = nullptr; std::size_t heap_bytes_used = 0; int actors_count = 0; int num = 0; // unique id of that state in the storage of all stored IDs int original_num = 0; // num field of the VisitedState to which I was declared equal to (used for dot_output) explicit VisitedState(unsigned long state_number); ~VisitedState() = default; }; class XBT_PRIVATE VisitedStates { std::vector> states_; public: void clear() { states_.clear(); } std::unique_ptr addVisitedState(unsigned long state_number, simgrid::mc::State* graph_state, bool compare_snpashots); private: void prune(); }; } } #endif SimGrid-3.18/src/mc/mc_smx.hpp0000644000175000017500000000323213217757316016510 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SMX_HPP #define SIMGRID_MC_SMX_HPP #include "src/mc/remote/RemoteClient.hpp" /** @file * @brief (Cross-process, MCer/MCed) Access to SMX structures * * We copy some C data structure from the MCed process in the MCer process. * This is implemented by: * * - `model_checker->process.smx_process_infos` * (copy of `simix_global->process_list`); * * - `model_checker->process.smx_old_process_infos` * (copy of `simix_global->process_to_destroy`); * * - `model_checker->hostnames`. * * The process lists are currently refreshed each time MCed code is executed. * We don't try to give a persistent MCer address for a given MCed process. * For this reason, a MCer simgrid::mc::Process* is currently not reusable after * MCed code. */ extern "C" { /** Get the issuer of a simcall (`req->issuer`) * * In split-process mode, it does the black magic necessary to get an address * of a (shallow) copy of the data structure the issuer SIMIX process in the local * address space. * * @param process the MCed process * @param req the simcall (copied in the local process) */ XBT_PRIVATE smx_actor_t MC_smx_simcall_get_issuer(s_smx_simcall_t const* req); XBT_PRIVATE const char* MC_smx_actor_get_name(smx_actor_t p); XBT_PRIVATE const char* MC_smx_actor_get_host_name(smx_actor_t p); XBT_PRIVATE int MC_smpi_process_count(void); XBT_PRIVATE unsigned long MC_smx_get_maxpid(void); } #endif SimGrid-3.18/src/mc/mc_request.hpp0000644000175000017500000000142113217757316017367 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REQUEST_HPP #define SIMGRID_MC_REQUEST_HPP #include "src/simix/smx_private.hpp" namespace simgrid { namespace mc { enum class RequestType { simix, executed, internal, }; XBT_PRIVATE bool request_depend(smx_simcall_t req1, smx_simcall_t req2); XBT_PRIVATE std::string request_to_string(smx_simcall_t req, int value, simgrid::mc::RequestType type); XBT_PRIVATE bool request_is_enabled_by_idx(smx_simcall_t req, unsigned int idx); XBT_PRIVATE std::string request_get_dot_output(smx_simcall_t req, int value); } } #endif SimGrid-3.18/src/mc/mc_smx.cpp0000644000175000017500000001455613217757316016516 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/Host.hpp" #include "src/mc/ModelChecker.hpp" #include "src/mc/mc_smx.hpp" using simgrid::mc::remote; /** HACK, Statically "upcast" a s_smx_actor_t into a ActorInformation * * This gets 'actorInfo' from '&actorInfo->copy'. It upcasts in the * sense that we could achieve the same thing by having ActorInformation * inherit from s_smx_actor_t but we don't really want to do that. */ static inline simgrid::mc::ActorInformation* actor_info_cast(smx_actor_t actor) { simgrid::mc::ActorInformation temp; std::size_t offset = (char*) temp.copy.getBuffer() - (char*)&temp; simgrid::mc::ActorInformation* process_info = (simgrid::mc::ActorInformation*)((char*)actor - offset); return process_info; } /** Load the remote list of processes into a vector * * @param process MCed process * @param target Local vector (to be filled with copies of `s_smx_actor_t`) * @param remote_dynar Address of the process dynar in the remote list */ static void MC_process_refresh_simix_actor_dynar(simgrid::mc::RemoteClient* process, std::vector& target, simgrid::mc::RemotePtr remote_dynar) { target.clear(); s_xbt_dynar_t dynar; process->read_bytes(&dynar, sizeof(dynar), remote_dynar); smx_actor_t* data = static_cast(::operator new(dynar.elmsize * dynar.used)); process->read_bytes(data, dynar.elmsize * dynar.used, simgrid::mc::RemotePtr(dynar.data)); // Load each element of the vector from the MCed process: for (unsigned int i = 0; i < dynar.used; ++i) { simgrid::mc::ActorInformation info; info.address = simgrid::mc::RemotePtr(data[i]); info.hostname = nullptr; process->read_bytes(&info.copy, sizeof(info.copy), remote(data[i])); target.push_back(std::move(info)); } ::operator delete(data); } namespace simgrid { namespace mc { void RemoteClient::refresh_simix() { if (this->cache_flags_ & RemoteClient::cache_simix_processes) return; // TODO, avoid to reload `&simix_global`, `simix_global`, `*simix_global` static_assert(std::is_same< std::unique_ptr, decltype(simix_global) >::value, "Unexpected type for simix_global"); static_assert(sizeof(simix_global) == sizeof(simgrid::simix::Global*), "Bad size for simix_global"); RemotePtr simix_global_p{this->read_variable("simix_global")}; // simix_global = REMOTE(*simix_global) Remote simix_global = this->read(simix_global_p); MC_process_refresh_simix_actor_dynar(this, this->smx_actors_infos, remote(simix_global.getBuffer()->actors_vector)); MC_process_refresh_simix_actor_dynar(this, this->smx_dead_actors_infos, remote(simix_global.getBuffer()->dead_actors_vector)); this->cache_flags_ |= RemoteClient::cache_simix_processes; } } } /** Get the issuer of a simcall (`req->issuer`) * * In split-process mode, it does the black magic necessary to get an address * of a (shallow) copy of the data structure the issuer SIMIX actor in the local * address space. * * @param process the MCed process * @param req the simcall (copied in the local process) */ smx_actor_t MC_smx_simcall_get_issuer(s_smx_simcall_t const* req) { xbt_assert(mc_model_checker != nullptr); // This is the address of the smx_actor in the MCed process: auto address = simgrid::mc::remote(req->issuer); // Lookup by address: for (auto& actor : mc_model_checker->process().actors()) if (actor.address == address) return actor.copy.getBuffer(); for (auto& actor : mc_model_checker->process().dead_actors()) if (actor.address == address) return actor.copy.getBuffer(); xbt_die("Issuer not found"); } const char* MC_smx_actor_get_host_name(smx_actor_t actor) { if (mc_model_checker == nullptr) return actor->host->getCname(); simgrid::mc::RemoteClient* process = &mc_model_checker->process(); /* HACK, Horrible hack to find the offset of the id in the simgrid::s4u::Host. Offsetof is not supported for non-POD types but this should work in practice for the targets currently supported by the MC as long as we do not add funny features to the Host class (such as virtual base). We are using a (C++11) unrestricted union in order to avoid any construction/destruction of the simgrid::s4u::Host. */ union fake_host { simgrid::s4u::Host host; fake_host() { /* Nothing to do*/} ~fake_host() { /* Nothing to do*/} }; fake_host foo; const size_t offset = (char*)&foo.host.getName() - (char*)&foo.host; // Read the simgrid::xbt::string in the MCed process: simgrid::mc::ActorInformation* info = actor_info_cast(actor); auto remote_string_address = remote((simgrid::xbt::string_data*)((char*)actor->host + offset)); simgrid::xbt::string_data remote_string = process->read(remote_string_address); char hostname[remote_string.len]; process->read_bytes(hostname, remote_string.len + 1, remote(remote_string.data)); info->hostname = mc_model_checker->get_host_name(hostname).c_str(); return info->hostname; } const char* MC_smx_actor_get_name(smx_actor_t actor) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); if (mc_model_checker == nullptr) return actor->name.c_str(); simgrid::mc::ActorInformation* info = actor_info_cast(actor); if (info->name.empty()) { simgrid::xbt::string_data string_data = simgrid::xbt::string::to_string_data(actor->name); info->name = process->read_string(remote(string_data.data), string_data.len); } return info->name.c_str(); } #if HAVE_SMPI int MC_smpi_process_count() { if (mc_model_checker == nullptr) return smpi_process_count(); int res; mc_model_checker->process().read_variable("process_count", &res, sizeof(res)); return res; } #endif unsigned long MC_smx_get_maxpid() { unsigned long maxpid; mc_model_checker->process().read_variable("simix_process_maxpid", &maxpid, sizeof(maxpid)); return maxpid; } SimGrid-3.18/src/mc/mc_safety.hpp0000644000175000017500000000074513217757316017202 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SAFETY_HPP #define SIMGRID_MC_SAFETY_HPP #include "xbt/base.h" namespace simgrid { namespace mc { enum class ReductionMode { unset, none, dpor, }; extern XBT_PRIVATE simgrid::mc::ReductionMode reduction_mode; } } #endif SimGrid-3.18/src/mc/mc_private.hpp0000644000175000017500000000307613217757316017361 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PRIVATE_HPP #define SIMGRID_MC_PRIVATE_HPP #include "mc/mc.h" #include "xbt/automaton.h" #include "src/mc/mc_forward.hpp" #include "src/xbt/memory_map.hpp" namespace simgrid { namespace mc { struct DerefAndCompareByActorsCountAndUsedHeap { template bool operator()(X const& a, Y const& b) { return std::make_pair(a->actors_count, a->heap_bytes_used) < std::make_pair(b->actors_count, b->heap_bytes_used); } }; } } extern "C" { /********************************* MC Global **********************************/ XBT_PRIVATE void MC_init_dot_output(); XBT_PRIVATE extern FILE* dot_output; XBT_PRIVATE void MC_show_deadlock(void); /********************************** Snapshot comparison **********************************/ //#define MC_DEBUG 1 #define MC_VERBOSE 1 /********************************** Miscellaneous **********************************/ } namespace simgrid { namespace mc { XBT_PRIVATE void find_object_address(std::vector const& maps, simgrid::mc::ObjectInformation* result); XBT_PRIVATE int snapshot_compare(int num1, simgrid::mc::Snapshot* s1, int num2, simgrid::mc::Snapshot* s2); // Move is somewhere else (in the LivenessChecker class, in the Session class?): extern XBT_PRIVATE xbt_automaton_t property_automaton; } } #endif SimGrid-3.18/src/mc/mc_global.cpp0000644000175000017500000001105713217757315017137 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "xbt/automaton.h" #include "xbt/backtrace.hpp" #include "xbt/dynar.h" #include "mc_base.h" #include "mc/mc.h" #ifndef _WIN32 #include #include #include #endif #include "src/simix/ActorImpl.hpp" #if SIMGRID_HAVE_MC #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_safety.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_snapshot.hpp" #include "src/mc/mc_unw.hpp" #include #endif #include "src/mc/Transition.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/remote/Client.hpp" #include "src/mc/remote/mc_protocol.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_global, mc, "Logging specific to MC (global)"); extern std::string _sg_mc_dot_output_file; namespace simgrid { namespace mc { std::vector processes_time; } } #if SIMGRID_HAVE_MC /* Liveness */ namespace simgrid { namespace mc { xbt_automaton_t property_automaton = nullptr; } } /* Dot output */ FILE *dot_output = nullptr; /******************************* Initialization of MC *******************************/ /*********************************************************************************/ void MC_init_dot_output() { dot_output = fopen(_sg_mc_dot_output_file.c_str(), "w"); if (dot_output == nullptr) { perror("Error open dot output file"); xbt_abort(); } fprintf(dot_output, "digraph graphname{\n fixedsize=true; rankdir=TB; ranksep=.25; edge [fontsize=12]; node [fontsize=10, shape=circle,width=.5 ]; graph [resolution=20, fontsize=10];\n"); } /******************************* Core of MC *******************************/ /**************************************************************************/ void MC_run() { simgrid::mc::processes_time.resize(SIMIX_process_get_maxpid()); MC_ignore_heap(simgrid::mc::processes_time.data(), simgrid::mc::processes_time.size() * sizeof(simgrid::mc::processes_time[0])); simgrid::mc::Client::get()->mainLoop(); simgrid::mc::processes_time.clear(); } void MC_show_deadlock() { XBT_INFO("**************************"); XBT_INFO("*** DEAD-LOCK DETECTED ***"); XBT_INFO("**************************"); XBT_INFO("Counter-example execution trace:"); for (auto const& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); } void MC_automaton_load(const char *file) { if (simgrid::mc::property_automaton == nullptr) simgrid::mc::property_automaton = xbt_automaton_new(); xbt_automaton_load(simgrid::mc::property_automaton, file); } namespace simgrid { namespace mc { void dumpStack(FILE* file, unw_cursor_t cursor) { int nframe = 0; char buffer[100]; unw_word_t off; do { const char* name = not unw_get_proc_name(&cursor, buffer, 100, &off) ? buffer : "?"; // Unmangle C++ names: auto realname = simgrid::xbt::demangle(name); #if defined(__x86_64__) unw_word_t rip = 0; unw_word_t rsp = 0; unw_get_reg(&cursor, UNW_X86_64_RIP, &rip); unw_get_reg(&cursor, UNW_X86_64_RSP, &rsp); fprintf(file, " %i: %s (RIP=0x%" PRIx64 " RSP=0x%" PRIx64 ")\n", nframe, realname.get(), (std::uint64_t)rip, (std::uint64_t)rsp); #else fprintf(file, " %i: %s\n", nframe, realname.get()); #endif ++nframe; } while(unw_step(&cursor)); } } } static void MC_dump_stacks(FILE* file) { int nstack = 0; for (auto const& stack : mc_model_checker->process().stack_areas()) { fprintf(file, "Stack %i:\n", nstack); nstack++; simgrid::mc::UnwindContext context; unw_context_t raw_context = (unw_context_t) mc_model_checker->process().read( simgrid::mc::remote((unw_context_t *)stack.context)); context.initialize(&mc_model_checker->process(), &raw_context); unw_cursor_t cursor = context.cursor(); simgrid::mc::dumpStack(file, cursor); } } #endif double MC_process_clock_get(smx_actor_t process) { if (simgrid::mc::processes_time.empty()) return 0; if (process != nullptr) return simgrid::mc::processes_time[process->pid]; return -1; } void MC_process_clock_add(smx_actor_t process, double amount) { simgrid::mc::processes_time[process->pid] += amount; } SimGrid-3.18/src/mc/mc_exit.hpp0000644000175000017500000000137213217757316016655 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_EXIT_HPP #define SIMGRID_MC_EXIT_HPP #include "xbt/base.h" #define SIMGRID_MC_EXIT_SUCCESS 0 #define SIMGRID_MC_EXIT_SAFETY 1 #define SIMGRID_MC_EXIT_LIVENESS 2 #define SIMGRID_MC_EXIT_DEADLOCK 3 #define SIMGRID_MC_EXIT_NON_TERMINATION 4 #define SIMGRID_MC_EXIT_NON_DETERMINISM 5 #define SIMGRID_MC_EXIT_PROGRAM_CRASH 6 #define SIMGRID_MC_EXIT_ERROR 63 namespace simgrid { namespace mc { XBT_PUBLIC_CLASS DeadlockError{}; XBT_PUBLIC_CLASS TerminationError{}; XBT_PUBLIC_CLASS LivenessError{}; } } #endif SimGrid-3.18/src/mc/PageStore.cpp0000644000175000017500000001756513217757316017124 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include // memcpy, memcmp #include #include #ifdef __FreeBSD__ # define MAP_POPULATE MAP_PREFAULT_READ #endif #include "xbt/base.h" #include "xbt/log.h" #include "xbt/sysdep.h" #include "src/internal_config.h" #include "src/mc/PageStore.hpp" #include "src/mc/mc_mmu.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_page_snapshot, mc, "Logging specific to mc_page_snapshot"); namespace simgrid { namespace mc { /** @brief Compute a hash for the given memory page * * The page is used before inserting the page in the page store * in order to find duplicate of this page in the page store. * * @param data Memory page * @return hash off the page */ static XBT_ALWAYS_INLINE PageStore::hash_type mc_hash_page(const void* data) { const std::uint64_t* values = (const uint64_t*) data; std::size_t n = xbt_pagesize / sizeof(uint64_t); // This djb2: std::uint64_t hash = 5381; for (std::size_t i = 0; i != n; ++i) hash = ((hash << 5) + hash) + values[i]; return hash; } // ***** snapshot_page_manager PageStore::PageStore(size_t size) : memory_(nullptr), capacity_(size), top_index_(0) { // Using mmap in order to be able to expand the region by relocating it somewhere else in the virtual memory space: void* memory = ::mmap(nullptr, size << xbt_pagebits, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (memory == MAP_FAILED) xbt_die("Could not mmap initial snapshot pages."); this->top_index_ = 0; this->memory_ = memory; this->page_counts_.resize(size); } PageStore::~PageStore() { ::munmap(this->memory_, this->capacity_ << xbt_pagebits); } void PageStore::resize(std::size_t size) { size_t old_bytesize = this->capacity_ << xbt_pagebits; size_t new_bytesize = size << xbt_pagebits; void *new_memory; // Expand the memory region by moving it into another // virtual memory address if necessary: #if HAVE_MREMAP new_memory = mremap(this->memory_, old_bytesize, new_bytesize, MREMAP_MAYMOVE); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); #else if (new_bytesize > old_bytesize) { // Grow: first try to add new space after current map new_memory = mmap((char *)this->memory_ + old_bytesize, new_bytesize-old_bytesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); // Check if expanding worked if (new_memory != (char *)this->memory_ + old_bytesize) { // New memory segment could not be put at the end of this->memory_, // so cancel this one and try to rellocate everything and copy data munmap(new_memory, new_bytesize-old_bytesize); new_memory = mmap(nullptr, new_bytesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); memcpy(new_memory, this->memory_, old_bytesize); munmap(this->memory_, old_bytesize); } } else { // We don't have functions to shrink a mapping, so leave memory as // it is for now new_memory = this->memory_; } #endif this->capacity_ = size; this->memory_ = new_memory; this->page_counts_.resize(size, 0); } /** Allocate a free page * * @return index of the free page */ std::size_t PageStore::alloc_page() { if (this->free_pages_.empty()) { // Expand the region: if (this->top_index_ == this->capacity_) // All the pages are allocated, we need add more pages: this->resize(2 * this->capacity_); // Use a page from the top: return this->top_index_++; } else { // Use a page from free_pages_ (inside of the region): size_t res = this->free_pages_[this->free_pages_.size() - 1]; this->free_pages_.pop_back(); return res; } } void PageStore::remove_page(std::size_t pageno) { this->free_pages_.push_back(pageno); const void* page = this->get_page(pageno); hash_type hash = mc_hash_page(page); this->hash_index_[hash].erase(pageno); } /** Store a page in memory */ std::size_t PageStore::store_page(void* page) { xbt_assert(top_index_ <= this->capacity_, "top_index is not consistent"); // First, we check if a page with the same content is already in the page store: // 1. compute the hash of the page // 2. find pages with the same hash using `hash_index_` // 3. find a page with the same content hash_type hash = mc_hash_page(page); // Try to find a duplicate in set of pages with the same hash: page_set_type& page_set = this->hash_index_[hash]; for (size_t const& pageno : page_set) { const void* snapshot_page = this->get_page(pageno); if (memcmp(page, snapshot_page, xbt_pagesize) == 0) { // If a page with the same content is already in the page store it's reused and its refcount is incremented. page_counts_[pageno]++; return pageno; } } // Otherwise, a new page is allocated in the page store and the content of the page is `memcpy()`-ed to this new page. std::size_t pageno = alloc_page(); xbt_assert(this->page_counts_[pageno]==0, "Allocated page is already used"); void* snapshot_page = (void*) this->get_page(pageno); memcpy(snapshot_page, page, xbt_pagesize); page_set.insert(pageno); page_counts_[pageno]++; return pageno; } } } #ifdef SIMGRID_TEST #include #include #include #include #include #include "src/mc/PageStore.hpp" static int value = 0; static void new_content(void* data, std::size_t size) { ::memset(data, ++value, size); } static void* getpage() { return mmap(nullptr, getpagesize(), PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); } XBT_TEST_SUITE("mc_page_store", "Page store"); XBT_TEST_UNIT("base", test_mc_page_store, "Test adding/removing pages in the store") { using simgrid::mc::PageStore; xbt_test_add("Init"); std::size_t pagesize = (size_t) getpagesize(); std::unique_ptr store = std::unique_ptr(new simgrid::mc::PageStore(500)); void* data = getpage(); xbt_test_assert(store->size()==0, "Bad size"); xbt_test_add("Store the page once"); new_content(data, pagesize); size_t pageno1 = store->store_page(data); xbt_test_assert(store->get_ref(pageno1)==1, "Bad refcount"); const void* copy = store->get_page(pageno1); xbt_test_assert(::memcmp(data, copy, pagesize)==0, "Page data should be the same"); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Store the same page again"); size_t pageno2 = store->store_page(data); xbt_test_assert(pageno1==pageno2, "Page should be the same"); xbt_test_assert(store->get_ref(pageno1)==2, "Bad refcount"); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Store a new page"); new_content(data, pagesize); size_t pageno3 = store->store_page(data); xbt_test_assert(pageno1 != pageno3, "New page should be different"); xbt_test_assert(store->size()==2, "Bad size"); xbt_test_add("Unref pages"); store->unref_page(pageno1); xbt_assert(store->get_ref(pageno1)==1, "Bad refcount"); xbt_assert(store->size()==2, "Bad size"); store->unref_page(pageno2); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Reallocate page"); new_content(data, pagesize); size_t pageno4 = store->store_page(data); xbt_test_assert(pageno1 == pageno4, "Page was not reused"); xbt_test_assert(store->get_ref(pageno4)==1, "Bad refcount"); xbt_test_assert(store->size()==2, "Bad size"); } #endif /* SIMGRID_TEST */ SimGrid-3.18/src/mc/Transition.hpp0000644000175000017500000000201013217757316017345 0ustar mquinsonmquinson/* Copyright (c) 2015-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_TRANSITION_HPP #define SIMGRID_MC_TRANSITION_HPP namespace simgrid { namespace mc { /** An element in the recorded path * * At each decision point, we need to record which process transition * is triggered and potentially which value is associated with this * transition. The value is used to find which communication is triggered * in things like waitany and for associating a given value of MC_random() * calls. */ struct Transition { int pid = 0; /* Which transition was executed for this simcall * * Some simcalls can lead to different transitions: * * * waitany/testany can trigger on different messages; * * * random can produce different values. */ int argument = 0; }; } } #endif SimGrid-3.18/src/mc/compare.cpp0000644000175000017500000016140213217757316016647 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file compare.cpp Memory snapshooting and comparison */ #include #include #include #include #include #include #include "xbt/dynar.h" #include "xbt/sysdep.h" #include #include #include #include "src/internal_config.h" #include "src/xbt/mmalloc/mmprivate.h" #if HAVE_SMPI #include "src/smpi/include/private.hpp" #endif #include "src/mc/Frame.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Type.hpp" #include "src/mc/Variable.hpp" #include "src/mc/mc_dwarf.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_snapshot.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_compare, xbt, "Logging specific to mc_compare in mc"); namespace simgrid { namespace mc { struct HeapLocation; typedef std::array HeapLocationPair; typedef std::set HeapLocationPairs; struct HeapArea; struct ProcessComparisonState; struct StateComparator; static int compare_heap_area( StateComparator& state, int process_index, const void *area1, const void* area2, Snapshot* snapshot1, Snapshot* snapshot2, HeapLocationPairs* previous, Type* type, int pointer_level); } } using simgrid::mc::remote; /*********************************** Heap comparison ***********************************/ /***************************************************************************************/ namespace simgrid { namespace mc { class HeapLocation { public: int block_ = 0; int fragment_ = 0; HeapLocation() = default; HeapLocation(int block, int fragment = 0) : block_(block), fragment_(fragment) {} bool operator==(HeapLocation const& that) const { return block_ == that.block_ && fragment_ == that.fragment_; } bool operator<(HeapLocation const& that) const { return std::make_pair(block_, fragment_) < std::make_pair(that.block_, that.fragment_); } }; static inline HeapLocationPair makeHeapLocationPair(int block1, int fragment1, int block2, int fragment2) { return simgrid::mc::HeapLocationPair{{ simgrid::mc::HeapLocation(block1, fragment1), simgrid::mc::HeapLocation(block2, fragment2) }}; } class HeapArea : public HeapLocation { public: bool valid_ = false; HeapArea() = default; explicit HeapArea(int block) : valid_(true) { block_ = block; } HeapArea(int block, int fragment) : valid_(true) { block_ = block; fragment_ = fragment; } }; class ProcessComparisonState { public: std::vector* to_ignore = nullptr; std::vector equals_to; std::vector types; std::size_t heapsize = 0; void initHeapInformation(xbt_mheap_t heap, std::vector* i); }; namespace { /** A hash which works with more stuff * * It can hash pairs: the standard hash currently doesn't include this. */ template class hash : public std::hash { }; template class hash> { public: std::size_t operator()(std::pairconst& x) const { hash h1; hash h2; return h1(x.first) ^ h2(x.second); } }; } class StateComparator { public: s_xbt_mheap_t std_heap_copy; std::size_t heaplimit; std::array processStates; std::unordered_set, hash>> compared_pointers; void clear() { compared_pointers.clear(); } int initHeapInformation( xbt_mheap_t heap1, xbt_mheap_t heap2, std::vector* i1, std::vector* i2); HeapArea& equals_to1_(std::size_t i, std::size_t j) { return processStates[0].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea& equals_to2_(std::size_t i, std::size_t j) { return processStates[1].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type*& types1_(std::size_t i, std::size_t j) { return processStates[0].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type*& types2_(std::size_t i, std::size_t j) { return processStates[1].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea const& equals_to1_(std::size_t i, std::size_t j) const { return processStates[0].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea const& equals_to2_(std::size_t i, std::size_t j) const { return processStates[1].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type* const& types1_(std::size_t i, std::size_t j) const { return processStates[0].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type* const& types2_(std::size_t i, std::size_t j) const { return processStates[1].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } /** Check whether two blocks are known to be matching * * @param b1 Block of state 1 * @param b2 Block of state 2 * @return if the blocks are known to be matching */ bool blocksEqual(int b1, int b2) const { return this->equals_to1_(b1, 0).block_ == b2 && this->equals_to2_(b2, 0).block_ == b1; } /** Check whether two fragments are known to be matching * * @param b1 Block of state 1 * @param f1 Fragment of state 1 * @param b2 Block of state 2 * @param f2 Fragment of state 2 * @return if the fragments are known to be matching */ int fragmentsEqual(int b1, int f1, int b2, int f2) const { return this->equals_to1_(b1, f1).block_ == b2 && this->equals_to1_(b1, f1).fragment_ == f2 && this->equals_to2_(b2, f2).block_ == b1 && this->equals_to2_(b2, f2).fragment_ == f1; } void match_equals(HeapLocationPairs* list); }; } } /************************************************************************************/ static ssize_t heap_comparison_ignore_size( std::vector* ignore_list, const void *address) { int start = 0; int end = ignore_list->size() - 1; while (start <= end) { unsigned int cursor = (start + end) / 2; simgrid::mc::IgnoredHeapRegion const& region = (*ignore_list)[cursor]; if (region.address == address) return region.size; if (region.address < address) start = cursor + 1; if (region.address > address) end = cursor - 1; } return -1; } static bool is_stack(const void *address) { for (auto const& stack : mc_model_checker->process().stack_areas()) if (address == stack.address) return true; return false; } // TODO, this should depend on the snapshot? static bool is_block_stack(int block) { for (auto const& stack : mc_model_checker->process().stack_areas()) if (block == stack.block) return true; return false; } namespace simgrid { namespace mc { void StateComparator::match_equals(HeapLocationPairs* list) { for (auto const& pair : *list) { if (pair[0].fragment_ != -1) { this->equals_to1_(pair[0].block_, pair[0].fragment_) = simgrid::mc::HeapArea(pair[1].block_, pair[1].fragment_); this->equals_to2_(pair[1].block_, pair[1].fragment_) = simgrid::mc::HeapArea(pair[0].block_, pair[0].fragment_); } else { this->equals_to1_(pair[0].block_, 0) = simgrid::mc::HeapArea(pair[1].block_, pair[1].fragment_); this->equals_to2_(pair[1].block_, 0) = simgrid::mc::HeapArea(pair[0].block_, pair[0].fragment_); } } } void ProcessComparisonState::initHeapInformation(xbt_mheap_t heap, std::vector* i) { auto heaplimit = heap->heaplimit; this->heapsize = heap->heapsize; this->to_ignore = i; this->equals_to.assign(heaplimit * MAX_FRAGMENT_PER_BLOCK, HeapArea()); this->types.assign(heaplimit * MAX_FRAGMENT_PER_BLOCK, nullptr); } int StateComparator::initHeapInformation(xbt_mheap_t heap1, xbt_mheap_t heap2, std::vector* i1, std::vector* i2) { if ((heap1->heaplimit != heap2->heaplimit) || (heap1->heapsize != heap2->heapsize)) return -1; this->heaplimit = heap1->heaplimit; this->std_heap_copy = *mc_model_checker->process().get_heap(); this->processStates[0].initHeapInformation(heap1, i1); this->processStates[1].initHeapInformation(heap2, i2); return 0; } // TODO, have a robust way to find it in O(1) static inline mc_mem_region_t MC_get_heap_region(simgrid::mc::Snapshot* snapshot) { for (auto const& region : snapshot->snapshot_regions) if (region->region_type() == simgrid::mc::RegionType::Heap) return region.get(); xbt_die("No heap region"); } static int mmalloc_compare_heap( simgrid::mc::StateComparator& state, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); /* Start comparison */ size_t i1; size_t i2; size_t j1; size_t j2; size_t k; void* addr_block1; void* addr_block2; void* addr_frag1; void* addr_frag2; int nb_diff1 = 0; int nb_diff2 = 0; int equal; /* Check busy blocks */ i1 = 1; malloc_info heapinfo_temp1; malloc_info heapinfo_temp2; malloc_info heapinfo_temp2b; mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); // This is the address of std_heap->heapinfo in the application process: void* heapinfo_address = &((xbt_mheap_t) process->heap_address)->heapinfo; // This is in snapshot do not use them directly: const malloc_info* heapinfos1 = snapshot1->read( RemotePtr((std::uint64_t)heapinfo_address), simgrid::mc::ProcessIndexMissing); const malloc_info* heapinfos2 = snapshot2->read( RemotePtr((std::uint64_t)heapinfo_address), simgrid::mc::ProcessIndexMissing); while (i1 < state.heaplimit) { const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read(heap_region1, &heapinfo_temp1, &heapinfos1[i1], sizeof(malloc_info)); const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read(heap_region2, &heapinfo_temp2, &heapinfos2[i1], sizeof(malloc_info)); if (heapinfo1->type == MMALLOC_TYPE_FREE || heapinfo1->type == MMALLOC_TYPE_HEAPINFO) { /* Free block */ i1 ++; continue; } if (heapinfo1->type < 0) { fprintf(stderr, "Unkown mmalloc block type.\n"); abort(); } addr_block1 = ((void*)(((ADDR2UINT(i1)) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase)); if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED) { /* Large block */ if (is_stack(addr_block1)) { for (k = 0; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i1, -1); for (k = 0; k < heapinfo2->busy_block.size; k++) state.equals_to2_(i1 + k, 0) = HeapArea(i1, -1); i1 += heapinfo1->busy_block.size; continue; } if (state.equals_to1_(i1, 0).valid_) { i1++; continue; } i2 = 1; equal = 0; /* Try first to associate to same block in the other heap */ if (heapinfo2->type == heapinfo1->type && state.equals_to2_(i1, 0).valid_ == 0) { addr_block2 = (ADDR2UINT(i1) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase; int res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_block1, addr_block2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { for (k = 1; k < heapinfo2->busy_block.size; k++) state.equals_to2_(i1 + k, 0) = HeapArea(i1, -1); for (k = 1; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i1, -1); equal = 1; i1 += heapinfo1->busy_block.size; } } while (i2 < state.heaplimit && not equal) { addr_block2 = (ADDR2UINT(i2) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase; if (i2 == i1) { i2++; continue; } const malloc_info* heapinfo2b = (const malloc_info*) MC_region_read(heap_region2, &heapinfo_temp2b, &heapinfos2[i2], sizeof(malloc_info)); if (heapinfo2b->type != MMALLOC_TYPE_UNFRAGMENTED) { i2++; continue; } if (state.equals_to2_(i2, 0).valid_) { i2++; continue; } int res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_block1, addr_block2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { for (k = 1; k < heapinfo2b->busy_block.size; k++) state.equals_to2_(i2 + k, 0) = HeapArea(i1, -1); for (k = 1; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i2, -1); equal = 1; i1 += heapinfo1->busy_block.size; } i2++; } if (not equal) { XBT_DEBUG("Block %zu not found (size_used = %zu, addr = %p)", i1, heapinfo1->busy_block.busy_size, addr_block1); i1 = state.heaplimit + 1; nb_diff1++; } } else { /* Fragmented block */ for (j1 = 0; j1 < (size_t) (BLOCKSIZE >> heapinfo1->type); j1++) { if (heapinfo1->busy_frag.frag_size[j1] == -1) /* Free fragment_ */ continue; if (state.equals_to1_(i1, j1).valid_) continue; addr_frag1 = (void*)((char*)addr_block1 + (j1 << heapinfo1->type)); i2 = 1; equal = 0; /* Try first to associate to same fragment_ in the other heap */ if (heapinfo2->type == heapinfo1->type && not state.equals_to2_(i1, j1).valid_) { addr_block2 = (ADDR2UINT(i1) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; addr_frag2 = (void *) ((char *) addr_block2 + (j1 << heapinfo2->type)); int res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_frag1, addr_frag2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) equal = 1; } while (i2 < state.heaplimit && not equal) { const malloc_info* heapinfo2b = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2b, &heapinfos2[i2], sizeof(malloc_info)); if (heapinfo2b->type == MMALLOC_TYPE_FREE || heapinfo2b->type == MMALLOC_TYPE_HEAPINFO) { i2 ++; continue; } // We currently do not match fragments with unfragmented blocks (maybe we should). if (heapinfo2b->type == MMALLOC_TYPE_UNFRAGMENTED) { i2++; continue; } if (heapinfo2b->type < 0) { fprintf(stderr, "Unknown mmalloc block type.\n"); abort(); } for (j2 = 0; j2 < (size_t) (BLOCKSIZE >> heapinfo2b->type); j2++) { if (i2 == i1 && j2 == j1) continue; if (state.equals_to2_(i2, j2).valid_) continue; addr_block2 = (ADDR2UINT(i2) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase; addr_frag2 = (void*)((char*)addr_block2 + (j2 << heapinfo2b->type)); int res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_frag1, addr_frag2, snapshot2, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { equal = 1; break; } } i2++; } if (not equal) { XBT_DEBUG("Block %zu, fragment_ %zu not found (size_used = %zd, address = %p)\n", i1, j1, heapinfo1->busy_frag.frag_size[j1], addr_frag1); i1 = state.heaplimit + 1; nb_diff1++; break; } } i1++; } } /* All blocks/fragments are equal to another block/fragment_ ? */ size_t i = 1; size_t j = 0; for(i = 1; i < state.heaplimit; i++) { const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read( heap_region1, &heapinfo_temp1, &heapinfos1[i], sizeof(malloc_info)); if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED && i1 == state.heaplimit && heapinfo1->busy_block.busy_size > 0 && not state.equals_to1_(i, 0).valid_) { XBT_DEBUG("Block %zu not found (size used = %zu)", i, heapinfo1->busy_block.busy_size); nb_diff1++; } if (heapinfo1->type <= 0) continue; for (j = 0; j < (size_t) (BLOCKSIZE >> heapinfo1->type); j++) if (i1 == state.heaplimit && heapinfo1->busy_frag.frag_size[j] > 0 && not state.equals_to1_(i, j).valid_) { XBT_DEBUG("Block %zu, Fragment %zu not found (size used = %zd)", i, j, heapinfo1->busy_frag.frag_size[j]); nb_diff1++; } } if (i1 == state.heaplimit) XBT_DEBUG("Number of blocks/fragments not found in heap1: %d", nb_diff1); for (i=1; i < state.heaplimit; i++) { const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2, &heapinfos2[i], sizeof(malloc_info)); if (heapinfo2->type == MMALLOC_TYPE_UNFRAGMENTED && i1 == state.heaplimit && heapinfo2->busy_block.busy_size > 0 && not state.equals_to2_(i, 0).valid_) { XBT_DEBUG("Block %zu not found (size used = %zu)", i, heapinfo2->busy_block.busy_size); nb_diff2++; } if (heapinfo2->type <= 0) continue; for (j = 0; j < (size_t) (BLOCKSIZE >> heapinfo2->type); j++) if (i1 == state.heaplimit && heapinfo2->busy_frag.frag_size[j] > 0 && not state.equals_to2_(i, j).valid_) { XBT_DEBUG("Block %zu, Fragment %zu not found (size used = %zd)", i, j, heapinfo2->busy_frag.frag_size[j]); nb_diff2++; } } if (i1 == state.heaplimit) XBT_DEBUG("Number of blocks/fragments not found in heap2: %d", nb_diff2); return nb_diff1 > 0 || nb_diff2 > 0; } /** * * @param state * @param real_area1 Process address for state 1 * @param real_area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous * @param size * @param check_ignore */ static int compare_heap_area_without_type( simgrid::mc::StateComparator& state, int process_index, const void *real_area1, const void *real_area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, int size, int check_ignore) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); for (int i = 0; i < size; ) { if (check_ignore > 0) { ssize_t ignore1 = heap_comparison_ignore_size( state.processStates[0].to_ignore, (char *) real_area1 + i); if (ignore1 != -1) { ssize_t ignore2 = heap_comparison_ignore_size( state.processStates[1].to_ignore, (char *) real_area2 + i); if (ignore2 == ignore1) { if (ignore1 == 0) { check_ignore--; return 0; } else { i = i + ignore2; check_ignore--; continue; } } } } if (MC_snapshot_region_memcmp(((char *) real_area1) + i, heap_region1, ((char *) real_area2) + i, heap_region2, 1) != 0) { int pointer_align = (i / sizeof(void *)) * sizeof(void *); const void* addr_pointed1 = snapshot1->read( remote((void**)((char *) real_area1 + pointer_align)), process_index); const void* addr_pointed2 = snapshot2->read( remote((void**)((char *) real_area2 + pointer_align)), process_index); if (process->in_maestro_stack(remote(addr_pointed1)) && process->in_maestro_stack(remote(addr_pointed2))) { i = pointer_align + sizeof(void *); continue; } if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) { // Both addreses are in the heap: int res_compare = compare_heap_area(state ,process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, nullptr, 0); if (res_compare == 1) return res_compare; i = pointer_align + sizeof(void *); continue; } return 1; } i++; } return 0; } /** * * @param state * @param real_area1 Process address for state 1 * @param real_area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous * @param type * @param area_size either a byte_size or an elements_count (?) * @param check_ignore * @param pointer_level * @return 0 (same), 1 (different), -1 (unknown) */ static int compare_heap_area_with_type( simgrid::mc::StateComparator& state, int process_index, const void *real_area1, const void *real_area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, simgrid::mc::Type* type, int area_size, int check_ignore, int pointer_level) { do { // HACK: This should not happen but in pratice, there are some // DW_TAG_typedef without an associated DW_AT_type: //<1><538832>: Abbrev Number: 111 (DW_TAG_typedef) // <538833> DW_AT_name : (indirect string, offset: 0x2292f3): gregset_t // <538837> DW_AT_decl_file : 98 // <538838> DW_AT_decl_line : 37 if (type == nullptr) return 0; if (is_stack(real_area1) && is_stack(real_area2)) return 0; if (check_ignore > 0) { ssize_t ignore1 = heap_comparison_ignore_size(state.processStates[0].to_ignore, real_area1); if (ignore1 > 0 && heap_comparison_ignore_size(state.processStates[1].to_ignore, real_area2) == ignore1) return 0; } simgrid::mc::Type* subtype; simgrid::mc::Type* subsubtype; int res; int elm_size; const void* addr_pointed1; const void* addr_pointed2; mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); switch (type->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: if (not type->name.empty() && type->name == "char") { /* String, hence random (arbitrary ?) size */ if (real_area1 == real_area2) return -1; else return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, area_size) != 0; } else { if (area_size != -1 && type->byte_size != area_size) return -1; else return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, type->byte_size) != 0; } break; case DW_TAG_enumeration_type: if (area_size != -1 && type->byte_size != area_size) return -1; return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, type->byte_size) != 0; case DW_TAG_typedef: case DW_TAG_const_type: case DW_TAG_volatile_type: // Poor man's TCO: type = type->subtype; continue; // restart case DW_TAG_array_type: subtype = type->subtype; switch (subtype->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_structure_type: case DW_TAG_class_type: case DW_TAG_union_type: if (subtype->full_type) subtype = subtype->full_type; elm_size = subtype->byte_size; break; // TODO, just remove the type indirection? case DW_TAG_const_type: case DW_TAG_typedef: case DW_TAG_volatile_type: subsubtype = subtype->subtype; if (subsubtype->full_type) subsubtype = subsubtype->full_type; elm_size = subsubtype->byte_size; break; default: return 0; break; } for (int i = 0; i < type->element_count; i++) { // TODO, add support for variable stride (DW_AT_byte_stride) res = compare_heap_area_with_type(state, process_index, (char*)real_area1 + (i * elm_size), (char*)real_area2 + (i * elm_size), snapshot1, snapshot2, previous, type->subtype, subtype->byte_size, check_ignore, pointer_level); if (res == 1) return res; } return 0; case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_pointer_type: if (type->subtype && type->subtype->type == DW_TAG_subroutine_type) { addr_pointed1 = snapshot1->read(remote((void**)real_area1), process_index); addr_pointed2 = snapshot2->read(remote((void**)real_area2), process_index); return (addr_pointed1 != addr_pointed2); } pointer_level++; if (pointer_level <= 1) { addr_pointed1 = snapshot1->read(remote((void**)real_area1), process_index); addr_pointed2 = snapshot2->read(remote((void**)real_area2), process_index); if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) return compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, type->subtype, pointer_level); else return (addr_pointed1 != addr_pointed2); } for (size_t i = 0; i < (area_size / sizeof(void*)); i++) { addr_pointed1 = snapshot1->read(remote((void**)((char*)real_area1 + i * sizeof(void*))), process_index); addr_pointed2 = snapshot2->read(remote((void**)((char*)real_area2 + i * sizeof(void*))), process_index); if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) res = compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, type->subtype, pointer_level); else res = (addr_pointed1 != addr_pointed2); if (res == 1) return res; } return 0; case DW_TAG_structure_type: case DW_TAG_class_type: if (type->full_type) type = type->full_type; if (area_size != -1 && type->byte_size != area_size) { if (area_size <= type->byte_size || area_size % type->byte_size != 0) return -1; for (size_t i = 0; i < (size_t)(area_size / type->byte_size); i++) { int res = compare_heap_area_with_type(state, process_index, (char*)real_area1 + i * type->byte_size, (char*)real_area2 + i * type->byte_size, snapshot1, snapshot2, previous, type, -1, check_ignore, 0); if (res == 1) return res; } } else { for (simgrid::mc::Member& member : type->members) { // TODO, optimize this? (for the offset case) void* real_member1 = simgrid::dwarf::resolve_member(real_area1, type, &member, (simgrid::mc::AddressSpace*)snapshot1, process_index); void* real_member2 = simgrid::dwarf::resolve_member(real_area2, type, &member, (simgrid::mc::AddressSpace*)snapshot2, process_index); int res = compare_heap_area_with_type(state, process_index, real_member1, real_member2, snapshot1, snapshot2, previous, member.type, -1, check_ignore, 0); if (res == 1) return res; } } return 0; case DW_TAG_union_type: return compare_heap_area_without_type(state, process_index, real_area1, real_area2, snapshot1, snapshot2, previous, type->byte_size, check_ignore); default: return 0; } xbt_die("Unreachable"); } while (true); } /** Infer the type of a part of the block from the type of the block * * TODO, handle DW_TAG_array_type as well as arrays of the object ((*p)[5], p[5]) * * TODO, handle subfields ((*p).bar.foo, (*p)[5].bar…) * * @param type DWARF type ID of the root address * @param area_size * @return DWARF type ID for given offset */ static simgrid::mc::Type* get_offset_type(void *real_base_address, simgrid::mc::Type* type, int offset, int area_size, simgrid::mc::Snapshot* snapshot, int process_index) { // Beginning of the block, the infered variable type if the type of the block: if (offset == 0) return type; switch (type->type) { case DW_TAG_structure_type: case DW_TAG_class_type: if (type->full_type) type = type->full_type; if (area_size != -1 && type->byte_size != area_size) { if (area_size > type->byte_size && area_size % type->byte_size == 0) return type; else return nullptr; } for (simgrid::mc::Member& member : type->members) { if (member.has_offset_location()) { // We have the offset, use it directly (shortcut): if (member.offset() == offset) return member.type; } else { void* real_member = simgrid::dwarf::resolve_member(real_base_address, type, &member, snapshot, process_index); if ((char*)real_member - (char*)real_base_address == offset) return member.type; } } return nullptr; default: /* FIXME: other cases ? */ return nullptr; } } /** * * @param area1 Process address for state 1 * @param area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous Pairs of blocks already compared on the current path (or nullptr) * @param type_id Type of variable * @param pointer_level * @return 0 (same), 1 (different), -1 */ static int compare_heap_area(simgrid::mc::StateComparator& state, int process_index, const void *area1, const void *area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, simgrid::mc::Type* type, int pointer_level) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); ssize_t block1; ssize_t block2; ssize_t size; int check_ignore = 0; int type_size = -1; int offset1 = 0; int offset2 = 0; int new_size1 = -1; int new_size2 = -1; simgrid::mc::Type* new_type1 = nullptr; simgrid::mc::Type* new_type2 = nullptr; bool match_pairs = false; // This is the address of std_heap->heapinfo in the application process: void* heapinfo_address = &((xbt_mheap_t) process->heap_address)->heapinfo; const malloc_info* heapinfos1 = snapshot1->read(remote((const malloc_info**)heapinfo_address), process_index); const malloc_info* heapinfos2 = snapshot2->read(remote((const malloc_info**)heapinfo_address), process_index); malloc_info heapinfo_temp1; malloc_info heapinfo_temp2; simgrid::mc::HeapLocationPairs current; if (previous == nullptr) { previous = ¤t; match_pairs = true; } // Get block number: block1 = ((char*)area1 - (char*)state.std_heap_copy.heapbase) / BLOCKSIZE + 1; block2 = ((char*)area2 - (char*)state.std_heap_copy.heapbase) / BLOCKSIZE + 1; // If either block is a stack block: if (is_block_stack((int) block1) && is_block_stack((int) block2)) { previous->insert(simgrid::mc::makeHeapLocationPair(block1, -1, block2, -1)); if (match_pairs) state.match_equals(previous); return 0; } // If either block is not in the expected area of memory: if (((char*)area1 < (char*)state.std_heap_copy.heapbase) || (block1 > (ssize_t)state.processStates[0].heapsize) || (block1 < 1) || ((char*)area2 < (char*)state.std_heap_copy.heapbase) || (block2 > (ssize_t)state.processStates[1].heapsize) || (block2 < 1)) { return 1; } // Process address of the block: void* real_addr_block1 = (ADDR2UINT(block1) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase; void* real_addr_block2 = (ADDR2UINT(block2) - 1) * BLOCKSIZE + (char*)state.std_heap_copy.heapbase; if (type) { if (type->full_type) type = type->full_type; // This assume that for "boring" types (volatile ...) byte_size is absent: while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; // Find type_size: if (type->type == DW_TAG_pointer_type || (type->type == DW_TAG_base_type && not type->name.empty() && type->name == "char")) type_size = -1; else type_size = type->byte_size; } mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read( heap_region1, &heapinfo_temp1, &heapinfos1[block1], sizeof(malloc_info)); const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2, &heapinfos2[block2], sizeof(malloc_info)); if ((heapinfo1->type == MMALLOC_TYPE_FREE || heapinfo1->type==MMALLOC_TYPE_HEAPINFO) && (heapinfo2->type == MMALLOC_TYPE_FREE || heapinfo2->type ==MMALLOC_TYPE_HEAPINFO)) { /* Free block */ if (match_pairs) state.match_equals(previous); return 0; } if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED && heapinfo2->type == MMALLOC_TYPE_UNFRAGMENTED) { /* Complete block */ // TODO, lookup variable type from block type as done for fragmented blocks if (state.equals_to1_(block1, 0).valid_ && state.equals_to2_(block2, 0).valid_ && state.blocksEqual(block1, block2)) { if (match_pairs) state.match_equals(previous); return 0; } if (type_size != -1 && type_size != (ssize_t)heapinfo1->busy_block.busy_size && type_size != (ssize_t)heapinfo2->busy_block.busy_size && (type->name.empty() || type->name == "struct s_smx_context")) { if (match_pairs) state.match_equals(previous); return -1; } if (heapinfo1->busy_block.size != heapinfo2->busy_block.size) return 1; if (heapinfo1->busy_block.busy_size != heapinfo2->busy_block.busy_size) return 1; if (not previous->insert(simgrid::mc::makeHeapLocationPair(block1, -1, block2, -1)).second) { if (match_pairs) state.match_equals(previous); return 0; } size = heapinfo1->busy_block.busy_size; // Remember (basic) type inference. // The current data structure only allows us to do this for the whole block. if (type != nullptr && area1 == real_addr_block1) state.types1_(block1, 0) = type; if (type != nullptr && area2 == real_addr_block2) state.types2_(block2, 0) = type; if (size <= 0) { if (match_pairs) state.match_equals(previous); return 0; } if (heapinfo1->busy_block.ignore > 0 && heapinfo2->busy_block.ignore == heapinfo1->busy_block.ignore) check_ignore = heapinfo1->busy_block.ignore; } else if ((heapinfo1->type > 0) && (heapinfo2->type > 0)) { /* Fragmented block */ // Fragment number: ssize_t frag1 = ((uintptr_t)(ADDR2UINT(area1) % (BLOCKSIZE))) >> heapinfo1->type; ssize_t frag2 = ((uintptr_t)(ADDR2UINT(area2) % (BLOCKSIZE))) >> heapinfo2->type; // Process address of the fragment_: void* real_addr_frag1 = (void*)((char*)real_addr_block1 + (frag1 << heapinfo1->type)); void* real_addr_frag2 = (void*)((char*)real_addr_block2 + (frag2 << heapinfo2->type)); // Check the size of the fragments against the size of the type: if (type_size != -1) { if (heapinfo1->busy_frag.frag_size[frag1] == -1 || heapinfo2->busy_frag.frag_size[frag2] == -1) { if (match_pairs) state.match_equals(previous); return -1; } // ? if (type_size != heapinfo1->busy_frag.frag_size[frag1] || type_size != heapinfo2->busy_frag.frag_size[frag2]) { if (match_pairs) state.match_equals(previous); return -1; } } // Check if the blocks are already matched together: if (state.equals_to1_(block1, frag1).valid_ && state.equals_to2_(block2, frag2).valid_ && offset1 == offset2 && state.fragmentsEqual(block1, frag1, block2, frag2)) { if (match_pairs) state.match_equals(previous); return 0; } // Compare the size of both fragments: if (heapinfo1->busy_frag.frag_size[frag1] != heapinfo2->busy_frag.frag_size[frag2]) { if (type_size == -1) { if (match_pairs) state.match_equals(previous); return -1; } else return 1; } // Size of the fragment_: size = heapinfo1->busy_frag.frag_size[frag1]; // Remember (basic) type inference. // The current data structure only allows us to do this for the whole fragment_. if (type != nullptr && area1 == real_addr_frag1) state.types1_(block1, frag1) = type; if (type != nullptr && area2 == real_addr_frag2) state.types2_(block2, frag2) = type; // The type of the variable is already known: if (type) { new_type1 = new_type2 = type; } // Type inference from the block type. else if (state.types1_(block1, frag1) != nullptr || state.types2_(block2, frag2) != nullptr) { offset1 = (char*)area1 - (char*)real_addr_frag1; offset2 = (char*)area2 - (char*)real_addr_frag2; if (state.types1_(block1, frag1) != nullptr && state.types2_(block2, frag2) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types1_(block1, frag1), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types2_(block2, frag2), offset1, size, snapshot2, process_index); } else if (state.types1_(block1, frag1) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types1_(block1, frag1), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types1_(block1, frag1), offset2, size, snapshot2, process_index); } else if (state.types2_(block2, frag2) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types2_(block2, frag2), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types2_(block2, frag2), offset2, size, snapshot2, process_index); } else { if (match_pairs) state.match_equals(previous); return -1; } if (new_type1 != nullptr && new_type2 != nullptr && new_type1 != new_type2) { type = new_type1; while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; new_size1 = type->byte_size; type = new_type2; while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; new_size2 = type->byte_size; } else { if (match_pairs) state.match_equals(previous); return -1; } } if (new_size1 > 0 && new_size1 == new_size2) { type = new_type1; size = new_size1; } if (offset1 == 0 && offset2 == 0 && not previous->insert(simgrid::mc::makeHeapLocationPair(block1, frag1, block2, frag2)).second) { if (match_pairs) state.match_equals(previous); return 0; } if (size <= 0) { if (match_pairs) state.match_equals(previous); return 0; } if ((heapinfo1->busy_frag.ignore[frag1] > 0) && (heapinfo2->busy_frag.ignore[frag2] == heapinfo1->busy_frag.ignore[frag1])) check_ignore = heapinfo1->busy_frag.ignore[frag1]; } else return 1; /* Start comparison */ int res_compare; if (type) res_compare = compare_heap_area_with_type(state, process_index, area1, area2, snapshot1, snapshot2, previous, type, size, check_ignore, pointer_level); else res_compare = compare_heap_area_without_type(state, process_index, area1, area2, snapshot1, snapshot2, previous, size, check_ignore); if (res_compare == 1) return res_compare; if (match_pairs) state.match_equals(previous); return 0; } } } /************************** Snapshot comparison *******************************/ /******************************************************************************/ static int compare_areas_with_type(simgrid::mc::StateComparator& state, int process_index, void* real_area1, simgrid::mc::Snapshot* snapshot1, mc_mem_region_t region1, void* real_area2, simgrid::mc::Snapshot* snapshot2, mc_mem_region_t region2, simgrid::mc::Type* type, int pointer_level) { simgrid::mc::RemoteClient* process = &mc_model_checker->process(); simgrid::mc::Type* subtype; simgrid::mc::Type* subsubtype; int elm_size; int i; int res; do { switch (type->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_union_type: return MC_snapshot_region_memcmp(real_area1, region1, real_area2, region2, type->byte_size) != 0; case DW_TAG_typedef: case DW_TAG_volatile_type: case DW_TAG_const_type: // Poor man's TCO: type = type->subtype; continue; // restart case DW_TAG_array_type: subtype = type->subtype; switch (subtype->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_structure_type: case DW_TAG_class_type: case DW_TAG_union_type: if (subtype->full_type) subtype = subtype->full_type; elm_size = subtype->byte_size; break; case DW_TAG_const_type: case DW_TAG_typedef: case DW_TAG_volatile_type: subsubtype = subtype->subtype; if (subsubtype->full_type) subsubtype = subsubtype->full_type; elm_size = subsubtype->byte_size; break; default: return 0; break; } for (i = 0; i < type->element_count; i++) { size_t off = i * elm_size; res = compare_areas_with_type(state, process_index, (char*)real_area1 + off, snapshot1, region1, (char*)real_area2 + off, snapshot2, region2, type->subtype, pointer_level); if (res == 1) return res; } break; case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: { void* addr_pointed1 = MC_region_read_pointer(region1, real_area1); void* addr_pointed2 = MC_region_read_pointer(region2, real_area2); if (type->subtype && type->subtype->type == DW_TAG_subroutine_type) return (addr_pointed1 != addr_pointed2); if (addr_pointed1 == nullptr && addr_pointed2 == nullptr) return 0; if (addr_pointed1 == nullptr || addr_pointed2 == nullptr) return 1; if (not state.compared_pointers.insert(std::make_pair(addr_pointed1, addr_pointed2)).second) return 0; pointer_level++; // Some cases are not handled here: // * the pointers lead to different areas (one to the heap, the other to the RW segment ...) // * a pointer leads to the read-only segment of the current object // * a pointer lead to a different ELF object if (addr_pointed1 > process->heap_address && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1)) { if (not(addr_pointed2 > process->heap_address && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2))) return 1; // The pointers are both in the heap: return simgrid::mc::compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, nullptr, type->subtype, pointer_level); } else if (region1->contain(simgrid::mc::remote(addr_pointed1))) { // The pointers are both in the current object R/W segment: if (not region2->contain(simgrid::mc::remote(addr_pointed2))) return 1; if (not type->type_id) return (addr_pointed1 != addr_pointed2); else return compare_areas_with_type(state, process_index, addr_pointed1, snapshot1, region1, addr_pointed2, snapshot2, region2, type->subtype, pointer_level); } else { // TODO, We do not handle very well the case where // it belongs to a different (non-heap) region from the current one. return (addr_pointed1 != addr_pointed2); } break; } case DW_TAG_structure_type: case DW_TAG_class_type: for (simgrid::mc::Member& member : type->members) { void* member1 = simgrid::dwarf::resolve_member(real_area1, type, &member, snapshot1, process_index); void* member2 = simgrid::dwarf::resolve_member(real_area2, type, &member, snapshot2, process_index); mc_mem_region_t subregion1 = mc_get_region_hinted(member1, snapshot1, process_index, region1); mc_mem_region_t subregion2 = mc_get_region_hinted(member2, snapshot2, process_index, region2); res = compare_areas_with_type(state, process_index, member1, snapshot1, subregion1, member2, snapshot2, subregion2, member.type, pointer_level); if (res == 1) return res; } break; case DW_TAG_subroutine_type: return -1; break; default: XBT_VERB("Unknown case: %d", type->type); break; } return 0; } while (true); } static int compare_global_variables( simgrid::mc::StateComparator& state, simgrid::mc::ObjectInformation* object_info, int process_index, mc_mem_region_t r1, mc_mem_region_t r2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2) { xbt_assert(r1 && r2, "Missing region."); #if HAVE_SMPI if (r1->storage_type() == simgrid::mc::StorageType::Privatized) { xbt_assert(process_index >= 0); if (r2->storage_type() != simgrid::mc::StorageType::Privatized) return 1; size_t process_count = MC_smpi_process_count(); xbt_assert(process_count == r1->privatized_data().size() && process_count == r2->privatized_data().size()); // Compare the global variables separately for each simulates process: for (size_t process_index = 0; process_index < process_count; process_index++) { if (compare_global_variables(state, object_info, process_index, &r1->privatized_data()[process_index], &r2->privatized_data()[process_index], snapshot1, snapshot2)) return 1; } return 0; } #else xbt_assert(r1->storage_type() != simgrid::mc::StorageType::Privatized); #endif xbt_assert(r2->storage_type() != simgrid::mc::StorageType::Privatized); std::vector& variables = object_info->global_variables; for (simgrid::mc::Variable const& current_var : variables) { // If the variable is not in this object, skip it: // We do not expect to find a pointer to something which is not reachable // by the global variables. if ((char *) current_var.address < (char *) object_info->start_rw || (char *) current_var.address > (char *) object_info->end_rw) continue; simgrid::mc::Type* bvariable_type = current_var.type; int res = compare_areas_with_type(state, process_index, (char *) current_var.address, snapshot1, r1, (char *) current_var.address, snapshot2, r2, bvariable_type, 0); if (res == 1) { XBT_VERB("Global variable %s (%p) is different between snapshots", current_var.name.c_str(), (char *) current_var.address); return 1; } } return 0; } static int compare_local_variables(simgrid::mc::StateComparator& state, int process_index, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, mc_snapshot_stack_t stack1, mc_snapshot_stack_t stack2) { if (stack1->local_variables.size() != stack2->local_variables.size()) { XBT_VERB("Different number of local variables"); return 1; } unsigned int cursor = 0; local_variable_t current_var1; local_variable_t current_var2; while (cursor < stack1->local_variables.size()) { current_var1 = &stack1->local_variables[cursor]; current_var2 = &stack1->local_variables[cursor]; if (current_var1->name != current_var2->name || current_var1->subprogram != current_var2->subprogram || current_var1->ip != current_var2->ip) { // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram XBT_VERB ("Different name of variable (%s - %s) " "or frame (%s - %s) or ip (%lu - %lu)", current_var1->name.c_str(), current_var2->name.c_str(), current_var1->subprogram->name.c_str(), current_var2->subprogram->name.c_str(), current_var1->ip, current_var2->ip); return 1; } // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram simgrid::mc::Type* subtype = current_var1->type; int res = compare_areas_with_type( state, process_index, current_var1->address, snapshot1, mc_get_snapshot_region(current_var1->address, snapshot1, process_index), current_var2->address, snapshot2, mc_get_snapshot_region(current_var2->address, snapshot2, process_index), subtype, 0); if (res == 1) { // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram XBT_VERB("Local variable %s (%p - %p) in frame %s " "is different between snapshots", current_var1->name.c_str(), current_var1->address, current_var2->address, current_var1->subprogram->name.c_str()); return res; } cursor++; } return 0; } namespace simgrid { namespace mc { static std::unique_ptr state_comparator; int snapshot_compare(int num1, simgrid::mc::Snapshot* s1, int num2, simgrid::mc::Snapshot* s2) { // TODO, make this a field of ModelChecker or something similar if (state_comparator == nullptr) state_comparator = std::unique_ptr(new StateComparator()); else state_comparator->clear(); simgrid::mc::RemoteClient* process = &mc_model_checker->process(); int errors = 0; int hash_result = 0; if (_sg_mc_hash) { hash_result = (s1->hash != s2->hash); if (hash_result) { XBT_VERB("(%d - %d) Different hash: 0x%" PRIx64 "--0x%" PRIx64, num1, num2, s1->hash, s2->hash); #ifndef MC_DEBUG return 1; #endif } else XBT_VERB("(%d - %d) Same hash: 0x%" PRIx64, num1, num2, s1->hash); } /* Compare enabled processes */ if (s1->enabled_processes != s2->enabled_processes) { XBT_VERB("(%d - %d) Different amount of enabled processes", num1, num2); return 1; } /* Compare size of stacks */ int is_diff = 0; for (unsigned long i = 0; i < s1->stacks.size(); i++) { size_t size_used1 = s1->stack_sizes[i]; size_t size_used2 = s2->stack_sizes[i]; if (size_used1 != size_used2) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different size used in stacks: %zu - %zu", num1, num2, size_used1, size_used2); errors++; is_diff = 1; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different size used in stacks: %zu - %zu", num1, num2, size_used1, size_used2); #endif return 1; #endif } } if (is_diff) // do not proceed if there is any stacks that don't match return 1; /* Init heap information used in heap comparison algorithm */ xbt_mheap_t heap1 = (xbt_mheap_t)s1->read_bytes( alloca(sizeof(struct mdesc)), sizeof(struct mdesc), remote(process->heap_address), simgrid::mc::ProcessIndexMissing, simgrid::mc::ReadOptions::lazy()); xbt_mheap_t heap2 = (xbt_mheap_t)s2->read_bytes( alloca(sizeof(struct mdesc)), sizeof(struct mdesc), remote(process->heap_address), simgrid::mc::ProcessIndexMissing, simgrid::mc::ReadOptions::lazy()); int res_init = state_comparator->initHeapInformation(heap1, heap2, &s1->to_ignore, &s2->to_ignore); if (res_init == -1) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different heap information", num1, num2); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different heap information", num1, num2); #endif return 1; #endif } /* Stacks comparison */ int diff_local = 0; for (unsigned int cursor = 0; cursor < s1->stacks.size(); cursor++) { mc_snapshot_stack_t stack1 = &s1->stacks[cursor]; mc_snapshot_stack_t stack2 = &s2->stacks[cursor]; if (stack1->process_index != stack2->process_index) { diff_local = 1; XBT_DEBUG("(%d - %d) Stacks with different process index (%i vs %i)", num1, num2, stack1->process_index, stack2->process_index); } else diff_local = compare_local_variables(*state_comparator, stack1->process_index, s1, s2, stack1, stack2); if (diff_local > 0) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different local variables between stacks %d", num1, num2, cursor + 1); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different local variables between stacks %u", num1, num2, cursor + 1); #endif return 1; #endif } } size_t regions_count = s1->snapshot_regions.size(); // TODO, raise a difference instead? xbt_assert(regions_count == s2->snapshot_regions.size()); for (size_t k = 0; k != regions_count; ++k) { mc_mem_region_t region1 = s1->snapshot_regions[k].get(); mc_mem_region_t region2 = s2->snapshot_regions[k].get(); // Preconditions: if (region1->region_type() != simgrid::mc::RegionType::Data) continue; xbt_assert(region1->region_type() == region2->region_type()); xbt_assert(region1->object_info() == region2->object_info()); xbt_assert(region1->object_info()); std::string const& name = region1->object_info()->file_name; /* Compare global variables */ if (compare_global_variables(*state_comparator, region1->object_info(), simgrid::mc::ProcessIndexDisabled, region1, region2, s1, s2)) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different global variables in %s", num1, num2, name.c_str()); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different global variables in %s", num1, num2, name.c_str()); #endif return 1; #endif } } /* Compare heap */ if (simgrid::mc::mmalloc_compare_heap(*state_comparator, s1, s2) > 0) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different heap (mmalloc_compare)", num1, num2); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different heap (mmalloc_compare)", num1, num2); #endif return 1; #endif } #ifdef MC_VERBOSE if (errors || hash_result) XBT_VERB("(%d - %d) Difference found", num1, num2); else XBT_VERB("(%d - %d) No difference found", num1, num2); #endif #if defined(MC_DEBUG) && defined(MC_VERBOSE) if (_sg_mc_hash) { // * false positive SHOULD be avoided. // * There MUST not be any false negative. XBT_VERB("(%d - %d) State equality hash test is %s %s", num1, num2, (hash_result != 0) == (errors != 0) ? "true" : "false", not hash_result ? "positive" : "negative"); } #endif return errors > 0 || hash_result; } } } SimGrid-3.18/src/mc/ObjectInformation.hpp0000644000175000017500000001251313217757316020640 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_OBJECT_INFORMATION_HPP #define SIMGRID_MC_OBJECT_INFORMATION_HPP #include #include #include #include #include "xbt/base.h" #include "src/xbt/memory_map.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/Type.hpp" #include "src/mc/Frame.hpp" #include "src/smpi/include/private.hpp" namespace simgrid { namespace mc { /** An entry in the functions index * * See the code of ObjectInformation::find_function. */ struct FunctionIndexEntry { void* low_pc; simgrid::mc::Frame* function; }; /** Information about an ELF module (executable or shared object) * * This contains all the information we need about an executable or * shared-object in the model-checked process: * * - where it is located in the virtual address space; * * - where are located its different memory mappings in the the * virtual address space; * * - all the debugging (DWARF) information * - types, * - location of the functions and their local variables, * - global variables, * * - etc. */ class ObjectInformation { public: ObjectInformation() = default; // Not copyable: ObjectInformation(ObjectInformation const&) = delete; ObjectInformation& operator=(ObjectInformation const&) = delete; // Flag: static const int Executable = 1; /** Bitfield of flags */ int flags = 0; std::string file_name; const void* start = nullptr; const void *end = nullptr; // Location of its text segment: char *start_exec = nullptr; char *end_exec = nullptr; // Location of the read-only part of its data segment: char *start_rw = nullptr; char *end_rw = nullptr; // Location of the read/write part of its data segment: char *start_ro = nullptr; char *end_ro = nullptr; /** All of its subprograms indexed by their address */ std::unordered_map subprograms; /** Index of functions by instruction address * * We need to efficiently find the function from any given instruction * address inside its range. This index is sorted by low_pc * * The entries are sorted by low_pc and a binary search can be used to look * them up. In order to have a better cache locality, we only keep the * information we need for the lookup in this vector. We could probably * replace subprograms by an ordered vector of Frame and replace this one b * a parallel `std::vector`. */ std::vector functions_index; // TODO, remove the mutable (to remove it we'll have to add a lot of const everywhere) mutable std::vector global_variables; /** Types indexed by DWARF ID */ std::unordered_map types; /** Types indexed by name * * Different compilation units have their separate type definitions * (for the same type). When we find an opaque type in one compilation unit, * we use this in order to try to find its definition in another compilation * unit. */ std::unordered_map full_types_by_name; /** Whether this module is an executable * * More precisely we check if this is an ET_EXE ELF. These ELF files * use fixed addresses instead of base-addres relative addresses. * Position independant executables are in fact ET_DYN. */ bool executable() const { return this->flags & simgrid::mc::ObjectInformation::Executable; } /** Base address of the module * * All the location information in ELF and DWARF are expressed as an offsets * from this base address: * * - location of the functions and globale variables; * * - the DWARF instruction `OP_addr` pushes this on the DWARF stack. **/ void* base_address() const; /** Find a function by instruction address * * @param ip instruction address * @return corresponding function (if any) or nullptr */ simgrid::mc::Frame* find_function(const void *ip) const; /** Find a global variable by name * * This is used to ignore global variables and to find well-known variables * (`__mmalloc_default_mdp`). * * @param name scopes name of the global variable (`myproject::Foo::count`) * @return corresponding variable (if any) or nullptr */ simgrid::mc::Variable* find_variable(const char* name) const; /** Remove a global variable (in order to ignore it) * * This is used to ignore a global variable for the snapshot comparison. */ void remove_global_variable(const char* name); /** Remove a loval variables (in order to ignore it) * * @param name Name of the globale variable * @param scope Namespaceed name of the function (or null for all functions) */ void remove_local_variable( const char* name, const char* scope); }; XBT_PRIVATE std::shared_ptr createObjectInformation( std::vector const& maps, const char* name); /** Augment the current module with informations about the other ones */ XBT_PRIVATE void postProcessObjectInformation(simgrid::mc::RemoteClient* process, simgrid::mc::ObjectInformation* info); } } #endif SimGrid-3.18/src/mc/ChunkedData.hpp0000644000175000017500000000511113217757316017373 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHUNKED_DATA_HPP #define SIMGRID_MC_CHUNKED_DATA_HPP #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/PageStore.hpp" namespace simgrid { namespace mc { /** A byte-string represented as a sequence of chunks from a PageStore * * In order to save memory when taking memory snapshots, a given byte-string * is split in fixed-size chunks. Identical chunks (either from the same * snapshot or more probably from different snpashots) share the same memory * storage. * * Thus a chunked is represented as a sequence of indices of each chunk. */ class ChunkedData { /** This is where we store the chunks */ PageStore* store_ = nullptr; /** Indices of the chunks in the `PageStore` */ std::vector pagenos_; public: ChunkedData() = default; void clear() { for (std::size_t const& pageno : pagenos_) store_->unref_page(pageno); pagenos_.clear(); } ~ChunkedData() { clear(); } // Copy and move ChunkedData(ChunkedData const& that) : store_ (that.store_) , pagenos_(that.pagenos_) { for (std::size_t const& pageno : pagenos_) store_->ref_page(pageno); } ChunkedData(ChunkedData&& that) : store_(that.store_) , pagenos_(std::move(that.pagenos_)) { that.store_ = nullptr; that.pagenos_.clear(); } ChunkedData& operator=(ChunkedData const& that) { this->clear(); store_ = that.store_; pagenos_ = that.pagenos_; for (std::size_t const& pageno : pagenos_) store_->ref_page(pageno); return *this; } ChunkedData& operator=(ChunkedData && that) { this->clear(); store_ = that.store_; that.store_ = nullptr; pagenos_ = std::move(that.pagenos_); that.pagenos_.clear(); return *this; } /** How many pages are used */ std::size_t page_count() const { return pagenos_.size(); } /** Get a chunk index */ std::size_t pageno(std::size_t i) const { return pagenos_[i]; } /** Get a view of the chunk indices */ const std::size_t* pagenos() const { return pagenos_.data(); } /** Get a a pointer to a chunk */ const void* page(std::size_t i) const { return store_->get_page(pagenos_[i]); } ChunkedData(PageStore& store, AddressSpace& as, RemotePtr addr, std::size_t page_count); }; } } #endif SimGrid-3.18/src/mc/RegionSnapshot.hpp0000644000175000017500000001700713217757316020172 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REGION_SNAPSHOT_HPP #define SIMGRID_MC_REGION_SNAPSHOT_HPP #include #include #include #include #include "xbt/base.h" #include "src/mc/AddressSpace.hpp" #include "src/mc/ChunkedData.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/remote/RemotePtr.hpp" namespace simgrid { namespace mc { enum class RegionType { Unknown = 0, Heap = 1, Data = 2 }; enum class StorageType { NoData = 0, Flat = 1, Chunked = 2, Privatized = 3 }; class Buffer { private: enum class Type { Malloc, Mmap }; void* data_ = nullptr; std::size_t size_; Type type_ = Type::Malloc; Buffer(std::size_t size, Type type = Type::Malloc); Buffer(void* data, std::size_t size, Type type = Type::Malloc) : data_(data), size_(size), type_(type) {} public: Buffer() = default; void clear() noexcept; ~Buffer() noexcept { clear(); } static Buffer malloc(std::size_t size) { return Buffer(size, Type::Malloc); } static Buffer mmap(std::size_t size) { return Buffer(size, Type::Mmap); } // No copy Buffer(Buffer const& buffer) = delete; Buffer& operator=(Buffer const& buffer) = delete; // Move Buffer(Buffer&& that) noexcept : data_(that.data_), size_(that.size_), type_(that.type_) { that.data_ = nullptr; that.size_ = 0; that.type_ = Type::Malloc; } Buffer& operator=(Buffer&& that) noexcept { clear(); data_ = that.data_; size_ = that.size_; type_ = that.type_; that.data_ = nullptr; that.size_ = 0; that.type_ = Type::Malloc; return *this; } void* get() { return data_; } const void* get() const { return data_; } std::size_t size() const { return size_; } }; /** A copy/snapshot of a given memory region * * Different types of region snapshot storage types exist: * * * flat/dense snapshots are a simple copy of the region; * * * sparse/per-page snapshots are snaapshots which shared * identical pages. * * * privatized (SMPI global variable privatization). * * This is handled with a variant based approach: * * * `storage_type` identified the type of storage; * * * an anonymous enum is used to distinguish the relevant types for * each type. */ class RegionSnapshot { public: static const RegionType UnknownRegion = RegionType::Unknown; static const RegionType HeapRegion = RegionType::Heap; static const RegionType DataRegion = RegionType::Data; private: RegionType region_type_; StorageType storage_type_; simgrid::mc::ObjectInformation* object_info_; /** @brief Virtual address of the region in the simulated process */ void *start_addr_; /** @brief Size of the data region in bytes */ std::size_t size_; /** @brief Permanent virtual address of the region * * This is usually the same address as the simuilated process address. * However, when using SMPI privatization of global variables, * each SMPI process has its own set of global variables stored * at a different virtual address. The scheduler maps those region * on the region of the global variables. * * */ void *permanent_addr_; Buffer flat_data_; ChunkedData page_numbers_; std::vector privatized_regions_; public: RegionSnapshot() : region_type_(UnknownRegion), storage_type_(StorageType::NoData), object_info_(nullptr), start_addr_(nullptr), size_(0), permanent_addr_(nullptr) {} RegionSnapshot(RegionType type, void *start_addr, void* permanent_addr, size_t size) : region_type_(type), storage_type_(StorageType::NoData), object_info_(nullptr), start_addr_(start_addr), size_(size), permanent_addr_(permanent_addr) {} ~RegionSnapshot() = default; RegionSnapshot(RegionSnapshot const&) = default; RegionSnapshot& operator=(RegionSnapshot const&) = default; RegionSnapshot(RegionSnapshot&& that) : region_type_(that.region_type_) , storage_type_(that.storage_type_) , object_info_(that.object_info_) , start_addr_(that.start_addr_) , size_(that.size_) , permanent_addr_(that.permanent_addr_) , flat_data_(std::move(that.flat_data_)) , page_numbers_(std::move(that.page_numbers_)) , privatized_regions_(std::move(that.privatized_regions_)) { that.clear(); } RegionSnapshot& operator=(RegionSnapshot&& that) { region_type_ = that.region_type_; storage_type_ = that.storage_type_; object_info_ = that.object_info_; start_addr_ = that.start_addr_; size_ = that.size_; permanent_addr_ = that.permanent_addr_; flat_data_ = std::move(that.flat_data_); page_numbers_ = std::move(that.page_numbers_); privatized_regions_ = std::move(that.privatized_regions_); that.clear(); return *this; } // Data void clear() { region_type_ = UnknownRegion; storage_type_ = StorageType::NoData; privatized_regions_.clear(); page_numbers_.clear(); flat_data_.clear(); object_info_ = nullptr; start_addr_ = nullptr; size_ = 0; permanent_addr_ = nullptr; } void clear_data() { storage_type_ = StorageType::NoData; flat_data_.clear(); page_numbers_.clear(); privatized_regions_.clear(); } void flat_data(Buffer data) { storage_type_ = StorageType::Flat; flat_data_ = std::move(data); page_numbers_.clear(); privatized_regions_.clear(); } const Buffer& flat_data() const { return flat_data_; } Buffer& flat_data() { return flat_data_; } void page_data(ChunkedData page_data) { storage_type_ = StorageType::Chunked; flat_data_.clear(); page_numbers_ = std::move(page_data); privatized_regions_.clear(); } ChunkedData const& page_data() const { return page_numbers_; } void privatized_data(std::vector data) { storage_type_ = StorageType::Privatized; flat_data_.clear(); page_numbers_.clear(); privatized_regions_ = std::move(data); } std::vector const& privatized_data() const { return privatized_regions_; } std::vector& privatized_data() { return privatized_regions_; } simgrid::mc::ObjectInformation* object_info() const { return object_info_; } void object_info(simgrid::mc::ObjectInformation* info) { object_info_ = info; } // Other getters RemotePtr start() const { return remote(start_addr_); } RemotePtr end() const { return remote((char*)start_addr_ + size_); } RemotePtr permanent_address() const { return remote(permanent_addr_); } std::size_t size() const { return size_; } StorageType storage_type() const { return storage_type_; } RegionType region_type() const { return region_type_; } bool contain(RemotePtr p) const { return p >= start() && p < end(); } }; RegionSnapshot privatized_region( RegionType region_type, void *start_addr, void* permanent_addr, std::size_t size); RegionSnapshot dense_region( RegionType type, void *start_addr, void* data_addr, std::size_t size); simgrid::mc::RegionSnapshot sparse_region( RegionType type, void *start_addr, void* data_addr, std::size_t size); simgrid::mc::RegionSnapshot region( RegionType type, void *start_addr, void* data_addr, std::size_t size); } } typedef simgrid::mc::RegionSnapshot s_mc_mem_region_t; typedef s_mc_mem_region_t* mc_mem_region_t; #endif SimGrid-3.18/src/mc/mc_dwarf_attrnames.cpp0000644000175000017500000001317513217757316021064 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include #include "src/mc/mc_dwarf.hpp" namespace { const std::unordered_map attrname_map = { {0x01, "DW_AT_sibling"}, {0x02, "DW_AT_location"}, {0x03, "DW_AT_name"}, {0x09, "DW_AT_ordering"}, {0x0a, "DW_AT_subscr_data"}, {0x0b, "DW_AT_byte_size"}, {0x0c, "DW_AT_bit_offset"}, {0x0d, "DW_AT_bit_size"}, {0x0f, "DW_AT_element_list"}, {0x10, "DW_AT_stmt_list"}, {0x11, "DW_AT_low_pc"}, {0x12, "DW_AT_high_pc"}, {0x13, "DW_AT_language"}, {0x14, "DW_AT_member"}, {0x15, "DW_AT_discr"}, {0x16, "DW_AT_discr_value"}, {0x17, "DW_AT_visibility"}, {0x18, "DW_AT_import"}, {0x19, "DW_AT_string_length"}, {0x1a, "DW_AT_common_reference"}, {0x1b, "DW_AT_comp_dir"}, {0x1c, "DW_AT_const_value"}, {0x1d, "DW_AT_containing_type"}, {0x1e, "DW_AT_default_value"}, {0x20, "DW_AT_inline"}, {0x21, "DW_AT_is_optional"}, {0x22, "DW_AT_lower_bound"}, {0x25, "DW_AT_producer"}, {0x27, "DW_AT_prototyped"}, {0x2a, "DW_AT_return_addr"}, {0x2c, "DW_AT_start_scope"}, {0x2e, "DW_AT_bit_stride"}, {0x2f, "DW_AT_upper_bound"}, {0x31, "DW_AT_abstract_origin"}, {0x32, "DW_AT_accessibility"}, {0x33, "DW_AT_address_class"}, {0x34, "DW_AT_artificial"}, {0x35, "DW_AT_base_types"}, {0x36, "DW_AT_calling_convention"}, {0x37, "DW_AT_count"}, {0x38, "DW_AT_data_member_location"}, {0x39, "DW_AT_decl_column"}, {0x3a, "DW_AT_decl_file"}, {0x3b, "DW_AT_decl_line"}, {0x3c, "DW_AT_declaration"}, {0x3d, "DW_AT_discr_list"}, {0x3e, "DW_AT_encoding"}, {0x3f, "DW_AT_external"}, {0x40, "DW_AT_frame_base"}, {0x41, "DW_AT_friend"}, {0x42, "DW_AT_identifier_case"}, {0x43, "DW_AT_macro_info"}, {0x44, "DW_AT_namelist_item"}, {0x45, "DW_AT_priority"}, {0x46, "DW_AT_segment"}, {0x47, "DW_AT_specification"}, {0x48, "DW_AT_static_link"}, {0x49, "DW_AT_type"}, {0x4a, "DW_AT_use_location"}, {0x4b, "DW_AT_variable_parameter"}, {0x4c, "DW_AT_virtuality"}, {0x4d, "DW_AT_vtable_elem_location"}, {0x4e, "DW_AT_allocated"}, {0x4f, "DW_AT_associated"}, {0x50, "DW_AT_data_location"}, {0x51, "DW_AT_byte_stride"}, {0x52, "DW_AT_entry_pc"}, {0x53, "DW_AT_use_UTF8"}, {0x54, "DW_AT_extension"}, {0x55, "DW_AT_ranges"}, {0x56, "DW_AT_trampoline"}, {0x57, "DW_AT_call_column"}, {0x58, "DW_AT_call_file"}, {0x59, "DW_AT_call_line"}, {0x5a, "DW_AT_description"}, {0x5b, "DW_AT_binary_scale"}, {0x5c, "DW_AT_decimal_scale"}, {0x5d, "DW_AT_small"}, {0x5e, "DW_AT_decimal_sign"}, {0x5f, "DW_AT_digit_count"}, {0x60, "DW_AT_picture_string"}, {0x61, "DW_AT_mutable"}, {0x62, "DW_AT_threads_scaled"}, {0x63, "DW_AT_explicit"}, {0x64, "DW_AT_object_pointer"}, {0x65, "DW_AT_endianity"}, {0x66, "DW_AT_elemental"}, {0x67, "DW_AT_pure"}, {0x68, "DW_AT_recursive"}, {0x69, "DW_AT_signature"}, {0x6a, "DW_AT_main_subprogram"}, {0x6b, "DW_AT_data_bit_offset"}, {0x6c, "DW_AT_const_expr"}, {0x6d, "DW_AT_enum_class"}, {0x6e, "DW_AT_linkage_name"}, {0x87, "DW_AT_noreturn"}, {0x2000, "DW_AT_lo_user"}, {0x2001, "DW_AT_MIPS_fde"}, {0x2002, "DW_AT_MIPS_loop_begin"}, {0x2003, "DW_AT_MIPS_tail_loop_begin"}, {0x2004, "DW_AT_MIPS_epilog_begin"}, {0x2005, "DW_AT_MIPS_loop_unroll_factor"}, {0x2006, "DW_AT_MIPS_software_pipeline_depth"}, {0x2007, "DW_AT_MIPS_linkage_name"}, {0x2008, "DW_AT_MIPS_stride"}, {0x2009, "DW_AT_MIPS_abstract_name"}, {0x200a, "DW_AT_MIPS_clone_origin"}, {0x200b, "DW_AT_MIPS_has_inlines"}, {0x200c, "DW_AT_MIPS_stride_byte"}, {0x200d, "DW_AT_MIPS_stride_elem"}, {0x200e, "DW_AT_MIPS_ptr_dopetype"}, {0x200f, "DW_AT_MIPS_allocatable_dopetype"}, {0x2010, "DW_AT_MIPS_assumed_shape_dopetype"}, {0x2011, "DW_AT_MIPS_assumed_size"}, {0x2101, "DW_AT_sf_names"}, {0x2102, "DW_AT_src_info"}, {0x2103, "DW_AT_mac_info"}, {0x2104, "DW_AT_src_coords"}, {0x2105, "DW_AT_body_begin"}, {0x2106, "DW_AT_body_end"}, {0x2107, "DW_AT_GNU_vector"}, {0x2108, "DW_AT_GNU_guarded_by"}, {0x2109, "DW_AT_GNU_pt_guarded_by"}, {0x210a, "DW_AT_GNU_guarded"}, {0x210b, "DW_AT_GNU_pt_guarded"}, {0x210c, "DW_AT_GNU_locks_excluded"}, {0x210d, "DW_AT_GNU_exclusive_locks_required"}, {0x210e, "DW_AT_GNU_shared_locks_required"}, {0x210f, "DW_AT_GNU_odr_signature"}, {0x2110, "DW_AT_GNU_template_name"}, {0x2111, "DW_AT_GNU_call_site_value"}, {0x2112, "DW_AT_GNU_call_site_data_value"}, {0x2113, "DW_AT_GNU_call_site_target"}, {0x2114, "DW_AT_GNU_call_site_target_clobbered"}, {0x2115, "DW_AT_GNU_tail_call"}, {0x2116, "DW_AT_GNU_all_tail_call_sites"}, {0x2117, "DW_AT_GNU_all_call_sites"}, {0x2118, "DW_AT_GNU_all_source_call_sites"}, {0x2119, "DW_AT_GNU_macros"}, {0x211a, "DW_AT_GNU_deleted"}, {0x3fff, "DW_AT_hi_user"}, }; } namespace simgrid { namespace dwarf { /** \brief Get the name of an attribute (DW_AT_*) from its code * * \param attr attribute code (see the DWARF specification) * \return name of the attribute */ XBT_PRIVATE const char *attrname(int attr) { auto name = attrname_map.find(attr); return name == attrname_map.end() ? "DW_AT_unknown" : name->second; } } } SimGrid-3.18/src/mc/mc_ignore.h0000644000175000017500000000107413217757316016626 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_IGNORE_H #define SIMGRID_MC_IGNORE_H #include "src/internal_config.h" #if HAVE_UCONTEXT_H #include /* context relative declarations */ SG_BEGIN_DECL(); XBT_PUBLIC(void) MC_register_stack_area(void *stack, smx_actor_t process, ucontext_t* context, size_t size); SG_END_DECL(); #endif #endif SimGrid-3.18/src/mc/PageStore.hpp0000644000175000017500000001435313217757321017115 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PAGESTORE_HPP #define SIMGRID_MC_PAGESTORE_HPP #include #include #include #include #include "xbt/base.h" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_mmu.hpp" namespace simgrid { namespace mc { /** @brief Storage for snapshot memory pages * * The first (lower) layer of the per-page snapshot mechanism is a page store: * its responsibility is to store immutable sharable reference-counted memory * pages independently of the snapshotting logic. Snapshot management and * representation is handled to an higher layer. READMORE * * Data structure: * * * A pointer (`memory_`) to a (currently anonymous) `mmap()`ed memory * region holding the memory pages (the address of the first page). * * We want to keep this memory region aligned on the memory pages (so * that we might be able to create non-linear memory mappings on those * pages in the future) and be able to expand it without coyping the * data (there will be a lot of pages here): we will be able to * efficiently expand the memory mapping using `mremap()`, moving it * to another virtual address if necessary. * * Because we will move this memory mapping on the virtual address * space, only the index of the page will be stored in the snapshots * and the page will always be looked up by going through `memory`: * * void* page = (char*) page_store->memory + page_index << pagebits; * * * The number of pages mapped in virtual memory (`capacity_`). Once all * those pages are used, we need to expand the page store with * `mremap()`. * * * A reference count for each memory page `page_counts_`. Each time a * snapshot references a page, the counter is incremented. If a * snapshot is freed, the reference count is decremented. When the * reference count, of a page reaches 0 it is added to a list of available * pages (`free_pages_`). * * * A list of free pages `free_pages_` which can be reused. This avoids having * to scan the reference count list to find a free page. * * * When we are expanding the memory map we do not want to add thousand of page * to the `free_pages_` list and remove them just afterwards. The `top_index_` * field is an index after which all pages are free and are not in the `free_pages_` * list. * * * When we are adding a page, we need to check if a page with the same * content is already in the page store in order to reuse it. For this * reason, we maintain an index (`hash_index_`) mapping the hash of a * page to the list of page indices with this hash. * We use a fast (non cryptographic) hash so there may be conflicts: * we must be able to store multiple indices for the same hash. * */ class PageStore { public: // Types typedef std::uint64_t hash_type; private: // Types // We are using a cheap hash to index a page. // We should expect collision and we need to associate multiple page indices // to the same hash. typedef std::unordered_set page_set_type; typedef std::unordered_map pages_map_type; // Fields: /** First page */ void* memory_; /** Number of available pages in virtual memory */ std::size_t capacity_; /** Top of the used pages (index of the next available page) */ std::size_t top_index_; /** Page reference count */ std::vector page_counts_; /** Index of available pages before the top */ std::vector free_pages_; /** Index from page hash to page index */ pages_map_type hash_index_; // Methods void resize(std::size_t size); std::size_t alloc_page(); void remove_page(std::size_t pageno); public: // Constructors PageStore(PageStore const&) = delete; PageStore& operator=(PageStore const&) = delete; explicit PageStore(std::size_t size); ~PageStore(); // Methods /** @brief Decrement the reference count for a given page * * Decrement the reference count of this page. Used when a snapshot is destroyed. * * If the reference count reaches zero, the page is recycled: * it is added to the `free_pages_` list and removed from the `hash_index_`. * * */ void unref_page(std::size_t pageno); /** @brief Increment the refcount for a given page * * This method used to increase a reference count of a page if we know * that the content of a page is the same as a page already in the page * store. * * This will be the case if a page if soft clean: we know that is has not * changed since the previous cnapshot/restoration and we can avoid * hashing the page, comparing byte-per-byte to candidates. * */ void ref_page(size_t pageno); /** @brief Store a page in the page store */ std::size_t store_page(void* page); /** @brief Get a page from its page number * * @param pageno Number of the memory page in the store * @return Start of the page */ const void* get_page(std::size_t pageno) const; // Debug/test methods /** @brief Get the number of references for a page */ std::size_t get_ref(std::size_t pageno); /** @brief Get the number of used pages */ std::size_t size(); /** @brief Get the capacity of the page store * * The capacity is expanded by a system call (mremap). * */ std::size_t capacity(); }; XBT_ALWAYS_INLINE void PageStore::unref_page(std::size_t pageno) { if ((--this->page_counts_[pageno]) == 0) this->remove_page(pageno); } XBT_ALWAYS_INLINE void PageStore::ref_page(size_t pageno) { ++this->page_counts_[pageno]; } XBT_ALWAYS_INLINE const void* PageStore::get_page(std::size_t pageno) const { return (void*) simgrid::mc::mmu::join(pageno, (std::uintptr_t) this->memory_); } XBT_ALWAYS_INLINE std::size_t PageStore::get_ref(std::size_t pageno) { return this->page_counts_[pageno]; } XBT_ALWAYS_INLINE std::size_t PageStore::size() { return this->top_index_ - this->free_pages_.size(); } XBT_ALWAYS_INLINE std::size_t PageStore::capacity() { return this->capacity_; } } } #endif SimGrid-3.18/src/mc/ModelChecker.cpp0000644000175000017500000002762413217757316017555 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include "xbt/automaton.h" #include "xbt/automaton.hpp" #include "xbt/log.h" #include "xbt/system_error.hpp" #include "simgrid/sg_config.h" #include "src/mc/ModelChecker.hpp" #include "src/mc/ModelChecker.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/Transition.hpp" #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_exit.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_record.hpp" #include "src/mc/remote/mc_protocol.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_ModelChecker, mc, "ModelChecker"); ::simgrid::mc::ModelChecker* mc_model_checker = nullptr; extern std::string _sg_mc_dot_output_file; using simgrid::mc::remote; #ifdef __linux__ # define WAITPID_CHECKED_FLAGS __WALL #else # define WAITPID_CHECKED_FLAGS 0 #endif namespace simgrid { namespace mc { ModelChecker::ModelChecker(std::unique_ptr process) : base_(nullptr) , socket_event_(nullptr) , signal_event_(nullptr) , page_store_(500) , process_(std::move(process)) , parent_snapshot_(nullptr) { } ModelChecker::~ModelChecker() { if (socket_event_ != nullptr) event_free(socket_event_); if (signal_event_ != nullptr) event_free(signal_event_); if (base_ != nullptr) event_base_free(base_); } void ModelChecker::start() { const pid_t pid = process_->pid(); base_ = event_base_new(); event_callback_fn event_callback = [](evutil_socket_t fd, short events, void *arg) { ((ModelChecker *)arg)->handle_events(fd, events); }; socket_event_ = event_new(base_, process_->getChannel().getSocket(), EV_READ|EV_PERSIST, event_callback, this); event_add(socket_event_, NULL); signal_event_ = event_new(base_, SIGCHLD, EV_SIGNAL|EV_PERSIST, event_callback, this); event_add(signal_event_, NULL); XBT_DEBUG("Waiting for the model-checked process"); int status; // The model-checked process SIGSTOP itself to signal it's ready: pid_t res = waitpid(pid, &status, WAITPID_CHECKED_FLAGS); if (res < 0 || not WIFSTOPPED(status) || WSTOPSIG(status) != SIGSTOP) xbt_die("Could not wait model-checked process"); process_->init(); if (not _sg_mc_dot_output_file.empty()) MC_init_dot_output(); setup_ignore(); #ifdef __linux__ ptrace(PTRACE_SETOPTIONS, pid, nullptr, PTRACE_O_TRACEEXIT); ptrace(PTRACE_CONT, pid, 0, 0); #elif defined BSD ptrace(PT_CONTINUE, pid, (caddr_t)1, 0); #else # error "no ptrace equivalent coded for this platform" #endif } static const std::pair ignored_local_variables[] = { std::pair{ "e", "*" }, std::pair{ "__ex_cleanup", "*" }, std::pair{ "__ex_mctx_en", "*" }, std::pair{ "__ex_mctx_me", "*" }, std::pair{ "__xbt_ex_ctx_ptr", "*" }, std::pair{ "_log_ev", "*" }, std::pair{ "_throw_ctx", "*" }, std::pair{ "ctx", "*" }, std::pair{ "self", "simcall_BODY_mc_snapshot" }, std::pair{ "next_context", "smx_ctx_sysv_suspend_serial" }, std::pair{ "i", "smx_ctx_sysv_suspend_serial" }, /* Ignore local variable about time used for tracing */ std::pair{ "start_time", "*" }, }; void ModelChecker::setup_ignore() { RemoteClient& process = this->process(); for (std::pair const& var : ignored_local_variables) process.ignore_local_variable(var.first, var.second); /* Static variable used for tracing */ process.ignore_global_variable("counter"); } void ModelChecker::shutdown() { XBT_DEBUG("Shuting down model-checker"); simgrid::mc::RemoteClient* process = &this->process(); if (process->running()) { XBT_DEBUG("Killing process"); kill(process->pid(), SIGKILL); process->terminate(); } } void ModelChecker::resume(simgrid::mc::RemoteClient& process) { int res = process.getChannel().send(MC_MESSAGE_CONTINUE); if (res) throw simgrid::xbt::errno_error(); process.clear_cache(); } static void MC_report_crash(int status) { XBT_INFO("**************************"); XBT_INFO("** CRASH IN THE PROGRAM **"); XBT_INFO("**************************"); if (WIFSIGNALED(status)) XBT_INFO("From signal: %s", strsignal(WTERMSIG(status))); else if (WIFEXITED(status)) XBT_INFO("From exit: %i", WEXITSTATUS(status)); if (WCOREDUMP(status)) XBT_INFO("A core dump was generated by the system."); else XBT_INFO("No core dump was generated by the system."); XBT_INFO("Counter-example execution trace:"); simgrid::mc::dumpRecordPath(); for (auto const& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); XBT_INFO("Stack trace:"); mc_model_checker->process().dumpStack(); } static void MC_report_assertion_error() { XBT_INFO("**************************"); XBT_INFO("*** PROPERTY NOT VALID ***"); XBT_INFO("**************************"); XBT_INFO("Counter-example execution trace:"); simgrid::mc::dumpRecordPath(); for (auto const& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); } bool ModelChecker::handle_message(char* buffer, ssize_t size) { s_mc_message_t base_message; if (size < (ssize_t) sizeof(base_message)) xbt_die("Broken message"); memcpy(&base_message, buffer, sizeof(base_message)); switch(base_message.type) { case MC_MESSAGE_IGNORE_HEAP: { s_mc_message_ignore_heap_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); IgnoredHeapRegion region; region.block = message.block; region.fragment = message.fragment; region.address = message.address; region.size = message.size; process().ignore_heap(region); break; } case MC_MESSAGE_UNIGNORE_HEAP: { s_mc_message_ignore_memory_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); process().unignore_heap((void*)(std::uintptr_t)message.addr, message.size); break; } case MC_MESSAGE_IGNORE_MEMORY: { s_mc_message_ignore_memory_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); this->process().ignore_region(message.addr, message.size); break; } case MC_MESSAGE_STACK_REGION: { s_mc_message_stack_region_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); this->process().stack_areas().push_back(message.stack_region); } break; case MC_MESSAGE_REGISTER_SYMBOL: { s_mc_message_register_symbol_t message; if (size != sizeof(message)) xbt_die("Broken message"); memcpy(&message, buffer, sizeof(message)); if (message.callback) xbt_die("Support for client-side function proposition is not implemented."); XBT_DEBUG("Received symbol: %s", message.name); if (simgrid::mc::property_automaton == nullptr) simgrid::mc::property_automaton = xbt_automaton_new(); simgrid::mc::RemoteClient* process = &this->process(); simgrid::mc::RemotePtr address = simgrid::mc::remote((int*)message.data); simgrid::xbt::add_proposition(simgrid::mc::property_automaton, message.name, [process, address]() { return process->read(address); }); break; } case MC_MESSAGE_WAITING: return false; case MC_MESSAGE_ASSERTION_FAILED: MC_report_assertion_error(); this->exit(SIMGRID_MC_EXIT_SAFETY); break; default: xbt_die("Unexpected message from model-checked application"); } return true; } /** Terminate the model-checker application */ void ModelChecker::exit(int status) { // TODO, terminate the model checker politely instead of exiting rudely if (process().running()) kill(process().pid(), SIGKILL); ::exit(status); } void ModelChecker::handle_events(int fd, short events) { if (events == EV_READ) { char buffer[MC_MESSAGE_LENGTH]; ssize_t size = process_->getChannel().receive(buffer, sizeof(buffer), false); if (size == -1 && errno != EAGAIN) throw simgrid::xbt::errno_error(); if (not handle_message(buffer, size)) { event_base_loopbreak(base_); } } else if (events == EV_SIGNAL) { on_signal(fd); } else { xbt_die("Unexpected event"); } } void ModelChecker::loop() { if (this->process().running()) event_base_dispatch(base_); } void ModelChecker::handle_waitpid() { XBT_DEBUG("Check for wait event"); int status; pid_t pid; while ((pid = waitpid(-1, &status, WNOHANG)) != 0) { if (pid == -1) { if (errno == ECHILD) { // No more children: if (this->process().running()) xbt_die("Inconsistent state"); else break; } else { XBT_ERROR("Could not wait for pid"); throw simgrid::xbt::errno_error(); } } if (pid == this->process().pid()) { // From PTRACE_O_TRACEEXIT: #ifdef __linux__ if (status>>8 == (SIGTRAP | (PTRACE_EVENT_EXIT<<8))) { if (ptrace(PTRACE_GETEVENTMSG, this->process().pid(), 0, &status) == -1) xbt_die("Could not get exit status"); if (WIFSIGNALED(status)) { MC_report_crash(status); mc_model_checker->exit(SIMGRID_MC_EXIT_PROGRAM_CRASH); } } #endif // We don't care about signals, just reinject them: if (WIFSTOPPED(status)) { XBT_DEBUG("Stopped with signal %i", (int) WSTOPSIG(status)); errno = 0; #ifdef __linux__ ptrace(PTRACE_CONT, this->process().pid(), 0, WSTOPSIG(status)); #elif defined BSD ptrace(PT_CONTINUE, this->process().pid(), (caddr_t)1, WSTOPSIG(status)); #endif if (errno != 0) xbt_die("Could not PTRACE_CONT"); } else if (WIFEXITED(status) || WIFSIGNALED(status)) { XBT_DEBUG("Child process is over"); this->process().terminate(); } } } } void ModelChecker::on_signal(int signo) { if (signo == SIGCHLD) this->handle_waitpid(); } void ModelChecker::wait_for_requests() { this->resume(process()); if (this->process().running()) event_base_dispatch(base_); } void ModelChecker::handle_simcall(Transition const& transition) { s_mc_message_simcall_handle_t m; memset(&m, 0, sizeof(m)); m.type = MC_MESSAGE_SIMCALL_HANDLE; m.pid = transition.pid; m.value = transition.argument; this->process_->getChannel().send(m); this->process_->clear_cache(); if (this->process_->running()) event_base_dispatch(base_); } bool ModelChecker::checkDeadlock() { int res; if ((res = this->process().getChannel().send(MC_MESSAGE_DEADLOCK_CHECK))) xbt_die("Could not check deadlock state"); s_mc_message_int_t message; ssize_t s = mc_model_checker->process().getChannel().receive(message); if (s == -1) xbt_die("Could not receive message"); if (s != sizeof(message) || message.type != MC_MESSAGE_DEADLOCK_CHECK_REPLY) xbt_die("Received unexpected message %s (%i, size=%i) " "expected MC_MESSAGE_DEADLOCK_CHECK_REPLY (%i, size=%i)", MC_message_type_name(message.type), (int) message.type, (int) s, (int) MC_MESSAGE_DEADLOCK_CHECK_REPLY, (int) sizeof(message) ); return message.value != 0; } } } SimGrid-3.18/src/mc/remote/0000755000175000017500000000000013217757316016004 5ustar mquinsonmquinsonSimGrid-3.18/src/mc/remote/Channel.hpp0000644000175000017500000000364513217757316020075 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHANNEL_HPP #define SIMGRID_MC_CHANNEL_HPP #include #include #include "src/mc/remote/mc_protocol.h" namespace simgrid { namespace mc { /** A channel for exchanging messages between model-checker and model-checked * * This abstracts away the way the messages are transferred. Currently, they * are sent over a (connected) `SOCK_SEQPACKET` socket. */ class Channel { int socket_ = -1; template static constexpr bool messageType() { return std::is_class::value && std::is_trivial::value; } public: Channel() = default; explicit Channel(int sock) : socket_(sock) {} ~Channel(); // No copy: Channel(Channel const&) = delete; Channel& operator=(Channel const&) = delete; // Move: Channel(Channel&& that) : socket_(that.socket_) { that.socket_ = -1; } Channel& operator=(Channel&& that) { this->socket_ = that.socket_; that.socket_ = -1; return *this; } // Send int send(const void* message, size_t size) const; int send(e_mc_message_type type) const { s_mc_message_t message = {type}; return this->send(&message, sizeof(message)); } /** @brief Send a message; returns 0 on success or errno on failure */ template typename std::enable_if(), int>::type send(M const& m) const { return this->send(&m, sizeof(M)); } // Receive ssize_t receive(void* message, size_t size, bool block = true) const; template typename std::enable_if(), ssize_t>::type receive(M& m) const { return this->receive(&m, sizeof(M)); } int getSocket() const { return socket_; } }; } } #endif SimGrid-3.18/src/mc/remote/mc_protocol.h0000644000175000017500000000540313217757316020477 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PROTOCOL_H #define SIMGRID_MC_PROTOCOL_H #include #include #include "mc/datatypes.h" #include "simgrid/forward.h" SG_BEGIN_DECL() // ***** Environment variables for passing context to the model-checked process /** Environment variable name set by `simgrid-mc` to enable MC support in the * children MC processes */ #define MC_ENV_VARIABLE "SIMGRID_MC" /** Environment variable name used to pass the communication socket */ #define MC_ENV_SOCKET_FD "SIMGRID_MC_SOCKET_FD" // ***** Messages enum e_mc_message_type { MC_MESSAGE_NONE, MC_MESSAGE_CONTINUE, MC_MESSAGE_IGNORE_HEAP, MC_MESSAGE_UNIGNORE_HEAP, MC_MESSAGE_IGNORE_MEMORY, MC_MESSAGE_STACK_REGION, MC_MESSAGE_REGISTER_SYMBOL, MC_MESSAGE_DEADLOCK_CHECK, MC_MESSAGE_DEADLOCK_CHECK_REPLY, MC_MESSAGE_WAITING, MC_MESSAGE_SIMCALL_HANDLE, MC_MESSAGE_ASSERTION_FAILED, // MCer request to finish the restoration: MC_MESSAGE_RESTORE, MC_MESSAGE_ACTOR_ENABLED, MC_MESSAGE_ACTOR_ENABLED_REPLY }; #define MC_MESSAGE_LENGTH 512 /** Basic structure for a MC message * * The current version of the client/server protocol sends C structures over `AF_LOCAL` * `SOCK_SEQPACKET` sockets. This means that the protocol is ABI/architecture specific: * we currently can't model-check a x86 process from a x86_64 process. * * Moreover the protocol is not stable. The same version of the library should be used * for the client and the server. */ /* Basic structure: all message start with a message type */ struct s_mc_message_t { enum e_mc_message_type type; }; struct s_mc_message_int_t { enum e_mc_message_type type; uint64_t value; }; /* Client->Server */ struct s_mc_message_ignore_heap_t { enum e_mc_message_type type; int block; int fragment; void* address; size_t size; }; struct s_mc_message_ignore_memory_t { enum e_mc_message_type type; uint64_t addr; size_t size; }; struct s_mc_message_stack_region_t { enum e_mc_message_type type; s_stack_region_t stack_region; }; struct s_mc_message_register_symbol_t { enum e_mc_message_type type; char name[128]; int (*callback)(void*); void* data; }; /* Server -> client */ struct s_mc_message_simcall_handle_t { enum e_mc_message_type type; unsigned long pid; int value; }; struct s_mc_message_restore_t { enum e_mc_message_type type; int index; }; struct s_mc_message_actor_enabled_t { enum e_mc_message_type type; aid_t aid; // actor ID }; XBT_PRIVATE const char* MC_message_type_name(enum e_mc_message_type type); SG_END_DECL() #endif SimGrid-3.18/src/mc/remote/RemotePtr.hpp0000644000175000017500000001130613217757316020437 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REMOTE_PTR_HPP #define SIMGRID_MC_REMOTE_PTR_HPP #include #include #include #include #include namespace simgrid { namespace mc { /** HACK, A value from another process * * This represents a value from another process: * * * constructor/destructor are disabled; * * * raw memory copy (std::memcpy) is used to copy Remote; * * * raw memory comparison is used to compare them; * * * when T is a trivial type, Remote is convertible to a T. * * We currently only handle the case where the type has the same layout * in the current process and in the target process: we don't handle * cross-architecture (such as 32-bit/64-bit access). */ template union Remote { private: T buffer; public: Remote() { /* Nothing to do */} ~Remote() { /* Nothing to do */} Remote(T const& p) { std::memcpy(static_cast(&buffer), static_cast(&p), sizeof(buffer)); } Remote(Remote const& that) { std::memcpy(static_cast(&buffer), static_cast(&that.buffer), sizeof(buffer)); } Remote& operator=(Remote const& that) { std::memcpy(static_cast(&buffer), static_cast(&that.buffer), sizeof(buffer)); return *this; } T* getBuffer() { return &buffer; } const T* getBuffer() const { return &buffer; } std::size_t getBufferSize() const { return sizeof(T); } operator T() const { //FIXME: assert turned off because smpi:Request is not seen as "trivial". // static_assert(std::is_trivial::value, "Cannot convert non trivial type"); return buffer; } void clear() { std::memset(static_cast(&buffer), 0, sizeof(T)); } }; /** Pointer to a remote address-space (process, snapshot) * * With this we can clearly identify the expected type of an address in the * remote process while avoiding to use native local pointers. * * Some operators (+/-) assume use the size of the underlying element. This * only works if the target applications is using the same target: it won't * work for example, when inspecting a 32 bit application from a 64 bit * model-checker. * * We do not actually store the target address space because we can * always detect it in context. This way `RemotePtr` is as efficient * as a `uint64_t`. */ template class RemotePtr { std::uint64_t address_; public: RemotePtr() : address_(0) {} explicit RemotePtr(std::nullptr_t) : address_(0) {} explicit RemotePtr(std::uint64_t address) : address_(address) {} explicit RemotePtr(T* address) : address_((std::uintptr_t)address) {} explicit RemotePtr(Remote p) : RemotePtr(*p.getBuffer()) {} std::uint64_t address() const { return address_; } /** Turn into a local pointer * (if the remote process is not, in fact, remote) */ T* local() const { return (T*)address_; } operator bool() const { return address_; } bool operator!() const { return not address_; } operator RemotePtr() const { return RemotePtr(address_); } RemotePtr& operator=(std::nullptr_t) { address_ = 0; return *this; } RemotePtr operator+(std::uint64_t n) const { return RemotePtr(address_ + n * sizeof(T)); } RemotePtr operator-(std::uint64_t n) const { return RemotePtr(address_ - n * sizeof(T)); } RemotePtr& operator+=(std::uint64_t n) { address_ += n * sizeof(T); return *this; } RemotePtr& operator-=(std::uint64_t n) { address_ -= n * sizeof(T); return *this; } }; template bool operator<(RemotePtr const& x, RemotePtr const& y) { return x.address() < y.address(); } template bool operator>(RemotePtr const& x, RemotePtr const& y) { return x.address() > y.address(); } template bool operator>=(RemotePtr const& x, RemotePtr const& y) { return x.address() >= y.address(); } template bool operator<=(RemotePtr const& x, RemotePtr const& y) { return x.address() <= y.address(); } template bool operator==(RemotePtr const& x, RemotePtr const& y) { return x.address() == y.address(); } template bool operator!=(RemotePtr const& x, RemotePtr const& y) { return x.address() != y.address(); } template inline RemotePtr remote(T* p) { return RemotePtr(p); } template inline RemotePtr remote(uint64_t p) { return RemotePtr(p); } } } #endif SimGrid-3.18/src/mc/remote/RemoteClient.cpp0000644000175000017500000004622413217757316021112 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #define _FILE_OFFSET_BITS 64 /* needed for pread_whole to work as expected on 32bits */ #include #include #include #include #include #include #include #include #include #include #include // PROT_* #include #include #include #include #include #include #include "xbt/base.h" #include "xbt/file.hpp" #include "xbt/log.h" #include #include "src/mc/mc_smx.hpp" #include "src/mc/mc_snapshot.hpp" #include "src/mc/mc_unw.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" #include "src/mc/remote/RemoteClient.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_process, mc, "MC process information"); namespace simgrid { namespace mc { // ***** Helper stuff // List of library which memory segments are not considered: static const std::vector filtered_libraries = { #ifdef __linux__ "ld", #elif defined __FreeBSD__ "ld-elf", "ld-elf32", "libkvm", /* kernel data access library */ "libprocstat", /* process and file information retrieval */ "libthr", /* thread library */ "libutil", #endif "libasan", /* gcc sanitizers */ "libargp", /* workarounds for glibc-less systems */ "libtsan", "libubsan", "libbz2", "libboost_chrono", "libboost_context", "libboost_context-mt", "libboost_system", "libboost_thread", "libc", "libc++", "libcdt", "libcgraph", "libcrypto", "libcxxrt", "libdl", "libdw", "libelf", "libevent", "libgcc_s", "liblua5.1", "liblua5.3", "liblzma", "libm", "libpthread", "librt", "libstdc++", "libunwind", "libunwind-x86_64", "libunwind-x86", "libunwind-ptrace", "libz"}; static bool is_simgrid_lib(const std::string& libname) { return libname == "libsimgrid"; } static bool is_filtered_lib(const std::string& libname) { return std::find(begin(filtered_libraries), end(filtered_libraries), libname) != end(filtered_libraries); } static std::string get_lib_name(const std::string& pathname) { constexpr char digits[] = ".0123456789"; std::string map_basename = simgrid::xbt::Path(pathname).getBasename(); std::string libname; size_t pos = map_basename.rfind(".so"); if (pos != std::string::npos && map_basename.find_first_not_of(digits, pos + 3) == std::string::npos) { // strip the extension (matching regex "\.so[.0-9]*$") libname.assign(map_basename, 0, pos); // strip the version suffix (matching regex "-[.0-9-]*$") while (true) { pos = libname.rfind('-'); if (pos == std::string::npos || libname.find_first_not_of(digits, pos + 1) != std::string::npos) break; libname.erase(pos); } } return libname; } static ssize_t pread_whole(int fd, void* buf, size_t count, off_t offset) { char* buffer = (char*)buf; ssize_t real_count = count; while (count) { ssize_t res = pread(fd, buffer, count, offset); if (res > 0) { count -= res; buffer += res; offset += res; } else if (res == 0) return -1; else if (errno != EINTR) { perror("pread_whole"); return -1; } } return real_count; } static ssize_t pwrite_whole(int fd, const void* buf, size_t count, off_t offset) { const char* buffer = (const char*)buf; ssize_t real_count = count; while (count) { ssize_t res = pwrite(fd, buffer, count, offset); if (res > 0) { count -= res; buffer += res; offset += res; } else if (res == 0) return -1; else if (errno != EINTR) return -1; } return real_count; } static pthread_once_t zero_buffer_flag = PTHREAD_ONCE_INIT; static const void* zero_buffer; static const size_t zero_buffer_size = 10 * 4096; static void zero_buffer_init() { int fd = open("/dev/zero", O_RDONLY); if (fd < 0) xbt_die("Could not open /dev/zero"); zero_buffer = mmap(nullptr, zero_buffer_size, PROT_READ, MAP_SHARED, fd, 0); if (zero_buffer == MAP_FAILED) xbt_die("Could not map the zero buffer"); close(fd); } int open_vm(pid_t pid, int flags) { const size_t buffer_size = 30; char buffer[buffer_size]; int res = snprintf(buffer, buffer_size, "/proc/%lli/mem", (long long)pid); if (res < 0 || (size_t)res >= buffer_size) { errno = ENAMETOOLONG; return -1; } return open(buffer, flags); } // ***** Process RemoteClient::RemoteClient(pid_t pid, int sockfd) : AddressSpace(this), pid_(pid), channel_(sockfd), running_(true) { } void RemoteClient::init() { this->memory_map_ = simgrid::xbt::get_memory_map(this->pid_); this->init_memory_map_info(); int fd = open_vm(this->pid_, O_RDWR); if (fd < 0) xbt_die("Could not open file for process virtual address space"); this->memory_file = fd; // Read std_heap (is a struct mdesc*): simgrid::mc::Variable* std_heap_var = this->find_variable("__mmalloc_default_mdp"); if (not std_heap_var) xbt_die("No heap information in the target process"); if (not std_heap_var->address) xbt_die("No constant address for this variable"); this->read_bytes(&this->heap_address, sizeof(mdesc*), remote(std_heap_var->address), simgrid::mc::ProcessIndexDisabled); this->smx_actors_infos.clear(); this->smx_dead_actors_infos.clear(); this->unw_addr_space = simgrid::mc::UnwindContext::createUnwindAddressSpace(); this->unw_underlying_addr_space = simgrid::unw::create_addr_space(); this->unw_underlying_context = simgrid::unw::create_context(this->unw_underlying_addr_space, this->pid_); } RemoteClient::~RemoteClient() { if (this->memory_file >= 0) close(this->memory_file); if (this->unw_underlying_addr_space != unw_local_addr_space) { if (this->unw_underlying_addr_space) unw_destroy_addr_space(this->unw_underlying_addr_space); if (this->unw_underlying_context) _UPT_destroy(this->unw_underlying_context); } unw_destroy_addr_space(this->unw_addr_space); } /** Refresh the information about the process * * Do not use directly, this is used by the getters when appropriate * in order to have fresh data. */ void RemoteClient::refresh_heap() { // Read/dereference/refresh the std_heap pointer: if (not this->heap) this->heap = std::unique_ptr(new s_xbt_mheap_t()); this->read_bytes(this->heap.get(), sizeof(mdesc), remote(this->heap_address), simgrid::mc::ProcessIndexDisabled); this->cache_flags_ |= RemoteClient::cache_heap; } /** Refresh the information about the process * * Do not use direclty, this is used by the getters when appropriate * in order to have fresh data. * */ void RemoteClient::refresh_malloc_info() { // Refresh process->heapinfo: if (this->cache_flags_ & RemoteClient::cache_malloc) return; size_t count = this->heap->heaplimit + 1; if (this->heap_info.size() < count) this->heap_info.resize(count); this->read_bytes(this->heap_info.data(), count * sizeof(malloc_info), remote(this->heap->heapinfo), simgrid::mc::ProcessIndexDisabled); this->cache_flags_ |= RemoteClient::cache_malloc; } /** @brief Finds the range of the different memory segments and binary paths */ void RemoteClient::init_memory_map_info() { XBT_DEBUG("Get debug information ..."); this->maestro_stack_start_ = nullptr; this->maestro_stack_end_ = nullptr; this->object_infos.resize(0); this->binary_info = nullptr; this->libsimgrid_info = nullptr; std::vector const& maps = this->memory_map_; const char* current_name = nullptr; this->object_infos.clear(); for (size_t i = 0; i < maps.size(); i++) { simgrid::xbt::VmMap const& reg = maps[i]; const char* pathname = maps[i].pathname.c_str(); // Nothing to do if (maps[i].pathname.empty()) { current_name = nullptr; continue; } // [stack], [vvar], [vsyscall], [vdso] ... if (pathname[0] == '[') { if ((reg.prot & PROT_WRITE) && not memcmp(pathname, "[stack]", 7)) { this->maestro_stack_start_ = remote(reg.start_addr); this->maestro_stack_end_ = remote(reg.end_addr); } current_name = nullptr; continue; } if (current_name && strcmp(current_name, pathname) == 0) continue; current_name = pathname; if (not(reg.prot & PROT_READ) && (reg.prot & PROT_EXEC)) continue; const bool is_executable = not i; std::string libname; if (not is_executable) { libname = get_lib_name(pathname); if (is_filtered_lib(libname)) { continue; } } std::shared_ptr info = simgrid::mc::createObjectInformation(this->memory_map_, pathname); this->object_infos.push_back(info); if (is_executable) this->binary_info = info; else if (is_simgrid_lib(libname)) this->libsimgrid_info = info; } // Resolve time (including across different objects): for (auto const& object_info : this->object_infos) postProcessObjectInformation(this, object_info.get()); xbt_assert(this->maestro_stack_start_, "Did not find maestro_stack_start"); xbt_assert(this->maestro_stack_end_, "Did not find maestro_stack_end"); XBT_DEBUG("Get debug information done !"); } std::shared_ptr RemoteClient::find_object_info(RemotePtr addr) const { for (auto const& object_info : this->object_infos) if (addr.address() >= (std::uint64_t)object_info->start && addr.address() <= (std::uint64_t)object_info->end) return object_info; return nullptr; } std::shared_ptr RemoteClient::find_object_info_exec(RemotePtr addr) const { for (std::shared_ptr const& info : this->object_infos) if (addr.address() >= (std::uint64_t)info->start_exec && addr.address() <= (std::uint64_t)info->end_exec) return info; return nullptr; } std::shared_ptr RemoteClient::find_object_info_rw(RemotePtr addr) const { for (std::shared_ptr const& info : this->object_infos) if (addr.address() >= (std::uint64_t)info->start_rw && addr.address() <= (std::uint64_t)info->end_rw) return info; return nullptr; } simgrid::mc::Frame* RemoteClient::find_function(RemotePtr ip) const { std::shared_ptr info = this->find_object_info_exec(ip); return info ? info->find_function((void*)ip.address()) : nullptr; } /** Find (one occurrence of) the named variable definition */ simgrid::mc::Variable* RemoteClient::find_variable(const char* name) const { // First lookup the variable in the executable shared object. // A global variable used directly by the executable code from a library // is reinstanciated in the executable memory .data/.bss. // We need to look up the variable in the executable first. if (this->binary_info) { std::shared_ptr const& info = this->binary_info; simgrid::mc::Variable* var = info->find_variable(name); if (var) return var; } for (std::shared_ptr const& info : this->object_infos) { simgrid::mc::Variable* var = info->find_variable(name); if (var) return var; } return nullptr; } void RemoteClient::read_variable(const char* name, void* target, size_t size) const { simgrid::mc::Variable* var = this->find_variable(name); xbt_assert(var->address, "No simple location for this variable"); xbt_assert(var->type->full_type, "Partial type for %s, cannot check size", name); xbt_assert((size_t)var->type->full_type->byte_size == size, "Unexpected size for %s (expected %zu, was %zu)", name, size, (size_t)var->type->full_type->byte_size); this->read_bytes(target, size, remote(var->address)); } std::string RemoteClient::read_string(RemotePtr address) const { if (not address) return {}; std::vector res(128); off_t off = 0; while (1) { ssize_t c = pread(this->memory_file, res.data() + off, res.size() - off, (off_t)address.address() + off); if (c == -1) { if (errno == EINTR) continue; else xbt_die("Could not read from from remote process"); } if (c == 0) xbt_die("Could not read string from remote process"); void* p = memchr(res.data() + off, '\0', c); if (p) return std::string(res.data()); off += c; if (off == (off_t)res.size()) res.resize(res.size() * 2); } } const void* RemoteClient::read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index, ReadOptions options) const { if (process_index != simgrid::mc::ProcessIndexDisabled) { std::shared_ptr const& info = this->find_object_info_rw(address); // Segment overlap is not handled. #if HAVE_SMPI if (info.get() && this->privatized(*info)) { if (process_index < 0) xbt_die("Missing process index"); if (process_index >= (int)MC_smpi_process_count()) xbt_die("Invalid process index"); // Read smpi_privatization_regions from MCed: smpi_privatization_region_t remote_smpi_privatization_regions = mc_model_checker->process().read_variable("smpi_privatization_regions"); s_smpi_privatization_region_t privatization_region = mc_model_checker->process().read( remote(remote_smpi_privatization_regions + process_index)); // Address translation in the privatization segment: size_t offset = address.address() - (std::uint64_t)info->start_rw; address = remote((char*)privatization_region.address + offset); } #endif } if (pread_whole(this->memory_file, buffer, size, (size_t)address.address()) < 0) xbt_die("Read at %p from process %lli failed", (void*)address.address(), (long long)this->pid_); return buffer; } /** Write data to a process memory * * @param buffer local memory address (source) * @param len data size * @param address target process memory address (target) */ void RemoteClient::write_bytes(const void* buffer, size_t len, RemotePtr address) { if (pwrite_whole(this->memory_file, buffer, len, (size_t)address.address()) < 0) xbt_die("Write to process %lli failed", (long long)this->pid_); } void RemoteClient::clear_bytes(RemotePtr address, size_t len) { pthread_once(&zero_buffer_flag, zero_buffer_init); while (len) { size_t s = len > zero_buffer_size ? zero_buffer_size : len; this->write_bytes(zero_buffer, s, address); address = remote((char*)address.address() + s); len -= s; } } void RemoteClient::ignore_region(std::uint64_t addr, std::size_t size) { IgnoredRegion region; region.addr = addr; region.size = size; if (ignored_regions_.empty()) { ignored_regions_.push_back(region); return; } unsigned int cursor = 0; IgnoredRegion* current_region = nullptr; int start = 0; int end = ignored_regions_.size() - 1; while (start <= end) { cursor = (start + end) / 2; current_region = &ignored_regions_[cursor]; if (current_region->addr == addr) { if (current_region->size == size) return; else if (current_region->size < size) start = cursor + 1; else end = cursor - 1; } else if (current_region->addr < addr) start = cursor + 1; else end = cursor - 1; } std::size_t position; if (current_region->addr == addr) { if (current_region->size < size) position = cursor + 1; else position = cursor; } else if (current_region->addr < addr) position = cursor + 1; else position = cursor; ignored_regions_.insert(ignored_regions_.begin() + position, region); } void RemoteClient::ignore_heap(IgnoredHeapRegion const& region) { if (ignored_heap_.empty()) { ignored_heap_.push_back(std::move(region)); return; } typedef std::vector::size_type size_type; size_type start = 0; size_type end = ignored_heap_.size() - 1; // Binary search the position of insertion: size_type cursor; while (start <= end) { cursor = start + (end - start) / 2; auto& current_region = ignored_heap_[cursor]; if (current_region.address == region.address) return; else if (current_region.address < region.address) start = cursor + 1; else if (cursor != 0) end = cursor - 1; // Avoid underflow: else break; } // Insert it mc_heap_ignore_region_t: if (ignored_heap_[cursor].address < region.address) ++cursor; ignored_heap_.insert(ignored_heap_.begin() + cursor, region); } void RemoteClient::unignore_heap(void* address, size_t size) { typedef std::vector::size_type size_type; size_type start = 0; size_type end = ignored_heap_.size() - 1; // Binary search: size_type cursor; while (start <= end) { cursor = (start + end) / 2; auto& region = ignored_heap_[cursor]; if (region.address < address) start = cursor + 1; else if ((char*)region.address <= ((char*)address + size)) { ignored_heap_.erase(ignored_heap_.begin() + cursor); return; } else if (cursor != 0) end = cursor - 1; // Avoid underflow: else break; } } void RemoteClient::ignore_local_variable(const char* var_name, const char* frame_name) { if (frame_name != nullptr && strcmp(frame_name, "*") == 0) frame_name = nullptr; for (std::shared_ptr const& info : this->object_infos) info->remove_local_variable(var_name, frame_name); } std::vector& RemoteClient::actors() { this->refresh_simix(); return smx_actors_infos; } std::vector& RemoteClient::dead_actors() { this->refresh_simix(); return smx_dead_actors_infos; } void RemoteClient::dumpStack() { unw_addr_space_t as = unw_create_addr_space(&_UPT_accessors, BYTE_ORDER); if (as == nullptr) { XBT_ERROR("Could not initialize ptrace address space"); return; } void* context = _UPT_create(this->pid_); if (context == nullptr) { unw_destroy_addr_space(as); XBT_ERROR("Could not initialize ptrace context"); return; } unw_cursor_t cursor; if (unw_init_remote(&cursor, as, context) != 0) { _UPT_destroy(context); unw_destroy_addr_space(as); XBT_ERROR("Could not initialiez ptrace cursor"); return; } simgrid::mc::dumpStack(stderr, cursor); _UPT_destroy(context); unw_destroy_addr_space(as); } bool RemoteClient::actor_is_enabled(aid_t pid) { s_mc_message_actor_enabled_t msg{MC_MESSAGE_ACTOR_ENABLED, pid}; process()->getChannel().send(msg); char buff[MC_MESSAGE_LENGTH]; ssize_t received = process()->getChannel().receive(buff, MC_MESSAGE_LENGTH, true); xbt_assert(received == sizeof(s_mc_message_int_t), "Unexpected size in answer to ACTOR_ENABLED"); return ((s_mc_message_int_t*)buff)->value; } } } SimGrid-3.18/src/mc/remote/Channel.cpp0000644000175000017500000000230113217757316020054 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/remote/Channel.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_Channel, mc, "MC interprocess communication"); namespace simgrid { namespace mc { Channel::~Channel() { if (this->socket_ >= 0) close(this->socket_); } /** @brief Send a message; returns 0 on success or errno on failure */ int Channel::send(const void* message, size_t size) const { XBT_DEBUG("Send %s", MC_message_type_name(*(e_mc_message_type*)message)); while (::send(this->socket_, message, size, 0) == -1) { if (errno != EINTR) return errno; } return 0; } ssize_t Channel::receive(void* message, size_t size, bool block) const { int res = recv(this->socket_, message, size, block ? 0 : MSG_DONTWAIT); if (res != -1) XBT_DEBUG("Receive %s", MC_message_type_name(*(e_mc_message_type*)message)); return res; } } } SimGrid-3.18/src/mc/remote/mc_protocol.cpp0000644000175000017500000000313413217757316021031 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include // std::size_t #include // perror #include #include #include #include #include "src/mc/remote/Client.hpp" #include "src/mc/remote/mc_protocol.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_protocol, mc, "Generic MC protocol logic"); const char* MC_message_type_name(e_mc_message_type type) { switch (type) { case MC_MESSAGE_NONE: return "NONE"; case MC_MESSAGE_CONTINUE: return "CONTINUE"; case MC_MESSAGE_IGNORE_HEAP: return "IGNORE_HEAP"; case MC_MESSAGE_UNIGNORE_HEAP: return "UNIGNORE_HEAP"; case MC_MESSAGE_IGNORE_MEMORY: return "IGNORE_MEMORY"; case MC_MESSAGE_STACK_REGION: return "STACK_REGION"; case MC_MESSAGE_REGISTER_SYMBOL: return "REGISTER_SYMBOL"; case MC_MESSAGE_DEADLOCK_CHECK: return "DEADLOCK_CHECK"; case MC_MESSAGE_DEADLOCK_CHECK_REPLY: return "DEADLOCK_CHECK_REPLY"; case MC_MESSAGE_WAITING: return "WAITING"; case MC_MESSAGE_SIMCALL_HANDLE: return "SIMCALL_HANDLE"; case MC_MESSAGE_ASSERTION_FAILED: return "ASSERTION_FAILED"; case MC_MESSAGE_ACTOR_ENABLED: return "ACTOR_ENABLED"; case MC_MESSAGE_ACTOR_ENABLED_REPLY: return "ACTOR_ENABLED_REPLY"; default: return "?"; } } SimGrid-3.18/src/mc/remote/Client.hpp0000644000175000017500000000346713217757316017745 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CLIENT_H #define SIMGRID_MC_CLIENT_H #include "src/internal_config.h" #include #include #include #include #include "src/mc/remote/Channel.hpp" #include "src/mc/remote/mc_protocol.h" namespace simgrid { namespace mc { /** Model-checked-side of the communication protocol * * Send messages to the model-checker and handles message from it. */ class XBT_PUBLIC() Client { private: Channel channel_; static std::unique_ptr instance_; public: Client(); explicit Client(int fd) : channel_(fd) {} void handleMessages(); private: void handleDeadlockCheck(s_mc_message_t* msg); void handleContinue(s_mc_message_t* msg); void handleSimcall(s_mc_message_simcall_handle_t* message); void handleRestore(s_mc_message_restore_t* msg); void handleActorEnabled(s_mc_message_actor_enabled_t* msg); public: Channel const& getChannel() const { return channel_; } Channel& getChannel() { return channel_; } void mainLoop(); void reportAssertionFailure(const char* description = nullptr); void ignoreMemory(void* addr, std::size_t size); void ignoreHeap(void* addr, std::size_t size); void unignoreHeap(void* addr, std::size_t size); void declareSymbol(const char* name, int* value); #if HAVE_UCONTEXT_H void declareStack(void* stack, size_t size, smx_actor_t process, ucontext_t* context); #endif // Singleton :/ // TODO, remove the singleton antipattern. static Client* initialize(); static Client* get() { return instance_.get(); } }; } } #endif SimGrid-3.18/src/mc/remote/Client.cpp0000644000175000017500000002072713217757316017736 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include "src/internal_config.h" #include "src/mc/mc_request.hpp" #include "src/mc/remote/Client.hpp" #include "src/mc/remote/mc_protocol.h" #include "src/smpi/include/private.hpp" // We won't need those once the separation MCer/MCed is complete: #include "src/mc/mc_smx.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_client, mc, "MC client logic"); namespace simgrid { namespace mc { std::unique_ptr Client::instance_; Client* Client::initialize() { // We are not in MC mode: // TODO, handle this more gracefully. if (not std::getenv(MC_ENV_SOCKET_FD)) return nullptr; // Do not break if we are called multiple times: if (instance_) return instance_.get(); _sg_do_model_check = 1; // Fetch socket from MC_ENV_SOCKET_FD: char* fd_env = std::getenv(MC_ENV_SOCKET_FD); if (not fd_env) xbt_die("No MC socket passed in the environment"); int fd = xbt_str_parse_int(fd_env, bprintf("Variable %s should contain a number but contains '%%s'", MC_ENV_SOCKET_FD)); XBT_DEBUG("Model-checked application found socket FD %i", fd); // Check the socket type/validity: int type; socklen_t socklen = sizeof(type); if (getsockopt(fd, SOL_SOCKET, SO_TYPE, &type, &socklen) != 0) xbt_die("Could not check socket type"); if (type != SOCK_SEQPACKET) xbt_die("Unexpected socket type %i", type); XBT_DEBUG("Model-checked application found expected socket type"); instance_ = std::unique_ptr(new simgrid::mc::Client(fd)); // Wait for the model-checker: errno = 0; #if defined __linux__ ptrace(PTRACE_TRACEME, 0, nullptr, nullptr); #elif defined BSD ptrace(PT_TRACE_ME, 0, nullptr, 0); #else #error "no ptrace equivalent coded for this platform" #endif if (errno != 0 || raise(SIGSTOP) != 0) xbt_die("Could not wait for the model-checker"); instance_->handleMessages(); return instance_.get(); } void Client::handleDeadlockCheck(s_mc_message_t* msg) { bool deadlock = false; if (not simix_global->process_list.empty()) { deadlock = true; for (auto const& kv : simix_global->process_list) if (simgrid::mc::actor_is_enabled(kv.second)) { deadlock = false; break; } } // Send result: s_mc_message_int_t answer{MC_MESSAGE_DEADLOCK_CHECK_REPLY, deadlock}; xbt_assert(channel_.send(answer) == 0, "Could not send response"); } void Client::handleContinue(s_mc_message_t* msg) { /* Nothing to do */ } void Client::handleSimcall(s_mc_message_simcall_handle_t* message) { smx_actor_t process = SIMIX_process_from_PID(message->pid); if (not process) xbt_die("Invalid pid %lu", (unsigned long)message->pid); SIMIX_simcall_handle(&process->simcall, message->value); if (channel_.send(MC_MESSAGE_WAITING)) xbt_die("Could not send MESSAGE_WAITING to model-checker"); } void Client::handleRestore(s_mc_message_restore_t* message) { #if HAVE_SMPI smpi_really_switch_data_segment(message->index); #endif } void Client::handleActorEnabled(s_mc_message_actor_enabled_t* msg) { bool res = simgrid::mc::actor_is_enabled(SIMIX_process_from_PID(msg->aid)); s_mc_message_int_t answer{MC_MESSAGE_ACTOR_ENABLED_REPLY, res}; channel_.send(answer); } void Client::handleMessages() { while (1) { XBT_DEBUG("Waiting messages from model-checker"); char message_buffer[MC_MESSAGE_LENGTH]; ssize_t received_size = channel_.receive(&message_buffer, sizeof(message_buffer)); if (received_size < 0) xbt_die("Could not receive commands from the model-checker"); s_mc_message_t* message = (s_mc_message_t*)message_buffer; switch (message->type) { case MC_MESSAGE_DEADLOCK_CHECK: xbt_assert(received_size == sizeof(s_mc_message_t), "Unexpected size for DEADLOCK_CHECK (%zd != %zu)", received_size, sizeof(s_mc_message_t)); handleDeadlockCheck(message); break; case MC_MESSAGE_CONTINUE: xbt_assert(received_size == sizeof(s_mc_message_t), "Unexpected size for MESSAGE_CONTINUE (%zd != %zu)", received_size, sizeof(s_mc_message_t)); handleContinue(message); return; case MC_MESSAGE_SIMCALL_HANDLE: xbt_assert(received_size == sizeof(s_mc_message_simcall_handle_t), "Unexpected size for SIMCALL_HANDLE (%zd != %zu)", received_size, sizeof(s_mc_message_simcall_handle_t)); handleSimcall((s_mc_message_simcall_handle_t*)message_buffer); break; case MC_MESSAGE_RESTORE: xbt_assert(received_size == sizeof(s_mc_message_t), "Unexpected size for MESSAGE_RESTORE (%zd != %zu)", received_size, sizeof(s_mc_message_t)); handleRestore((s_mc_message_restore_t*)message_buffer); break; case MC_MESSAGE_ACTOR_ENABLED: xbt_assert(received_size == sizeof(s_mc_message_actor_enabled_t), "Unexpected size for ACTOR_ENABLED (%zd != %zu)", received_size, sizeof(s_mc_message_actor_enabled_t)); handleActorEnabled((s_mc_message_actor_enabled_t*)message_buffer); break; default: xbt_die("Received unexpected message %s (%i)", MC_message_type_name(message->type), message->type); break; } } } void Client::mainLoop() { while (1) { simgrid::mc::wait_for_requests(); xbt_assert(channel_.send(MC_MESSAGE_WAITING) == 0, "Could not send WAITING message to model-checker"); this->handleMessages(); } } void Client::reportAssertionFailure(const char* description) { if (channel_.send(MC_MESSAGE_ASSERTION_FAILED)) xbt_die("Could not send assertion to model-checker"); this->handleMessages(); } void Client::ignoreMemory(void* addr, std::size_t size) { s_mc_message_ignore_memory_t message; message.type = MC_MESSAGE_IGNORE_MEMORY; message.addr = (std::uintptr_t)addr; message.size = size; if (channel_.send(message)) xbt_die("Could not send IGNORE_MEMORY mesage to model-checker"); } void Client::ignoreHeap(void* address, std::size_t size) { xbt_mheap_t heap = mmalloc_get_current_heap(); s_mc_message_ignore_heap_t message; message.type = MC_MESSAGE_IGNORE_HEAP; message.address = address; message.size = size; message.block = ((char*)address - (char*)heap->heapbase) / BLOCKSIZE + 1; if (heap->heapinfo[message.block].type == 0) { message.fragment = -1; heap->heapinfo[message.block].busy_block.ignore++; } else { message.fragment = ((uintptr_t)(ADDR2UINT(address) % (BLOCKSIZE))) >> heap->heapinfo[message.block].type; heap->heapinfo[message.block].busy_frag.ignore[message.fragment]++; } if (channel_.send(message)) xbt_die("Could not send ignored region to MCer"); } void Client::unignoreHeap(void* address, std::size_t size) { s_mc_message_ignore_memory_t message; message.type = MC_MESSAGE_UNIGNORE_HEAP; message.addr = (std::uintptr_t)address; message.size = size; if (channel_.send(message)) xbt_die("Could not send IGNORE_HEAP message to model-checker"); } void Client::declareSymbol(const char* name, int* value) { s_mc_message_register_symbol_t message; message.type = MC_MESSAGE_REGISTER_SYMBOL; if (strlen(name) + 1 > sizeof(message.name)) xbt_die("Symbol is too long"); strncpy(message.name, name, sizeof(message.name)); message.callback = nullptr; message.data = value; if (channel_.send(message)) xbt_die("Could send REGISTER_SYMBOL message to model-checker"); } void Client::declareStack(void* stack, size_t size, smx_actor_t process, ucontext_t* context) { xbt_mheap_t heap = mmalloc_get_current_heap(); s_stack_region_t region; memset(®ion, 0, sizeof(region)); region.address = stack; region.context = context; region.size = size; region.block = ((char*)stack - (char*)heap->heapbase) / BLOCKSIZE + 1; #if HAVE_SMPI if (smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP && process) region.process_index = process->pid - 1; else #endif region.process_index = -1; s_mc_message_stack_region_t message; message.type = MC_MESSAGE_STACK_REGION; message.stack_region = region; if (channel_.send(message)) xbt_die("Could not send STACK_REGION to model-checker"); } } } SimGrid-3.18/src/mc/remote/RemoteClient.hpp0000644000175000017500000002245713217757316021121 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PROCESS_H #define SIMGRID_MC_PROCESS_H #include #include #include #include #include #include #include #include #include "xbt/base.h" #include #include "src/xbt/mmalloc/mmprivate.h" #include "src/mc/remote/Channel.hpp" #include "src/mc/remote/RemotePtr.hpp" #include "src/simix/popping_private.hpp" #include "src/simix/smx_private.hpp" #include #include "src/xbt/memory_map.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/mc_base.h" #include "src/mc/mc_forward.hpp" #include "src/mc/remote/mc_protocol.h" namespace simgrid { namespace mc { class ActorInformation { public: /** MCed address of the process */ RemotePtr address{nullptr}; Remote copy; /** Hostname (owned by `mc_modelchecker->hostnames`) */ const char* hostname = nullptr; std::string name; void clear() { name.clear(); address = nullptr; hostname = nullptr; } }; struct IgnoredRegion { std::uint64_t addr; std::size_t size; }; struct IgnoredHeapRegion { int block; int fragment; void* address; std::size_t size; }; /** The Model-Checked process, seen from the MCer perspective * * This class is mixing a lot of different responsibilities and is tied * to SIMIX. It should probably be split into different classes. * * Responsibilities: * * - reading from the process memory (`AddressSpace`); * - accessing the system state of the process (heap, …); * - storing the SIMIX state of the process; * - privatization; * - communication with the model-checked process; * - stack unwinding; * - etc. */ class RemoteClient final : public AddressSpace { private: // Those flags are used to track down which cached information // is still up to date and which information needs to be updated. static constexpr int cache_none = 0; static constexpr int cache_heap = 1; static constexpr int cache_malloc = 2; static constexpr int cache_simix_processes = 4; public: RemoteClient(pid_t pid, int sockfd); ~RemoteClient(); void init(); RemoteClient(RemoteClient const&) = delete; RemoteClient(RemoteClient&&) = delete; RemoteClient& operator=(RemoteClient const&) = delete; RemoteClient& operator=(RemoteClient&&) = delete; // Read memory: const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const override; void read_variable(const char* name, void* target, size_t size) const; template void read_variable(const char* name, T* target) const { read_variable(name, target, sizeof(*target)); } template Remote read_variable(const char* name) const { Remote res; read_variable(name, res.getBuffer(), sizeof(T)); return res; } std::string read_string(RemotePtr address) const; using AddressSpace::read_string; // Write memory: void write_bytes(const void* buffer, size_t len, RemotePtr address); void clear_bytes(RemotePtr address, size_t len); // Debug information: std::shared_ptr find_object_info(RemotePtr addr) const; std::shared_ptr find_object_info_exec(RemotePtr addr) const; std::shared_ptr find_object_info_rw(RemotePtr addr) const; simgrid::mc::Frame* find_function(RemotePtr ip) const; simgrid::mc::Variable* find_variable(const char* name) const; // Heap access: xbt_mheap_t get_heap() { if (not(this->cache_flags_ & RemoteClient::cache_heap)) this->refresh_heap(); return this->heap.get(); } const malloc_info* get_malloc_info() { if (not(this->cache_flags_ & RemoteClient::cache_malloc)) this->refresh_malloc_info(); return this->heap_info.data(); } void clear_cache() { this->cache_flags_ = RemoteClient::cache_none; } Channel const& getChannel() const { return channel_; } Channel& getChannel() { return channel_; } std::vector const& ignored_regions() const { return ignored_regions_; } void ignore_region(std::uint64_t address, std::size_t size); pid_t pid() const { return pid_; } bool in_maestro_stack(RemotePtr p) const { return p >= this->maestro_stack_start_ && p < this->maestro_stack_end_; } bool running() const { return running_; } void terminate() { running_ = false; } bool privatized(ObjectInformation const& info) const { return privatized_ && info.executable(); } bool privatized() const { return privatized_; } void privatized(bool privatized) { privatized_ = privatized; } void ignore_global_variable(const char* name) { for (std::shared_ptr const& info : this->object_infos) info->remove_global_variable(name); } std::vector& stack_areas() { return stack_areas_; } std::vector const& stack_areas() const { return stack_areas_; } std::vector const& ignored_heap() const { return ignored_heap_; } void ignore_heap(IgnoredHeapRegion const& region); void unignore_heap(void* address, size_t size); void ignore_local_variable(const char* var_name, const char* frame_name); std::vector& actors(); std::vector& dead_actors(); /** Get a local description of a remote SIMIX actor */ simgrid::mc::ActorInformation* resolveActorInfo(simgrid::mc::RemotePtr actor) { xbt_assert(mc_model_checker != nullptr); if (not actor) return nullptr; this->refresh_simix(); for (auto& actor_info : this->smx_actors_infos) if (actor_info.address == actor) return &actor_info; for (auto& actor_info : this->smx_dead_actors_infos) if (actor_info.address == actor) return &actor_info; return nullptr; } /** Get a local copy of the SIMIX actor structure */ simgrid::simix::ActorImpl* resolveActor(simgrid::mc::RemotePtr process) { simgrid::mc::ActorInformation* actor_info = this->resolveActorInfo(process); if (actor_info) return actor_info->copy.getBuffer(); else return nullptr; } void dumpStack(); private: void init_memory_map_info(); void refresh_heap(); void refresh_malloc_info(); void refresh_simix(); pid_t pid_ = -1; Channel channel_; bool running_ = false; std::vector memory_map_; RemotePtr maestro_stack_start_; RemotePtr maestro_stack_end_; int memory_file = -1; std::vector ignored_regions_; bool privatized_ = false; std::vector stack_areas_; std::vector ignored_heap_; public: // object info // TODO, make private (first, objectify simgrid::mc::ObjectInformation*) std::vector> object_infos; std::shared_ptr libsimgrid_info; std::shared_ptr binary_info; // Copies of MCed SMX data structures /** Copy of `simix_global->process_list` * * See mc_smx.c. */ std::vector smx_actors_infos; /** Copy of `simix_global->process_to_destroy` * * See mc_smx.c. */ std::vector smx_dead_actors_infos; private: /** State of the cache (which variables are up to date) */ int cache_flags_ = RemoteClient::cache_none; public: /** Address of the heap structure in the MCed process. */ void* heap_address; /** Copy of the heap structure of the process * * This is refreshed with the `MC_process_refresh` call. * This is not used if the process is the current one: * use `get_heap_info()` in order to use it. */ std::unique_ptr heap; /** Copy of the allocation info structure * * This is refreshed with the `MC_process_refresh` call. * This is not used if the process is the current one: * use `get_malloc_info()` in order to use it. */ std::vector heap_info; // Libunwind-data /** Full-featured MC-aware libunwind address space for the process * * This address space is using a simgrid::mc::UnwindContext* * (with simgrid::mc::Process* / simgrid::mc::AddressSpace* * and unw_context_t). */ unw_addr_space_t unw_addr_space; /** Underlying libunwind address-space * * The `find_proc_info`, `put_unwind_info`, `get_dyn_info_list_addr` * operations of the native MC address space is currently delegated * to this address space (either the local or a ptrace unwinder). */ unw_addr_space_t unw_underlying_addr_space; /** The corresponding context */ void* unw_underlying_context; /* Check whether the given actor is enabled */ bool actor_is_enabled(aid_t pid); }; /** Open a FD to a remote process memory (`/dev/$pid/mem`) */ XBT_PRIVATE int open_vm(pid_t pid, int flags); } } #endif SimGrid-3.18/src/mc/mc_base.h0000644000175000017500000000262613217757315016260 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_BASE_H #define SIMGRID_MC_BASE_H #include "simgrid/forward.h" #ifdef __cplusplus #include namespace simgrid { namespace mc { /** Execute everything which is invisible * * Execute all the processes that are ready to run and all invisible simcalls * iteratively until there doesn't remain any. At this point, the function * returns to the caller which can handle the visible (and ready) simcalls. */ XBT_PRIVATE void wait_for_requests(); XBT_PRIVATE extern std::vector processes_time; /** Execute a given simcall */ XBT_PRIVATE void handle_simcall(smx_simcall_t req, int req_num); /** Is the process ready to execute its simcall? * * This is true if the request associated with the process is ready. * * Most requests are always enabled but WAIT and WAITANY * are not always enabled: a WAIT where the communication does not * have both a source and a destination yet is not enabled * (unless timeout is enabled in the wait and enabeld in SimGridMC). */ XBT_PRIVATE bool actor_is_enabled(smx_actor_t process); /** Check if the given simcall is visible */ XBT_PRIVATE bool request_is_visible(smx_simcall_t req); } } #endif #endif SimGrid-3.18/src/mc/mc_member.cpp0000644000175000017500000000232013217757316017140 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/mc/Type.hpp" #include "src/mc/mc_dwarf.hpp" #include "src/mc/mc_private.hpp" namespace simgrid { namespace dwarf { /** Resolve snapshot in the process address space * * @param object Process address of the struct/class * @param type Type of the struct/class * @param member Member description * @param snapshot Snapshot (or nullptr) * @return Process address of the given member of the 'object' struct/class */ void *resolve_member( const void *base, simgrid::mc::Type* type, simgrid::mc::Member* member, simgrid::mc::AddressSpace* address_space, int process_index) { ExpressionContext state; state.frame_base = nullptr; state.cursor = nullptr; state.address_space = address_space; state.process_index = process_index; ExpressionStack stack; stack.push((ExpressionStack::value_type) base); simgrid::dwarf::execute(member->location_expression, state, stack); return (void*) stack.top(); } } } SimGrid-3.18/src/mc/mc_config.cpp0000644000175000017500000001704013217757315017142 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/config.h" #include "xbt/log.h" #include #include "src/mc/mc_replay.hpp" #include #include #if SIMGRID_HAVE_MC #include "src/mc/mc_private.hpp" #include "src/mc/mc_safety.hpp" #endif #include "src/mc/mc_record.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_config, mc, "Configuration of the Model Checker"); #if SIMGRID_HAVE_MC namespace simgrid { namespace mc { /* Configuration support */ simgrid::mc::ReductionMode reduction_mode = simgrid::mc::ReductionMode::unset; } } #endif #if !SIMGRID_HAVE_MC #define _sg_do_model_check 0 #endif int _sg_mc_timeout = 0; void _mc_cfg_cb_timeout(const char *name) { if (_sg_cfg_init_status && not(_sg_do_model_check || not MC_record_path.empty())) xbt_die("You are specifying a value to enable/disable timeout for wait requests after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_timeout = xbt_cfg_get_boolean(name); } #if SIMGRID_HAVE_MC int _sg_do_model_check = 0; int _sg_do_model_check_record = 0; int _sg_mc_checkpoint = 0; int _sg_mc_sparse_checkpoint = 0; int _sg_mc_ksm = 0; std::string _sg_mc_property_file; int _sg_mc_hash = 0; int _sg_mc_max_depth = 1000; int _sg_mc_max_visited_states = 0; std::string _sg_mc_dot_output_file; int _sg_mc_comms_determinism = 0; int _sg_mc_send_determinism = 0; int _sg_mc_snapshot_fds = 0; int _sg_mc_termination = 0; void _mc_cfg_cb_reduce(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a reduction strategy after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); std::string val = xbt_cfg_get_string(name); if (val == "none") simgrid::mc::reduction_mode = simgrid::mc::ReductionMode::none; else if (val == "dpor") simgrid::mc::reduction_mode = simgrid::mc::ReductionMode::dpor; else xbt_die("configuration option %s can only take 'none' or 'dpor' as a value", name); } void _mc_cfg_cb_checkpoint(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a checkpointing value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_checkpoint = xbt_cfg_get_int(name); } void _mc_cfg_cb_sparse_checkpoint(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die("You are specifying a checkpointing value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_sparse_checkpoint = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_ksm(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die("You are specifying a KSM value after the initialization (through MSG_config?), but model-checking was not activated at config time (through --cfg=model-check:1). This won't work, sorry."); _sg_mc_ksm = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_property(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a property after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_property_file = xbt_cfg_get_string(name); } void _mc_cfg_cb_hash(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the use of global hash to speedup state comparaison, but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_hash = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_snapshot_fds(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the use of FD snapshotting, but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_snapshot_fds = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_max_depth(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a max depth value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_max_depth = xbt_cfg_get_int(name); } void _mc_cfg_cb_visited(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a number of stored visited states after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_max_visited_states = xbt_cfg_get_int(name); } void _mc_cfg_cb_dot_output(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a file name for a dot output of graph state after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_dot_output_file = xbt_cfg_get_string(name); } void _mc_cfg_cb_comms_determinism(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of determinism in the communications schemes after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_comms_determinism = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_send_determinism(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of send-determinism in the communications schemes after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_send_determinism = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_termination(const char *name) { if (_sg_cfg_init_status && not _sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of non progressive cycles after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_termination = xbt_cfg_get_boolean(name); } #endif SimGrid-3.18/src/mc/mc_hash.hpp0000644000175000017500000000076213217757316016631 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_HASH_HPP #define SIMGRID_MC_HASH_HPP #include "xbt/base.h" #include "src/mc/mc_forward.hpp" namespace simgrid { namespace mc { typedef std::uint64_t hash_type; XBT_PRIVATE hash_type hash(simgrid::mc::Snapshot const& snapshot); } } #endif SimGrid-3.18/src/mc/VisitedState.cpp0000644000175000017500000001036113217757316017626 0ustar mquinsonmquinson/* Copyright (c) 2011-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "xbt/log.h" #include "xbt/sysdep.h" #include "src/mc/VisitedState.hpp" #include "src/mc/mc_comm_pattern.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/remote/RemoteClient.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_VisitedState, mc, "Logging specific to state equality detection mechanisms"); namespace simgrid { namespace mc { static int snapshot_compare(simgrid::mc::VisitedState* state1, simgrid::mc::VisitedState* state2) { simgrid::mc::Snapshot* s1 = state1->system_state.get(); simgrid::mc::Snapshot* s2 = state2->system_state.get(); int num1 = state1->num; int num2 = state2->num; return snapshot_compare(num1, s1, num2, s2); } /** @brief Save the current state */ VisitedState::VisitedState(unsigned long state_number) : num(state_number) { simgrid::mc::RemoteClient* process = &(mc_model_checker->process()); this->heap_bytes_used = mmalloc_get_bytes_used_remote( process->get_heap()->heaplimit, process->get_malloc_info()); this->actors_count = mc_model_checker->process().actors().size(); this->system_state = simgrid::mc::take_snapshot(state_number); this->original_num = -1; } void VisitedStates::prune() { while (states_.size() > (std::size_t)_sg_mc_max_visited_states) { XBT_DEBUG("Try to remove visited state (maximum number of stored states reached)"); auto min_element = boost::range::min_element(states_, [](std::unique_ptr& a, std::unique_ptr& b) { return a->num < b->num; }); xbt_assert(min_element != states_.end()); // and drop it: states_.erase(min_element); XBT_DEBUG("Remove visited state (maximum number of stored states reached)"); } } /** \brief Checks whether a given state has already been visited by the algorithm. */ std::unique_ptr VisitedStates::addVisitedState( unsigned long state_number, simgrid::mc::State* graph_state, bool compare_snpashots) { std::unique_ptr new_state = std::unique_ptr(new VisitedState(state_number)); graph_state->system_state = new_state->system_state; XBT_DEBUG("Snapshot %p of visited state %d (exploration stack state %d)", new_state->system_state.get(), new_state->num, graph_state->num); auto range = boost::range::equal_range(states_, new_state.get(), simgrid::mc::DerefAndCompareByActorsCountAndUsedHeap()); if (compare_snpashots) for (auto i = range.first; i != range.second; ++i) { auto& visited_state = *i; if (snapshot_compare(visited_state.get(), new_state.get()) == 0) { // The state has been visited: std::unique_ptr old_state = std::move(visited_state); if (old_state->original_num == -1) // I'm the copy of an original process new_state->original_num = old_state->num; else // I'm the copy of a copy new_state->original_num = old_state->original_num; if (dot_output == nullptr) XBT_DEBUG("State %d already visited ! (equal to state %d)", new_state->num, old_state->num); else XBT_DEBUG("State %d already visited ! (equal to state %d (state %d in dot_output))", new_state->num, old_state->num, new_state->original_num); /* Replace the old state with the new one (with a bigger num) (when the max number of visited states is reached, the oldest one is removed according to its number (= with the min number) */ XBT_DEBUG("Replace visited state %d with the new visited state %d", old_state->num, new_state->num); visited_state = std::move(new_state); return old_state; } } XBT_DEBUG("Insert new visited state %d (total : %lu)", new_state->num, (unsigned long) states_.size()); states_.insert(range.first, std::move(new_state)); this->prune(); return nullptr; } } } SimGrid-3.18/src/mc/mc_record.hpp0000644000175000017500000000303513217757321017154 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file mc_record.hpp * * This file contains the MC replay/record functionnality. * A MC path may be recorded by using ``-cfg=model-check/record:1`'`. * The path is written in the log output and an be replayed with MC disabled * (even with an non-LC build) with `--cfg=model-check/replay:$replayPath`. * * The same version of Simgrid should be used and the same arguments should be * passed to the application (without the MC specific arguments). */ #ifndef SIMGRID_MC_RECORD_HPP #define SIMGRID_MC_RECORD_HPP #include "src/mc/Transition.hpp" #include "xbt/base.h" #include namespace simgrid { namespace mc { typedef std::vector RecordTrace; /** Convert a string representation of the path into a array of `simgrid::mc::Transition` */ XBT_PRIVATE RecordTrace parseRecordTrace(const char* data); XBT_PRIVATE std::string traceToString(simgrid::mc::RecordTrace const& trace); XBT_PRIVATE void dumpRecordPath(); XBT_PRIVATE void replay(RecordTrace const& trace); XBT_PRIVATE void replay(std::string trace); } } /** Whether the MC record mode is enabled * * The behaviour is not changed. The only real difference is that * the path is writtent in the log when an interesting path is found. */ #define MC_record_is_active() _sg_do_model_check_record // **** Data conversion #endif SimGrid-3.18/src/mc/mc_snapshot.hpp0000644000175000017500000001663513217757316017553 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SNAPSHOT_HPP #define SIMGRID_MC_SNAPSHOT_HPP #include #include #include #include #include "src/mc/ModelChecker.hpp" #include "src/mc/RegionSnapshot.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_unw.hpp" extern "C" { // ***** Snapshot region XBT_PRIVATE void mc_region_restore_sparse(simgrid::mc::RemoteClient* process, mc_mem_region_t reg); static XBT_ALWAYS_INLINE void* mc_translate_address_region_chunked(uintptr_t addr, mc_mem_region_t region) { auto split = simgrid::mc::mmu::split(addr - region->start().address()); auto pageno = split.first; auto offset = split.second; const void* snapshot_page = region->page_data().page(pageno); return (char*)snapshot_page + offset; } static XBT_ALWAYS_INLINE void* mc_translate_address_region(uintptr_t addr, mc_mem_region_t region, int process_index) { switch (region->storage_type()) { case simgrid::mc::StorageType::Flat: { uintptr_t offset = (uintptr_t)addr - (uintptr_t)region->start().address(); return (void*)((uintptr_t)region->flat_data().get() + offset); } case simgrid::mc::StorageType::Chunked: return mc_translate_address_region_chunked(addr, region); case simgrid::mc::StorageType::Privatized: { xbt_assert(process_index >= 0, "Missing process index for privatized region"); xbt_assert((size_t)process_index < region->privatized_data().size(), "Out of range process index"); simgrid::mc::RegionSnapshot& subregion = region->privatized_data()[process_index]; return mc_translate_address_region(addr, &subregion, process_index); } default: // includes StorageType::NoData xbt_die("Storage type not supported"); } } XBT_PRIVATE mc_mem_region_t mc_get_snapshot_region(const void* addr, const simgrid::mc::Snapshot* snapshot, int process_index); } // ***** MC Snapshot /** Ignored data * * Some parts of the snapshot are ignored by zeroing them out: the real * values is stored here. * */ struct s_mc_snapshot_ignored_data_t { void* start; std::vector data; }; struct s_fd_infos_t { std::string filename; int number; off_t current_position; int flags; }; /** Information about a given stack frame */ struct s_mc_stack_frame_t { /** Instruction pointer */ unw_word_t ip; /** Stack pointer */ unw_word_t sp; unw_word_t frame_base; simgrid::mc::Frame* frame; std::string frame_name; unw_cursor_t unw_cursor; }; typedef s_mc_stack_frame_t* mc_stack_frame_t; struct s_local_variable_t { simgrid::mc::Frame* subprogram; unsigned long ip; std::string name; simgrid::mc::Type* type; void* address; int region; }; typedef s_local_variable_t* local_variable_t; struct XBT_PRIVATE s_mc_snapshot_stack_t { std::vector local_variables; simgrid::mc::UnwindContext context; std::vector stack_frames; int process_index; }; typedef s_mc_snapshot_stack_t* mc_snapshot_stack_t; namespace simgrid { namespace mc { class XBT_PRIVATE Snapshot final : public AddressSpace { public: Snapshot(RemoteClient* process, int num_state); ~Snapshot() = default; const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const override; // To be private int num_state; std::size_t heap_bytes_used; std::vector> snapshot_regions; std::set enabled_processes; int privatization_index; std::vector stack_sizes; std::vector stacks; std::vector to_ignore; std::uint64_t hash; std::vector ignored_data; std::vector current_fds; }; } } extern "C" { static XBT_ALWAYS_INLINE mc_mem_region_t mc_get_region_hinted(void* addr, simgrid::mc::Snapshot* snapshot, int process_index, mc_mem_region_t region) { if (region->contain(simgrid::mc::remote(addr))) return region; else return mc_get_snapshot_region(addr, snapshot, process_index); } static const void* mc_snapshot_get_heap_end(simgrid::mc::Snapshot* snapshot); } namespace simgrid { namespace mc { XBT_PRIVATE std::shared_ptr take_snapshot(int num_state); XBT_PRIVATE void restore_snapshot(std::shared_ptr snapshot); } } extern "C" { XBT_PRIVATE void mc_restore_page_snapshot_region(simgrid::mc::RemoteClient* process, void* start_addr, simgrid::mc::ChunkedData const& pagenos); const void* MC_region_read_fragmented(mc_mem_region_t region, void* target, const void* addr, std::size_t size); int MC_snapshot_region_memcmp(const void* addr1, mc_mem_region_t region1, const void* addr2, mc_mem_region_t region2, std::size_t size); XBT_PRIVATE int MC_snapshot_memcmp(const void* addr1, simgrid::mc::Snapshot* snapshot1, const void* addr2, simgrid::mc::Snapshot* snapshot2, int process_index, std::size_t size); static XBT_ALWAYS_INLINE const void* mc_snapshot_get_heap_end(simgrid::mc::Snapshot* snapshot) { if (snapshot == nullptr) xbt_die("snapshot is nullptr"); return mc_model_checker->process().get_heap()->breakval; } /** @brief Read memory from a snapshot region * * @param addr Process (non-snapshot) address of the data * @param region Snapshot memory region where the data is located * @param target Buffer to store the value * @param size Size of the data to read in bytes * @return Pointer where the data is located (target buffer of original location) */ static XBT_ALWAYS_INLINE const void* MC_region_read(mc_mem_region_t region, void* target, const void* addr, std::size_t size) { xbt_assert(region); std::uintptr_t offset = (std::uintptr_t)addr - (std::uintptr_t)region->start().address(); xbt_assert(region->contain(simgrid::mc::remote(addr)), "Trying to read out of the region boundary."); switch (region->storage_type()) { case simgrid::mc::StorageType::Flat: return (char*)region->flat_data().get() + offset; case simgrid::mc::StorageType::Chunked: { // Last byte of the region: void* end = (char*)addr + size - 1; if (simgrid::mc::mmu::sameChunk((std::uintptr_t)addr, (std::uintptr_t)end)) { // The memory is contained in a single page: return mc_translate_address_region_chunked((uintptr_t)addr, region); } // Otherwise, the memory spans several pages: return MC_region_read_fragmented(region, target, addr, size); } default: // includes StorageType::NoData and StorageType::Privatized (we currently do not pass the process_index to this // function so we assume that the privatized region has been resolved in the callers) xbt_die("Storage type not supported"); } } static XBT_ALWAYS_INLINE void* MC_region_read_pointer(mc_mem_region_t region, const void* addr) { void* res; return *(void**)MC_region_read(region, &res, addr, sizeof(void*)); } } #endif SimGrid-3.18/src/mc/Variable.hpp0000644000175000017500000000231413217757316016747 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_VARIABLE_HPP #define SIMGRID_MC_VARIABLE_HPP #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace mc { /** A variable (global or local) in the model-checked program */ class Variable { public: Variable() = default; std::uint32_t id = 0; bool global = false; std::string name; unsigned type_id = 0; simgrid::mc::Type* type = nullptr; /** Address of the variable (if it is fixed) */ void* address = nullptr; /** Description of the location of the variable (if it's not fixed) */ simgrid::dwarf::LocationList location_list; /** Offset of validity of the variable (DW_AT_start_scope) * * This is an offset from the variable scope beginning. This variable * is only valid starting from this offset. */ std::size_t start_scope = 0; simgrid::mc::ObjectInformation* object_info = nullptr; }; } } #endif SimGrid-3.18/src/mc/mc_hash.cpp0000644000175000017500000000221313217757316016615 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "xbt/log.h" #include "mc/datatypes.h" #include "src/mc/mc_hash.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_snapshot.hpp" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_hash, mc, "Logging specific to mc_hash"); namespace simgrid { namespace mc { namespace { class djb_hash { hash_type state_ = 5381LL; public: template void update(T& x) { state_ = (state_ << 5) + state_ + state_; } hash_type value() { return state_; } }; } hash_type hash(Snapshot const& snapshot) { XBT_DEBUG("START hash %i", snapshot.num_state); djb_hash hash; // TODO, nb_processes // TODO, heap_bytes_used // TODO, root variables // TODO, basic stack frame information // TODO, stack frame local variables XBT_DEBUG("END hash %i", snapshot.num_state); return hash.value(); } } } SimGrid-3.18/src/mc/Session.cpp0000644000175000017500000001125413217757316016643 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "xbt/log.h" #include "xbt/system_error.hpp" #include #include #include #include "src/mc/Session.hpp" #include "src/mc/checker/Checker.hpp" #include "src/mc/mc_private.hpp" #include "src/mc/mc_state.hpp" #include "src/smpi/include/private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_Session, mc, "Model-checker session"); extern std::string _sg_mc_dot_output_file; namespace simgrid { namespace mc { static void setup_child_environment(int socket) { #ifdef __linux__ // Make sure we do not outlive our parent: sigset_t mask; sigemptyset (&mask); if (sigprocmask(SIG_SETMASK, &mask, nullptr) < 0) throw simgrid::xbt::errno_error("Could not unblock signals"); if (prctl(PR_SET_PDEATHSIG, SIGHUP) != 0) throw simgrid::xbt::errno_error("Could not PR_SET_PDEATHSIG"); #endif int res; // Remove CLOEXEC in order to pass the socket to the exec-ed program: int fdflags = fcntl(socket, F_GETFD, 0); if (fdflags == -1 || fcntl(socket, F_SETFD, fdflags & ~FD_CLOEXEC) == -1) throw simgrid::xbt::errno_error("Could not remove CLOEXEC for socket"); // Set environment: setenv(MC_ENV_VARIABLE, "1", 1); // Disable lazy relocation in the model-checked process. // We don't want the model-checked process to modify its .got.plt during // snapshot. setenv("LC_BIND_NOW", "1", 1); char buffer[64]; res = std::snprintf(buffer, sizeof(buffer), "%i", socket); if ((size_t) res >= sizeof(buffer) || res == -1) std::abort(); setenv(MC_ENV_SOCKET_FD, buffer, 1); } /** Execute some code in a forked process */ template static inline pid_t do_fork(F code) { pid_t pid = fork(); if (pid < 0) throw simgrid::xbt::errno_error("Could not fork model-checked process"); if (pid != 0) return pid; // Child-process: try { code(); _exit(EXIT_SUCCESS); } catch(...) { // The callback should catch exceptions: std::terminate(); } } Session::Session(pid_t pid, int socket) { std::unique_ptr process(new simgrid::mc::RemoteClient(pid, socket)); // TODO, automatic detection of the config from the process process->privatized(smpi_privatize_global_variables != SMPI_PRIVATIZE_NONE); modelChecker_ = std::unique_ptr( new simgrid::mc::ModelChecker(std::move(process))); xbt_assert(mc_model_checker == nullptr); mc_model_checker = modelChecker_.get(); mc_model_checker->start(); } Session::~Session() { this->close(); } void Session::initialize() { xbt_assert(initialSnapshot_ == nullptr); mc_model_checker->wait_for_requests(); initialSnapshot_ = simgrid::mc::take_snapshot(0); } void Session::execute(Transition const& transition) { modelChecker_->handle_simcall(transition); modelChecker_->wait_for_requests(); } void Session::restoreInitialState() { simgrid::mc::restore_snapshot(this->initialSnapshot_); } void Session::logState() { mc_model_checker->getChecker()->logState(); if (not _sg_mc_dot_output_file.empty()) { fprintf(dot_output, "}\n"); fclose(dot_output); } if (getenv("SIMGRID_MC_SYSTEM_STATISTICS")){ int ret=system("free"); if(ret!=0)XBT_WARN("system call did not return 0, but %d",ret); } } // static Session* Session::fork(std::function code) { // Create a AF_LOCAL socketpair used for exchanging messages // between the model-checker process (ourselves) and the model-checked // process: int res; int sockets[2]; res = socketpair(AF_LOCAL, SOCK_SEQPACKET | SOCK_CLOEXEC, 0, sockets); if (res == -1) throw simgrid::xbt::errno_error("Could not create socketpair"); pid_t pid = do_fork([sockets, &code] { ::close(sockets[1]); setup_child_environment(sockets[0]); code(); xbt_die("The model-checked process failed to exec()"); }); // Parent (model-checker): ::close(sockets[0]); return new Session(pid, sockets[1]); } // static Session* Session::spawnv(const char *path, char *const argv[]) { return Session::fork([path, argv] { execv(path, argv); }); } // static Session* Session::spawnvp(const char *file, char *const argv[]) { return Session::fork([file, argv] { execvp(file, argv); }); } void Session::close() { initialSnapshot_ = nullptr; if (modelChecker_) { modelChecker_->shutdown(); modelChecker_ = nullptr; mc_model_checker = nullptr; } } simgrid::mc::Session* session; } } SimGrid-3.18/src/mc/DwarfExpression.hpp0000644000175000017500000001025313217757316020346 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_DWARF_EXPRESSION_HPP #define SIMGRID_MC_DWARF_EXPRESSION_HPP #include #include #include // runtime_error #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/AddressSpace.hpp" /** @file DwarfExpression.hpp * * Evaluation of DWARF location expressions. */ namespace simgrid { namespace dwarf { /** A DWARF expression * * DWARF defines a simple stack-based VM for evaluating expressions * (such as locations of variables, etc.): a DWARF expression is * just a sequence of dwarf instructions. We currently directly use * `Dwarf_Op` from `dwarf.h` for dwarf instructions. */ typedef std::vector DwarfExpression; /** Context of evaluation of a DWARF expression * * Some DWARF instructions need to read the CPU registers, * the process memory, etc. All those informations are gathered in * the evaluation context. */ class ExpressionContext { public: ExpressionContext() : cursor(nullptr), frame_base(nullptr), address_space(nullptr), object_info(nullptr), process_index(simgrid::mc::ProcessIndexMissing) {} /** CPU state (registers) */ unw_cursor_t* cursor; void* frame_base; /** Address space used to read memory */ simgrid::mc::AddressSpace* address_space; simgrid::mc::ObjectInformation* object_info; int process_index; }; /** When an error happens in the execution of a DWARF expression */ class evaluation_error : public std::runtime_error { public: explicit evaluation_error(const char* what) : std::runtime_error(what) {} }; /** A stack for evaluating a DWARF expression * * DWARF expressions work by manipulating a stack of integer values. */ class ExpressionStack { public: typedef std::uintptr_t value_type; static const std::size_t max_size = 64; private: // Values of the stack (the top is stack_[size_ - 1]): uintptr_t stack_[max_size] {0}; size_t size_; public: ExpressionStack() : size_(0) {} // Access: std::size_t size() const { return size_; } bool empty() const { return size_ == 0; } void clear() { size_ = 0; } uintptr_t& operator[](int i) { return stack_[i]; } uintptr_t const& operator[](int i) const { return stack_[i]; } /** Top of the stack */ value_type& top() { if (size_ == 0) throw evaluation_error("Empty stack"); return stack_[size_ - 1]; } /** Access the i-th element from the top of the stack */ value_type& top(unsigned i) { if (size_ < i) throw evaluation_error("Invalid element"); return stack_[size_ - 1 - i]; } /** Push a value on the top of the stack */ void push(value_type value) { if (size_ == max_size) throw evaluation_error("DWARF stack overflow"); stack_[size_++] = value; } /* Pop a value from the top of the stack */ value_type pop() { if (size_ == 0) throw evaluation_error("DWARF stack underflow"); return stack_[--size_]; } // These are DWARF operations (DW_OP_foo): /* Push a copy of the top-value (DW_OP_dup) */ void dup() { push(top()); } /* Swap the two top-most values */ void swap() { std::swap(top(), top(1)); } }; /** Executes a DWARF expression * * @param ops DWARF expression instructions * @param n number of instructions * @param context evaluation context (registers, memory, etc.) * @param stack DWARf stack where the operations are executed */ void execute(const Dwarf_Op* ops, std::size_t n, ExpressionContext const& context, ExpressionStack& stack); /** Executes/evaluates a DWARF expression * * @param expression DWARF expression to execute * @param context evaluation context (registers, memory, etc.) * @param stack DWARf stack where the operations are executed */ inline void execute(simgrid::dwarf::DwarfExpression const& expression, ExpressionContext const& context, ExpressionStack& stack) { execute(expression.data(), expression.size(), context, stack); } } } #endif SimGrid-3.18/src/mc/Type.hpp0000644000175000017500000000604313217757316016146 0ustar mquinsonmquinson/* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_TYPE_HPP #define SIMGRID_MC_TYPE_HPP #include #include #include #include "xbt/asserts.h" #include "xbt/base.h" #include #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace mc { /** A member of a structure, union * * Inheritance is seen as a special member as well. */ class Member { public: typedef int flags_type; static constexpr flags_type INHERITANCE_FLAG = 1; static constexpr flags_type VIRTUAL_POINTER_FLAG = 2; Member() = default; /** Whether this member represent some inherited part of the object */ flags_type flags = 0; /** Name of the member (if any) */ std::string name; /** DWARF location expression for locating the location of the member */ simgrid::dwarf::DwarfExpression location_expression; std::size_t byte_size = 0; // Do we really need this? unsigned type_id = 0; simgrid::mc::Type* type = nullptr; bool isInheritance() const { return this->flags & INHERITANCE_FLAG; } bool isVirtualPointer() const { return this->flags & VIRTUAL_POINTER_FLAG; } /** Whether the member is at a fixed offset from the base address */ bool has_offset_location() const { // Recognize the expression `DW_OP_plus_uconst(offset)`: return location_expression.size() == 1 && location_expression[0].atom == DW_OP_plus_uconst; } /** Get the offset of the member * * This is only valid is the member is at a fixed offset from the base. * This is often the case (for C types, C++ type without virtual * inheritance). * * If the location is more complex, the location expression has * to be evaluated (which might need accessing the memory). */ int offset() const { xbt_assert(this->has_offset_location()); return this->location_expression[0].number; } /** Set the location of the member as a fixed offset */ void offset(int new_offset) { // Set the expression to be `DW_OP_plus_uconst(offset)`: Dwarf_Op op; op.atom = DW_OP_plus_uconst; op.number = new_offset; this->location_expression = { op }; } }; /** A type in the model-checked program */ class Type { public: Type() = default; /** The DWARF TAG of the type (e.g. DW_TAG_array_type) */ int type = 0; unsigned id = 0; /* Offset in the section (in hexadecimal form) */ std::string name; /* Name of the type */ int byte_size = 0; /* Size in bytes */ int element_count = 0; /* Number of elements for array type */ unsigned type_id = 0; /* DW_AT_type id */ std::vector members; /* if DW_TAG_structure_type, DW_TAG_class_type, DW_TAG_union_type*/ simgrid::mc::Type* subtype = nullptr; // DW_AT_type simgrid::mc::Type* full_type = nullptr; // The same (but more complete) type }; } } #endif SimGrid-3.18/src/mc/ObjectInformation.cpp0000644000175000017500000001470413217757316020637 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "src/mc/Frame.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" namespace simgrid { namespace mc { /* For an executable object, addresses are virtual address * (there is no offset) i.e. * \f$\text{virtual address} = \{dwarf address}\f$ * * For a shared object, the addreses are offset from the begining * of the shared object (the base address of the mapped shared * object must be used as offset * i.e. \f$\text{virtual address} = \text{shared object base address} * + \text{dwarf address}\f$. */ void *ObjectInformation::base_address() const { // For an executable (more precisely for a ET_EXEC) the base it 0: if (this->executable()) return nullptr; // For an a shared-object (ET_DYN, including position-independant executables) // the base address is its lowest address: void *result = this->start_exec; if (this->start_rw != nullptr && result > (void *) this->start_rw) result = this->start_rw; if (this->start_ro != nullptr && result > (void *) this->start_ro) result = this->start_ro; return result; } simgrid::mc::Frame* ObjectInformation::find_function(const void *ip) const { /* This is implemented by binary search on a sorted array. * * We do quite a lot of those so we want this to be cache efficient. * We pack the only information we need in the index entries in order * to successfully do the binary search. We do not need the high_pc * during the binary search (only at the end) so it is not included * in the index entry. We could use parallel arrays as well. * * We cannot really use the std:: algorithm for this. * We could use std::binary_search by including the high_pc inside * the FunctionIndexEntry. */ const simgrid::mc::FunctionIndexEntry* base = this->functions_index.data(); int i = 0; int j = this->functions_index.size() - 1; while (j >= i) { int k = i + ((j - i) / 2); /* In most of the search, we do not dereference the base[k].function. * This way the memory accesses are located in the base[k] array. */ if (ip < base[k].low_pc) j = k - 1; else if (k < j && ip >= base[k + 1].low_pc) i = k + 1; /* At this point, the search is over. * Either we have found the correct function or we do not know * any function corresponding to this instruction address. * Only at the point do we dereference the function pointer. */ else if ((std::uint64_t) ip < base[k].function->range.end()) return base[k].function; else return nullptr; } return nullptr; } simgrid::mc::Variable* ObjectInformation::find_variable(const char* name) const { for (simgrid::mc::Variable& variable : this->global_variables) if(variable.name == name) return &variable; return nullptr; } void ObjectInformation::remove_global_variable(const char* name) { typedef std::vector::size_type size_type; if (this->global_variables.empty()) return; // Binary search: size_type first = 0; size_type last = this->global_variables.size() - 1; while (first <= last) { size_type cursor = first + (last - first) / 2; simgrid::mc::Variable& current_var = this->global_variables[cursor]; int cmp = current_var.name.compare(name); if (cmp == 0) { // Find the whole range: size_type first = cursor; while (first != 0 && this->global_variables[first - 1].name == name) first--; size_type size = this->global_variables.size(); size_type last = cursor; while (last != size - 1 && this->global_variables[last + 1].name == name) last++; // Remove the whole range: this->global_variables.erase( this->global_variables.begin() + first, this->global_variables.begin() + last + 1); return; } else if (cmp < 0) first = cursor + 1; else if (cursor != 0) last = cursor - 1; else break; } } /** Ignore a local variable in a scope * * Ignore all instances of variables with a given name in * any (possibly inlined) subprogram with a given namespaced * name. * * @param var_name Name of the local variable to ignore * @param subprogram_name Name of the subprogram to ignore (nullptr for any) * @param subprogram (possibly inlined) Subprogram of the scope current scope * @param scope Current scope */ static void remove_local_variable(simgrid::mc::Frame& scope, const char *var_name, const char *subprogram_name, simgrid::mc::Frame const& subprogram) { typedef std::vector::size_type size_type; // If the current subprogram matches the given name: if ((subprogram_name == nullptr || (not subprogram.name.empty() && subprogram.name == subprogram_name)) && not scope.variables.empty()) { // Try to find the variable and remove it: size_type start = 0; size_type end = scope.variables.size() - 1; // Binary search: while (start <= end) { size_type cursor = start + (end - start) / 2; simgrid::mc::Variable& current_var = scope.variables[cursor]; int compare = current_var.name.compare(var_name); if (compare == 0) { // Variable found, remove it: scope.variables.erase(scope.variables.begin() + cursor); break; } else if (compare < 0) start = cursor + 1; else if (cursor != 0) end = cursor - 1; else break; } } // And recursive processing in nested scopes: for (simgrid::mc::Frame& nested_scope : scope.scopes) { // The new scope may be an inlined subroutine, in this case we want to use its // namespaced name in recursive calls: simgrid::mc::Frame const& nested_subprogram = nested_scope.tag == DW_TAG_inlined_subroutine ? nested_scope : subprogram; remove_local_variable(nested_scope, var_name, subprogram_name, nested_subprogram); } } void ObjectInformation::remove_local_variable( const char* var_name, const char* subprogram_name) { for (auto& entry : this->subprograms) simgrid::mc::remove_local_variable(entry.second, var_name, subprogram_name, entry.second); } } } SimGrid-3.18/src/mc/mc_xbt.cpp0000644000175000017500000000214413217757316016472 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "src/mc/AddressSpace.hpp" #include "src/mc/mc_xbt.hpp" #include "src/mc/remote/RemotePtr.hpp" #include "xbt/dynar.h" #include "xbt/sysdep.h" namespace simgrid { namespace mc { void read_element(AddressSpace const& as, void* local, RemotePtr addr, std::size_t i, std::size_t len) { s_xbt_dynar_t d; as.read_bytes(&d, sizeof(d), addr); if (i >= d.used) xbt_die("Out of bound index %zu/%lu", i, d.used); if (len != d.elmsize) xbt_die("Bad size in simgrid::mc::read_element"); as.read_bytes(local, len, remote(xbt_dynar_get_ptr(&d, i))); } std::size_t read_length(AddressSpace const& as, RemotePtr addr) { if (not addr) return 0; unsigned long res; as.read_bytes(&res, sizeof(res), remote(&((xbt_dynar_t)addr.address())->used)); return res; } } } SimGrid-3.18/src/mc/mc_request.cpp0000644000175000017500000004635213217757316017376 0ustar mquinsonmquinson/* Copyright (c) 2008-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "src/include/mc/mc.h" #include "src/mc/ModelChecker.hpp" #include "src/mc/mc_request.hpp" #include "src/mc/mc_smx.hpp" #include "src/mc/mc_xbt.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_request, mc, "Logging specific to MC (request)"); static char *pointer_to_string(void *pointer); static char *buff_size_to_string(size_t size); static inline simgrid::kernel::activity::CommImpl* MC_get_comm(smx_simcall_t r) { switch (r->call ) { case SIMCALL_COMM_WAIT: return static_cast(simcall_comm_wait__getraw__comm(r)); case SIMCALL_COMM_TEST: return static_cast(simcall_comm_test__getraw__comm(r)); default: return nullptr; } } static inline smx_mailbox_t MC_get_mbox(smx_simcall_t r) { switch(r->call) { case SIMCALL_COMM_ISEND: return simcall_comm_isend__get__mbox(r); case SIMCALL_COMM_IRECV: return simcall_comm_irecv__get__mbox(r); default: return nullptr; } } namespace simgrid { namespace mc { // Does half the job static inline bool request_depend_asymmetric(smx_simcall_t r1, smx_simcall_t r2) { if (r1->call == SIMCALL_COMM_ISEND && r2->call == SIMCALL_COMM_IRECV) return false; if (r1->call == SIMCALL_COMM_IRECV && r2->call == SIMCALL_COMM_ISEND) return false; // Those are internal requests, we do not need indirection // because those objects are copies: simgrid::kernel::activity::CommImpl* synchro1 = MC_get_comm(r1); simgrid::kernel::activity::CommImpl* synchro2 = MC_get_comm(r2); if ((r1->call == SIMCALL_COMM_ISEND || r1->call == SIMCALL_COMM_IRECV) && r2->call == SIMCALL_COMM_WAIT) { smx_mailbox_t mbox = MC_get_mbox(r1); if (mbox != synchro2->mbox_cpy && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->issuer != synchro2->src_proc) && (r1->issuer != synchro2->dst_proc) && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->call == SIMCALL_COMM_ISEND) && (synchro2->type == SIMIX_COMM_SEND) && (synchro2->src_buff != simcall_comm_isend__get__src_buff(r1)) && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->call == SIMCALL_COMM_IRECV) && (synchro2->type == SIMIX_COMM_RECEIVE) && (synchro2->dst_buff != simcall_comm_irecv__get__dst_buff(r1)) && simcall_comm_wait__get__timeout(r2) <= 0) return false; } /* FIXME: the following rule assumes that the result of the * isend/irecv call is not stored in a buffer used in the * test call. */ #if 0 if((r1->call == SIMCALL_COMM_ISEND || r1->call == SIMCALL_COMM_IRECV) && r2->call == SIMCALL_COMM_TEST) return FALSE; #endif if (r1->call == SIMCALL_COMM_WAIT && (r2->call == SIMCALL_COMM_WAIT || r2->call == SIMCALL_COMM_TEST) && (synchro1->src_proc == nullptr || synchro1->dst_proc == nullptr)) return false; if (r1->call == SIMCALL_COMM_TEST && (simcall_comm_test__get__comm(r1) == nullptr || synchro1->src_buff == nullptr || synchro1->dst_buff == nullptr)) return false; if (r1->call == SIMCALL_COMM_TEST && r2->call == SIMCALL_COMM_WAIT && synchro1->src_buff == synchro2->src_buff && synchro1->dst_buff == synchro2->dst_buff) return false; if (r1->call == SIMCALL_COMM_WAIT && r2->call == SIMCALL_COMM_TEST && synchro1->src_buff != nullptr && synchro1->dst_buff != nullptr && synchro2->src_buff != nullptr && synchro2->dst_buff != nullptr && synchro1->dst_buff != synchro2->src_buff && synchro1->dst_buff != synchro2->dst_buff && synchro2->dst_buff != synchro1->src_buff) return false; return true; } // Those are internal_req bool request_depend(smx_simcall_t r1, smx_simcall_t r2) { if (r1->issuer == r2->issuer) return false; /* Wait with timeout transitions are not considered by the independence theorem, thus we consider them as dependent with all other transitions */ if ((r1->call == SIMCALL_COMM_WAIT && simcall_comm_wait__get__timeout(r1) > 0) || (r2->call == SIMCALL_COMM_WAIT && simcall_comm_wait__get__timeout(r2) > 0)) return TRUE; if (r1->call != r2->call) return request_depend_asymmetric(r1, r2) && request_depend_asymmetric(r2, r1); // Those are internal requests, we do not need indirection // because those objects are copies: simgrid::kernel::activity::CommImpl* synchro1 = MC_get_comm(r1); simgrid::kernel::activity::CommImpl* synchro2 = MC_get_comm(r2); switch(r1->call) { case SIMCALL_COMM_ISEND: return simcall_comm_isend__get__mbox(r1) == simcall_comm_isend__get__mbox(r2); case SIMCALL_COMM_IRECV: return simcall_comm_irecv__get__mbox(r1) == simcall_comm_irecv__get__mbox(r2); case SIMCALL_COMM_WAIT: if (synchro1->src_buff == synchro2->src_buff && synchro1->dst_buff == synchro2->dst_buff) return false; if (synchro1->src_buff != nullptr && synchro1->dst_buff != nullptr && synchro2->src_buff != nullptr && synchro2->dst_buff != nullptr && synchro1->dst_buff != synchro2->src_buff && synchro1->dst_buff != synchro2->dst_buff && synchro2->dst_buff != synchro1->src_buff) return false; return true; default: return true; } } } } static char *pointer_to_string(void *pointer) { if (XBT_LOG_ISENABLED(mc_request, xbt_log_priority_verbose)) return bprintf("%p", pointer); return xbt_strdup("(verbose only)"); } static char *buff_size_to_string(size_t buff_size) { if (XBT_LOG_ISENABLED(mc_request, xbt_log_priority_verbose)) return bprintf("%zu", buff_size); return xbt_strdup("(verbose only)"); } std::string simgrid::mc::request_to_string(smx_simcall_t req, int value, simgrid::mc::RequestType request_type) { xbt_assert(mc_model_checker != nullptr, "Must be called from MCer"); bool use_remote_comm = true; switch(request_type) { case simgrid::mc::RequestType::simix: use_remote_comm = true; break; case simgrid::mc::RequestType::executed: case simgrid::mc::RequestType::internal: use_remote_comm = false; break; default: THROW_IMPOSSIBLE; } const char* type = nullptr; char *args = nullptr; smx_actor_t issuer = MC_smx_simcall_get_issuer(req); switch (req->call) { case SIMCALL_COMM_ISEND: { type = "iSend"; char* p = pointer_to_string(simcall_comm_isend__get__src_buff(req)); char* bs = buff_size_to_string(simcall_comm_isend__get__src_buff_size(req)); if (issuer->host) args = bprintf("src=(%lu)%s (%s), buff=%s, size=%s", issuer->pid, MC_smx_actor_get_host_name(issuer), MC_smx_actor_get_name(issuer), p, bs); else args = bprintf("src=(%lu)%s, buff=%s, size=%s", issuer->pid, MC_smx_actor_get_name(issuer), p, bs); xbt_free(bs); xbt_free(p); break; } case SIMCALL_COMM_IRECV: { size_t* remote_size = simcall_comm_irecv__get__dst_buff_size(req); size_t size = 0; if (remote_size) mc_model_checker->process().read_bytes(&size, sizeof(size), remote(remote_size)); type = "iRecv"; char* p = pointer_to_string(simcall_comm_irecv__get__dst_buff(req)); char* bs = buff_size_to_string(size); if (issuer->host) args = bprintf("dst=(%lu)%s (%s), buff=%s, size=%s", issuer->pid, MC_smx_actor_get_host_name(issuer), MC_smx_actor_get_name(issuer), p, bs); else args = bprintf("dst=(%lu)%s, buff=%s, size=%s", issuer->pid, MC_smx_actor_get_name(issuer), p, bs); xbt_free(bs); xbt_free(p); break; } case SIMCALL_COMM_WAIT: { simgrid::kernel::activity::CommImpl* remote_act = static_cast(simcall_comm_wait__getraw__comm(req)); char* p; if (value == -1) { type = "WaitTimeout"; p = pointer_to_string(remote_act); args = bprintf("comm=%s", p); } else { type = "Wait"; p = pointer_to_string(remote_act); simgrid::mc::Remote temp_synchro; simgrid::kernel::activity::CommImpl* act; if (use_remote_comm) { mc_model_checker->process().read(temp_synchro, remote(static_cast(remote_act))); act = temp_synchro.getBuffer(); } else act = remote_act; smx_actor_t src_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(act->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(act->dst_proc)); args = bprintf("comm=%s [(%lu)%s (%s)-> (%lu)%s (%s)]", p, src_proc ? src_proc->pid : 0, src_proc ? MC_smx_actor_get_host_name(src_proc) : "", src_proc ? MC_smx_actor_get_name(src_proc) : "", dst_proc ? dst_proc->pid : 0, dst_proc ? MC_smx_actor_get_host_name(dst_proc) : "", dst_proc ? MC_smx_actor_get_name(dst_proc) : ""); } xbt_free(p); break; } case SIMCALL_COMM_TEST: { simgrid::kernel::activity::CommImpl* remote_act = static_cast(simcall_comm_test__getraw__comm(req)); simgrid::mc::Remote temp_synchro; simgrid::kernel::activity::CommImpl* act; if (use_remote_comm) { mc_model_checker->process().read(temp_synchro, remote(static_cast(remote_act))); act = temp_synchro.getBuffer(); } else act = remote_act; char* p; if (act->src_proc == nullptr || act->dst_proc == nullptr) { type = "Test FALSE"; p = pointer_to_string(remote_act); args = bprintf("comm=%s", p); } else { type = "Test TRUE"; p = pointer_to_string(remote_act); smx_actor_t src_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(act->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(act->dst_proc)); args = bprintf("comm=%s [(%lu)%s (%s) -> (%lu)%s (%s)]", p, src_proc->pid, MC_smx_actor_get_name(src_proc), MC_smx_actor_get_host_name(src_proc), dst_proc->pid, MC_smx_actor_get_name(dst_proc), MC_smx_actor_get_host_name(dst_proc)); } xbt_free(p); break; } case SIMCALL_COMM_WAITANY: { type = "WaitAny"; s_xbt_dynar_t comms; mc_model_checker->process().read_bytes( &comms, sizeof(comms), remote(simcall_comm_waitany__get__comms(req))); if (not xbt_dynar_is_empty(&comms)) { smx_activity_t remote_sync; read_element(mc_model_checker->process(), &remote_sync, remote(simcall_comm_waitany__get__comms(req)), value, sizeof(remote_sync)); char* p = pointer_to_string(remote_sync.get()); args = bprintf("comm=%s (%d of %lu)", p, value + 1, xbt_dynar_length(&comms)); xbt_free(p); } else args = bprintf("comm at idx %d", value); break; } case SIMCALL_COMM_TESTANY: if (value == -1) { type = "TestAny FALSE"; args = xbt_strdup("-"); } else { type = "TestAny"; args = bprintf("(%d of %zu)", value + 1, simcall_comm_testany__get__count(req)); } break; case SIMCALL_MUTEX_TRYLOCK: case SIMCALL_MUTEX_LOCK: { if (req->call == SIMCALL_MUTEX_LOCK) type = "Mutex LOCK"; else type = "Mutex TRYLOCK"; simgrid::mc::Remote mutex; mc_model_checker->process().read_bytes(mutex.getBuffer(), sizeof(mutex), remote( req->call == SIMCALL_MUTEX_LOCK ? simcall_mutex_lock__get__mutex(req) : simcall_mutex_trylock__get__mutex(req) )); args = bprintf("locked = %d, owner = %d, sleeping = n/a", mutex.getBuffer()->locked, mutex.getBuffer()->owner != nullptr ? (int)mc_model_checker->process().resolveActor(simgrid::mc::remote(mutex.getBuffer()->owner))->pid : -1); break; } case SIMCALL_MC_RANDOM: type = "MC_RANDOM"; args = bprintf("%d", value); break; default: type = SIMIX_simcall_name(req->call); args = bprintf("??"); break; } std::string str; if (args != nullptr) str = simgrid::xbt::string_printf("[(%lu)%s (%s)] %s(%s)", issuer->pid, MC_smx_actor_get_host_name(issuer), MC_smx_actor_get_name(issuer), type, args); else str = simgrid::xbt::string_printf("[(%lu)%s (%s)] %s ", issuer->pid, MC_smx_actor_get_host_name(issuer), MC_smx_actor_get_name(issuer), type); xbt_free(args); return str; } namespace simgrid { namespace mc { bool request_is_enabled_by_idx(smx_simcall_t req, unsigned int idx) { simgrid::kernel::activity::ActivityImpl* remote_act = nullptr; switch (req->call) { case SIMCALL_COMM_WAIT: /* FIXME: check also that src and dst processes are not suspended */ remote_act = simcall_comm_wait__getraw__comm(req); break; case SIMCALL_COMM_WAITANY: read_element(mc_model_checker->process(), &remote_act, remote(simcall_comm_waitany__getraw__comms(req)), idx, sizeof(remote_act)); break; case SIMCALL_COMM_TESTANY: remote_act = mc_model_checker->process().read(remote(simcall_comm_testany__getraw__comms(req) + idx)); break; default: return true; } simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote(static_cast(remote_act))); simgrid::kernel::activity::CommImpl* comm = temp_comm.getBuffer(); return comm->src_proc && comm->dst_proc; } static const char* colors[] = { "blue", "red", "green3", "goldenrod", "brown", "purple", "magenta", "turquoise4", "gray25", "forestgreen", "hotpink", "lightblue", "tan", }; static inline const char* get_color(int id) { return colors[id % (sizeof(colors) / sizeof(colors[0])) ]; } std::string request_get_dot_output(smx_simcall_t req, int value) { std::string label; const smx_actor_t issuer = MC_smx_simcall_get_issuer(req); switch (req->call) { case SIMCALL_COMM_ISEND: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] iSend", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = bprintf("[(%lu)] iSend", issuer->pid); break; case SIMCALL_COMM_IRECV: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] iRecv", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] iRecv", issuer->pid); break; case SIMCALL_COMM_WAIT: if (value == -1) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] WaitTimeout", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] WaitTimeout", issuer->pid); } else { simgrid::kernel::activity::ActivityImpl* remote_act = simcall_comm_wait__getraw__comm(req); simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote(static_cast(remote_act))); simgrid::kernel::activity::CommImpl* comm = temp_comm.getBuffer(); smx_actor_t src_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(comm->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveActor(simgrid::mc::remote(comm->dst_proc)); if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Wait [(%lu)->(%lu)]", issuer->pid, MC_smx_actor_get_host_name(issuer), src_proc ? src_proc->pid : 0, dst_proc ? dst_proc->pid : 0); else label = simgrid::xbt::string_printf("[(%lu)] Wait [(%lu)->(%lu)]", issuer->pid, src_proc ? src_proc->pid : 0, dst_proc ? dst_proc->pid : 0); } break; case SIMCALL_COMM_TEST: { simgrid::kernel::activity::ActivityImpl* remote_act = simcall_comm_test__getraw__comm(req); simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote(static_cast(remote_act))); simgrid::kernel::activity::CommImpl* comm = temp_comm.getBuffer(); if (comm->src_proc == nullptr || comm->dst_proc == nullptr) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Test FALSE", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = bprintf("[(%lu)] Test FALSE", issuer->pid); } else { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Test TRUE", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] Test TRUE", issuer->pid); } break; } case SIMCALL_COMM_WAITANY: { unsigned long comms_size = read_length( mc_model_checker->process(), remote(simcall_comm_waitany__get__comms(req))); if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] WaitAny [%d of %lu]", issuer->pid, MC_smx_actor_get_host_name(issuer), value + 1, comms_size); else label = simgrid::xbt::string_printf("[(%lu)] WaitAny [%d of %lu]", issuer->pid, value + 1, comms_size); break; } case SIMCALL_COMM_TESTANY: if (value == -1) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] TestAny FALSE", issuer->pid, MC_smx_actor_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] TestAny FALSE", issuer->pid); } else { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] TestAny TRUE [%d of %lu]", issuer->pid, MC_smx_actor_get_host_name(issuer), value + 1, simcall_comm_testany__get__count(req)); else label = simgrid::xbt::string_printf("[(%lu)] TestAny TRUE [%d of %lu]", issuer->pid, value + 1, simcall_comm_testany__get__count(req)); } break; case SIMCALL_MUTEX_TRYLOCK: label = simgrid::xbt::string_printf("[(%lu)] Mutex TRYLOCK", issuer->pid); break; case SIMCALL_MUTEX_LOCK: label = simgrid::xbt::string_printf("[(%lu)] Mutex LOCK", issuer->pid); break; case SIMCALL_MC_RANDOM: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] MC_RANDOM (%d)", issuer->pid, MC_smx_actor_get_host_name(issuer), value); else label = simgrid::xbt::string_printf("[(%lu)] MC_RANDOM (%d)", issuer->pid, value); break; default: THROW_UNIMPLEMENTED; } const char* color = get_color(issuer->pid - 1); return simgrid::xbt::string_printf( "label = \"%s\", color = %s, fontcolor = %s", label.c_str(), color, color); } } } SimGrid-3.18/src/mc/AddressSpace.hpp0000644000175000017500000001141013217757316017560 0ustar mquinsonmquinson/* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_ADDRESS_SPACE_H #define SIMGRID_MC_ADDRESS_SPACE_H #include #include #include #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/remote/RemotePtr.hpp" namespace simgrid { namespace mc { /** Process index used when no process is available (SMPI privatization) * * The expected behavior is that if a process index is needed it will fail. * */ const int ProcessIndexMissing = -1; /** Process index used when we don't care about the process index (SMPI privatization) * */ const int ProcessIndexDisabled = -2; /** Constant used when any process will do (SMPI privatization) * * Note: This is is index of the first process. */ const int ProcessIndexAny = 0; /** Options for read operations * * This is a set of flags managed with bitwise operators. Only the * meaningful operations are defined: addition, conversions to/from * integers are not allowed. */ class ReadOptions { std::uint32_t value_; constexpr explicit ReadOptions(std::uint32_t value) : value_(value) {} public: constexpr ReadOptions() : value_(0) {} explicit constexpr operator bool() const { return value_ != 0; } constexpr bool operator!() const { return value_ == 0; } constexpr ReadOptions operator|(ReadOptions const& that) const { return ReadOptions(value_ | that.value_); } constexpr ReadOptions operator&(ReadOptions const& that) const { return ReadOptions(value_ & that.value_); } constexpr ReadOptions operator^(ReadOptions const& that) const { return ReadOptions(value_ ^ that.value_); } constexpr ReadOptions operator~() const { return ReadOptions(~value_); } ReadOptions& operator|=(ReadOptions const& that) { value_ |= that.value_; return *this; } ReadOptions& operator&=(ReadOptions const& that) { value_ &= that.value_; return *this; } ReadOptions& operator^=(ReadOptions const& that) { value_ &= that.value_; return *this; } /** Copy the data to the given buffer */ static constexpr ReadOptions none() { return ReadOptions(0); } /** Allows to return a pointer to another buffer where the data is * available instead of copying the data into the buffer */ static constexpr ReadOptions lazy() { return ReadOptions(1); } }; /** A given state of a given process (abstract base class) * * Currently, this might either be: * * * the current state of an existing process; * * * a snapshot. * * In order to support SMPI privatization, the can read the memory from the * context of a given SMPI process: if specified, the code reads data from the * correct SMPI privatization VMA. */ class AddressSpace { private: RemoteClient* process_; public: explicit AddressSpace(RemoteClient* process) : process_(process) {} virtual ~AddressSpace() = default; /** The process of this address space * * This is where we can get debug informations, memory layout, etc. */ simgrid::mc::RemoteClient* process() const { return process_; } /** Read data from the address space * * @param buffer target buffer for the data * @param size number of bytes to read * @param address remote source address of the data * @param process_index which process (used for SMPI privatization) * @param options */ virtual const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const = 0; /** Read a given data structure from the address space */ template inline void read(T *buffer, RemotePtr ptr, int process_index = ProcessIndexAny) const { this->read_bytes(buffer, sizeof(T), ptr, process_index); } template inline void read(Remote& buffer, RemotePtr ptr, int process_index = ProcessIndexAny) const { this->read_bytes(buffer.getBuffer(), sizeof(T), ptr, process_index); } /** Read a given data structure from the addres space * * This version returns by value. */ template inline Remote read(RemotePtr ptr, int process_index = ProcessIndexMissing) const { Remote res; this->read_bytes(&res, sizeof(T), ptr, process_index); return res; } /** Read a string of known size */ std::string read_string(RemotePtr address, std::size_t len) const { std::string res; res.resize(len); this->read_bytes(&res[0], len, address); return res; } }; } } #endif SimGrid-3.18/src/simdag/0000755000175000017500000000000013217757321015352 5ustar mquinsonmquinsonSimGrid-3.18/src/simdag/sd_dotloader.cpp0000644000175000017500000002312213217757316020525 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simdag_private.hpp" #include "simgrid/simdag.h" #include "src/internal_config.h" #include "xbt/file.hpp" #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_dotparse, sd, "Parsing DOT files"); #if HAVE_GRAPHVIZ #include xbt_dynar_t SD_dotload_generic(const char* filename, bool sequential, bool schedule); static void dot_task_p_free(void *task) { SD_task_destroy(*(SD_task_t *)task); } /** @brief loads a DOT file describing a DAG * * See http://www.graphviz.org/doc/info/lang.html for more details. * The size attribute of a node describes: * - for a compute task: the amount of flops to execute * - for a communication task : the amount of bytes to transfer * If this attribute is ommited, the default value is zero. */ xbt_dynar_t SD_dotload(const char *filename) { return SD_dotload_generic(filename, true, false); } xbt_dynar_t SD_PTG_dotload(const char * filename) { return SD_dotload_generic(filename, false, false); } xbt_dynar_t SD_dotload_with_sched(const char *filename) { return SD_dotload_generic(filename, true, true); } static int edge_compare(const void *a, const void *b) { unsigned va = AGSEQ(*(Agedge_t **)a); unsigned vb = AGSEQ(*(Agedge_t **)b); if (va == vb) return 0; else return (va < vb ? -1 : 1); } xbt_dynar_t SD_dotload_generic(const char* filename, bool sequential, bool schedule) { xbt_assert(filename, "Unable to use a null file descriptor\n"); FILE *in_file = fopen(filename, "r"); xbt_assert(in_file != nullptr, "Failed to open file: %s", filename); unsigned int i; SD_task_t root; SD_task_t end; SD_task_t task; std::vector* computer; std::unordered_map*> computers; bool schedule_success = true; std::unordered_map jobs; xbt_dynar_t result = xbt_dynar_new(sizeof(SD_task_t), dot_task_p_free); Agraph_t * dag_dot = agread(in_file, NIL(Agdisc_t *)); /* Create all the nodes */ Agnode_t *node = nullptr; for (node = agfstnode(dag_dot); node; node = agnxtnode(dag_dot, node)) { char *name = agnameof(node); double amount = atof(agget(node, (char*)"size")); if (jobs.find(name) == jobs.end()) { if (sequential) { XBT_DEBUG("See ", name, amount); task = SD_task_create_comp_seq(name, nullptr , amount); } else { double alpha = atof(agget(node, (char *) "alpha")); XBT_DEBUG("See ", name, amount, alpha); task = SD_task_create_comp_par_amdahl(name, nullptr , amount, alpha); } jobs.insert({std::string(name), task}); if (strcmp(name,"root") && strcmp(name,"end")) xbt_dynar_push(result, &task); if ((sequential) && ((schedule && schedule_success) || XBT_LOG_ISENABLED(sd_dotparse, xbt_log_priority_verbose))) { /* try to take the information to schedule the task only if all is right*/ char *char_performer = agget(node, (char *) "performer"); char *char_order = agget(node, (char *) "order"); /* Tasks will execute on in a given "order" on a given set of "performer" hosts */ int performer = ((not char_performer || not strcmp(char_performer, "")) ? -1 : atoi(char_performer)); int order = ((not char_order || not strcmp(char_order, "")) ? -1 : atoi(char_order)); if ((performer != -1 && order != -1) && performer < static_cast(sg_host_count())) { /* required parameters are given and less performers than hosts are required */ XBT_DEBUG ("Task '%s' is scheduled on workstation '%d' in position '%d'", task->name, performer, order); auto comp = computers.find(char_performer); if (comp != computers.end()) { computer = comp->second; } else { computer = new std::vector; computers.insert({char_performer, computer}); } if (static_cast(order) < computer->size()) { SD_task_t task_test = computer->at(order); if (task_test && task_test != task) { /* the user gave the same order to several tasks */ schedule_success = false; XBT_VERB("Task '%s' wants to start on performer '%s' at the same position '%s' as task '%s'", task_test->name, char_performer, char_order, task->name); continue; } } else computer->resize(order); computer->insert(computer->begin() + order, task); } else { /* one of required parameters is not given */ schedule_success = false; XBT_VERB("The schedule is ignored, task '%s' can not be scheduled on %d hosts", task->name, performer); } } } else { XBT_WARN("Task '%s' is defined more than once", name); } } /*Check if 'root' and 'end' nodes have been explicitly declared. If not, create them. */ if (jobs.find("root") == jobs.end()) root = (sequential ? SD_task_create_comp_seq("root", nullptr, 0) : SD_task_create_comp_par_amdahl("root", nullptr, 0, 0)); else root = jobs.at("root"); SD_task_set_state(root, SD_SCHEDULABLE); /* by design the root task is always SCHEDULABLE */ xbt_dynar_insert_at(result, 0, &root); /* Put it at the beginning of the dynar */ if (jobs.find("end") == jobs.end()) end = (sequential ? SD_task_create_comp_seq("end", nullptr, 0) : SD_task_create_comp_par_amdahl("end", nullptr, 0, 0)); else end = jobs.at("end"); /* Create edges */ xbt_dynar_t edges = xbt_dynar_new(sizeof(Agedge_t*), nullptr); for (node = agfstnode(dag_dot); node; node = agnxtnode(dag_dot, node)) { Agedge_t * edge; xbt_dynar_reset(edges); for (edge = agfstout(dag_dot, node); edge; edge = agnxtout(dag_dot, edge)) xbt_dynar_push_as(edges, Agedge_t *, edge); /* Be sure edges are sorted */ xbt_dynar_sort(edges, edge_compare); xbt_dynar_foreach(edges, i, edge) { char *src_name=agnameof(agtail(edge)); char *dst_name=agnameof(aghead(edge)); double size = atof(agget(edge, (char *) "size")); SD_task_t src = jobs.at(src_name); SD_task_t dst = jobs.at(dst_name); if (size > 0) { std::string name = std::string(src_name) + "->" + dst_name; XBT_DEBUG("See ", name.c_str(), size); if (jobs.find(name) == jobs.end()) { if (sequential) task = SD_task_create_comm_e2e(name.c_str(), nullptr, size); else task = SD_task_create_comm_par_mxn_1d_block(name.c_str(), nullptr, size); SD_task_dependency_add(nullptr, nullptr, src, task); SD_task_dependency_add(nullptr, nullptr, task, dst); jobs.insert({name, task}); xbt_dynar_push(result, &task); } else { XBT_WARN("Task '%s' is defined more than once", name.c_str()); } } else { SD_task_dependency_add(nullptr, nullptr, src, dst); } } } xbt_dynar_free(&edges); XBT_DEBUG("All tasks have been created, put %s at the end of the dynar", end->name); xbt_dynar_push(result, &end); /* Connect entry tasks to 'root', and exit tasks to 'end'*/ xbt_dynar_foreach (result, i, task){ if (task->predecessors->empty() && task->inputs->empty() && task != root) { XBT_DEBUG("Task '%s' has no source. Add dependency from 'root'", task->name); SD_task_dependency_add(nullptr, nullptr, root, task); } if (task->successors->empty() && task->outputs->empty() && task != end) { XBT_DEBUG("Task '%s' has no destination. Add dependency to 'end'", task->name); SD_task_dependency_add(nullptr, nullptr, task, end); } } agclose(dag_dot); fclose(in_file); if(schedule){ if (schedule_success) { sg_host_t* workstations = sg_host_list(); for (auto const& elm : computers) { SD_task_t previous_task = nullptr; for (auto const& task : *elm.second) { /* add dependency between the previous and the task to avoid parallel execution */ if(task){ if (previous_task && not SD_task_dependency_exists(previous_task, task)) SD_task_dependency_add(nullptr, nullptr, previous_task, task); SD_task_schedulel(task, 1, workstations[atoi(elm.first.c_str())]); previous_task = task; } } delete elm.second; } xbt_free(workstations); } else { XBT_WARN("The scheduling is ignored"); for (auto const& elm : computers) delete elm.second; xbt_dynar_free(&result); result = nullptr; } } if (result && not acyclic_graph_detail(result)) { std::string base = simgrid::xbt::Path(filename).getBasename(); XBT_ERROR("The DOT described in %s is not a DAG. It contains a cycle.", base.c_str()); xbt_dynar_free(&result); result = nullptr; } return result; } #else xbt_dynar_t SD_dotload(const char *filename) { xbt_die("SD_dotload_generic() is not usable because graphviz was not found.\n" "Please install graphviz, graphviz-dev, and libgraphviz-dev (and erase CMakeCache.txt) before recompiling."); } xbt_dynar_t SD_dotload_with_sched(const char *filename) { return SD_dotload(filename); } xbt_dynar_t SD_PTG_dotload(const char * filename) { return SD_dotload(filename); } #endif SimGrid-3.18/src/simdag/dax_dtd.c0000644000175000017500000040023113217757321017125 0ustar mquinsonmquinson #define YY_INT_ALIGNED short int /* A lexical scanner generated by flex */ #define yy_create_buffer dax__create_buffer #define yy_delete_buffer dax__delete_buffer #define yy_flex_debug dax__flex_debug #define yy_init_buffer dax__init_buffer #define yy_flush_buffer dax__flush_buffer #define yy_load_buffer_state dax__load_buffer_state #define yy_switch_to_buffer dax__switch_to_buffer #define yyin dax_in #define yyleng dax_leng #define yylex dax_lex #define yylineno dax_lineno #define yyout dax_out #define yyrestart dax_restart #define yytext dax_text #define yywrap dax_wrap #define yyalloc dax_alloc #define yyrealloc dax_realloc #define yyfree dax_free #define FLEX_SCANNER #define YY_FLEX_MAJOR_VERSION 2 #define YY_FLEX_MINOR_VERSION 6 #define YY_FLEX_SUBMINOR_VERSION 1 #if YY_FLEX_SUBMINOR_VERSION > 0 #define FLEX_BETA #endif /* First, we deal with platform-specific or compiler-specific issues. */ /* begin standard C headers. */ #include #include #include #include /* end standard C headers. */ /* flex integer type definitions */ #ifndef FLEXINT_H #define FLEXINT_H /* C99 systems have . Non-C99 systems may or may not. */ #if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 says to define __STDC_LIMIT_MACROS before including stdint.h, * if you want the limit (max/min) macros for int types. */ #ifndef __STDC_LIMIT_MACROS #define __STDC_LIMIT_MACROS 1 #endif #include typedef int8_t flex_int8_t; typedef uint8_t flex_uint8_t; typedef int16_t flex_int16_t; typedef uint16_t flex_uint16_t; typedef int32_t flex_int32_t; typedef uint32_t flex_uint32_t; #else typedef signed char flex_int8_t; typedef short int flex_int16_t; typedef int flex_int32_t; typedef unsigned char flex_uint8_t; typedef unsigned short int flex_uint16_t; typedef unsigned int flex_uint32_t; /* Limits of integral types. */ #ifndef INT8_MIN #define INT8_MIN (-128) #endif #ifndef INT16_MIN #define INT16_MIN (-32767-1) #endif #ifndef INT32_MIN #define INT32_MIN (-2147483647-1) #endif #ifndef INT8_MAX #define INT8_MAX (127) #endif #ifndef INT16_MAX #define INT16_MAX (32767) #endif #ifndef INT32_MAX #define INT32_MAX (2147483647) #endif #ifndef UINT8_MAX #define UINT8_MAX (255U) #endif #ifndef UINT16_MAX #define UINT16_MAX (65535U) #endif #ifndef UINT32_MAX #define UINT32_MAX (4294967295U) #endif #endif /* ! C99 */ #endif /* ! FLEXINT_H */ /* TODO: this is always defined, so inline it */ #define yyconst const #if defined(__GNUC__) && __GNUC__ >= 3 #define yynoreturn __attribute__((__noreturn__)) #else #define yynoreturn #endif /* Returned upon end-of-file. */ #define YY_NULL 0 /* Promotes a possibly negative, possibly signed char to an unsigned * integer for use as an array index. If the signed char is negative, * we want to instead treat it as an 8-bit unsigned char, hence the * double cast. */ #define YY_SC_TO_UI(c) ((unsigned int) (unsigned char) c) /* Enter a start condition. This macro really ought to take a parameter, * but we do it the disgusting crufty way forced on us by the ()-less * definition of BEGIN. */ #define BEGIN (yy_start) = 1 + 2 * /* Translate the current start state into a value that can be later handed * to BEGIN to return to the state. The YYSTATE alias is for lex * compatibility. */ #define YY_START (((yy_start) - 1) / 2) #define YYSTATE YY_START /* Action number for EOF rule of a given start state. */ #define YY_STATE_EOF(state) (YY_END_OF_BUFFER + state + 1) /* Special action meaning "start processing a new file". */ #define YY_NEW_FILE dax_restart(dax_in ) #define YY_END_OF_BUFFER_CHAR 0 /* Size of default input buffer. */ #ifndef YY_BUF_SIZE #ifdef __ia64__ /* On IA-64, the buffer size is 16k, not 8k. * Moreover, YY_BUF_SIZE is 2*YY_READ_BUF_SIZE in the general case. * Ditto for the __ia64__ case accordingly. */ #define YY_BUF_SIZE 32768 #else #define YY_BUF_SIZE 16384 #endif /* __ia64__ */ #endif /* The state buf must be large enough to hold one state per character in the main buffer. */ #define YY_STATE_BUF_SIZE ((YY_BUF_SIZE + 2) * sizeof(yy_state_type)) #ifndef YY_TYPEDEF_YY_BUFFER_STATE #define YY_TYPEDEF_YY_BUFFER_STATE typedef struct yy_buffer_state *YY_BUFFER_STATE; #endif #ifndef YY_TYPEDEF_YY_SIZE_T #define YY_TYPEDEF_YY_SIZE_T typedef size_t yy_size_t; #endif extern unsigned int dax_leng; extern FILE *dax_in, *dax_out; #define EOB_ACT_CONTINUE_SCAN 0 #define EOB_ACT_END_OF_FILE 1 #define EOB_ACT_LAST_MATCH 2 /* Note: We specifically omit the test for yy_rule_can_match_eol because it requires * access to the local variable yy_act. Since yyless() is a macro, it would break * existing scanners that call yyless() from OUTSIDE dax_lex. * One obvious solution it to make yy_act a global. I tried that, and saw * a 5% performance hit in a non-dax_lineno scanner, because yy_act is * normally declared as a register variable-- so it is not worth it. */ #define YY_LESS_LINENO(n) \ do { \ unsigned int yyl;\ for ( yyl = n; yyl < dax_leng; ++yyl )\ if ( dax_text[yyl] == '\n' )\ --dax_lineno;\ }while(0) #define YY_LINENO_REWIND_TO(dst) \ do {\ const char *p;\ for ( p = yy_cp-1; p >= (dst); --p)\ if ( *p == '\n' )\ --dax_lineno;\ }while(0) /* Return all but the first "n" matched characters back to the input stream. */ #define yyless(n) \ do \ { \ /* Undo effects of setting up dax_text. */ \ int yyless_macro_arg = (n); \ YY_LESS_LINENO(yyless_macro_arg);\ *yy_cp = (yy_hold_char); \ YY_RESTORE_YY_MORE_OFFSET \ (yy_c_buf_p) = yy_cp = yy_bp + yyless_macro_arg - YY_MORE_ADJ; \ YY_DO_BEFORE_ACTION; /* set up dax_text again */ \ } \ while ( 0 ) #define unput(c) yyunput( c, (yytext_ptr) ) #ifndef YY_STRUCT_YY_BUFFER_STATE #define YY_STRUCT_YY_BUFFER_STATE struct yy_buffer_state { FILE *yy_input_file; char *yy_ch_buf; /* input buffer */ char *yy_buf_pos; /* current position in input buffer */ /* Size of input buffer in bytes, not including room for EOB * characters. */ int yy_buf_size; /* Number of characters read into yy_ch_buf, not including EOB * characters. */ int yy_n_chars; /* Whether we "own" the buffer - i.e., we know we created it, * and can realloc() it to grow it, and should free() it to * delete it. */ int yy_is_our_buffer; /* Whether this is an "interactive" input source; if so, and * if we're using stdio for input, then we want to use getc() * instead of fread(), to make sure we stop fetching input after * each newline. */ int yy_is_interactive; /* Whether we're considered to be at the beginning of a line. * If so, '^' rules will be active on the next match, otherwise * not. */ int yy_at_bol; int yy_bs_lineno; /**< The line count. */ int yy_bs_column; /**< The column count. */ /* Whether to try to fill the input buffer when we reach the * end of it. */ int yy_fill_buffer; int yy_buffer_status; #define YY_BUFFER_NEW 0 #define YY_BUFFER_NORMAL 1 /* When an EOF's been seen but there's still some text to process * then we mark the buffer as YY_EOF_PENDING, to indicate that we * shouldn't try reading from the input source any more. We might * still have a bunch of tokens to match, though, because of * possible backing-up. * * When we actually see the EOF, we change the status to "new" * (via dax_restart()), so that the user can continue scanning by * just pointing dax_in at a new input file. */ #define YY_BUFFER_EOF_PENDING 2 }; #endif /* !YY_STRUCT_YY_BUFFER_STATE */ /* Stack of input buffers. */ static size_t yy_buffer_stack_top = 0; /**< index of top of stack. */ static size_t yy_buffer_stack_max = 0; /**< capacity of stack. */ static YY_BUFFER_STATE * yy_buffer_stack = NULL; /**< Stack as an array. */ /* We provide macros for accessing buffer states in case in the * future we want to put the buffer states in a more general * "scanner state". * * Returns the top of the stack, or NULL. */ #define YY_CURRENT_BUFFER ( (yy_buffer_stack) \ ? (yy_buffer_stack)[(yy_buffer_stack_top)] \ : NULL) /* Same as previous macro, but useful when we know that the buffer stack is not * NULL or when we need an lvalue. For internal use only. */ #define YY_CURRENT_BUFFER_LVALUE (yy_buffer_stack)[(yy_buffer_stack_top)] /* yy_hold_char holds the character lost when dax_text is formed. */ static char yy_hold_char; static int yy_n_chars; /* number of characters read into yy_ch_buf */ unsigned int dax_leng; /* Points to current character in buffer. */ static char *yy_c_buf_p = NULL; static int yy_init = 0; /* whether we need to initialize */ static int yy_start = 0; /* start state number */ /* Flag which is used to allow dax_wrap()'s to do buffer switches * instead of setting up a fresh dax_in. A bit of a hack ... */ static int yy_did_buffer_switch_on_eof; void dax_restart (FILE *input_file ); void dax__switch_to_buffer (YY_BUFFER_STATE new_buffer ); YY_BUFFER_STATE dax__create_buffer (FILE *file,int size ); void dax__delete_buffer (YY_BUFFER_STATE b ); void dax__flush_buffer (YY_BUFFER_STATE b ); void dax_push_buffer_state (YY_BUFFER_STATE new_buffer ); void dax_pop_buffer_state (void ); static void dax_ensure_buffer_stack (void ); static void dax__load_buffer_state (void ); static void dax__init_buffer (YY_BUFFER_STATE b,FILE *file ); #define YY_FLUSH_BUFFER dax__flush_buffer(YY_CURRENT_BUFFER ) YY_BUFFER_STATE dax__scan_buffer (char *base,yy_size_t size ); YY_BUFFER_STATE dax__scan_string (yyconst char *yy_str ); YY_BUFFER_STATE dax__scan_bytes (yyconst char *bytes,int len ); void *dax_alloc (yy_size_t ); void *dax_realloc (void *,yy_size_t ); void dax_free (void * ); #define yy_new_buffer dax__create_buffer #define yy_set_interactive(is_interactive) \ { \ if ( ! YY_CURRENT_BUFFER ){ \ dax_ensure_buffer_stack (); \ YY_CURRENT_BUFFER_LVALUE = \ dax__create_buffer(dax_in,YY_BUF_SIZE ); \ } \ YY_CURRENT_BUFFER_LVALUE->yy_is_interactive = is_interactive; \ } #define yy_set_bol(at_bol) \ { \ if ( ! YY_CURRENT_BUFFER ){\ dax_ensure_buffer_stack (); \ YY_CURRENT_BUFFER_LVALUE = \ dax__create_buffer(dax_in,YY_BUF_SIZE ); \ } \ YY_CURRENT_BUFFER_LVALUE->yy_at_bol = at_bol; \ } #define YY_AT_BOL() (YY_CURRENT_BUFFER_LVALUE->yy_at_bol) /* Begin user sect3 */ #define dax_wrap() (/*CONSTCOND*/1) #define YY_SKIP_YYWRAP typedef unsigned char YY_CHAR; FILE *dax_in = NULL, *dax_out = NULL; typedef int yy_state_type; extern int dax_lineno; int dax_lineno = 1; extern char *dax_text; #ifdef yytext_ptr #undef yytext_ptr #endif #define yytext_ptr dax_text static yy_state_type yy_get_previous_state (void ); static yy_state_type yy_try_NUL_trans (yy_state_type current_state ); static int yy_get_next_buffer (void ); static void yynoreturn yy_fatal_error (yyconst char* msg ); /* Done after the current pattern has been matched and before the * corresponding action - sets up dax_text. */ #define YY_DO_BEFORE_ACTION \ (yytext_ptr) = yy_bp; \ dax_leng = (int) (yy_cp - yy_bp); \ (yy_hold_char) = *yy_cp; \ *yy_cp = '\0'; \ (yy_c_buf_p) = yy_cp; #define YY_NUM_RULES 140 #define YY_END_OF_BUFFER 141 /* This struct is not used in this scanner, but its presence is necessary. */ struct yy_trans_info { flex_int32_t yy_verify; flex_int32_t yy_nxt; }; static yyconst flex_int16_t yy_accept[775] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 141, 139, 15, 10, 10, 15, 15, 119, 10, 119, 5, 6, 5, 8, 9, 8, 135, 127, 128, 136, 133, 136, 134, 138, 127, 128, 138, 139, 40, 10, 40, 40, 40, 38, 40, 40, 40, 40, 40, 40, 40, 44, 10, 44, 44, 139, 139, 44, 51, 10, 51, 51, 51, 49, 51, 55, 10, 55, 139, 55, 72, 10, 72, 72, 72, 70, 72, 72, 72, 72, 72, 76, 10, 76, 139, 76, 83, 10, 83, 83, 83, 81, 83, 87, 10, 87, 114, 10, 114, 114, 114, 112, 114, 114, 114, 114, 114, 114, 118, 10, 118, 136, 135, 10, 0, 2, 2, 0, 4, 7, 130, 129, 0, 0, 0, 0, 0, 0, 0, 39, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 0, 0, 0, 0, 0, 0, 50, 52, 52, 0, 0, 71, 73, 73, 73, 73, 73, 73, 0, 0, 82, 84, 84, 0, 113, 115, 115, 115, 115, 115, 115, 115, 115, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 137, 0, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 52, 0, 0, 0, 0, 0, 73, 73, 73, 73, 0, 0, 0, 84, 0, 0, 115, 115, 115, 115, 115, 115, 115, 0, 0, 0, 132, 0, 14, 1, 0, 0, 125, 0, 0, 0, 122, 121, 0, 0, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 43, 0, 0, 0, 57, 0, 0, 56, 0, 0, 0, 0, 0, 54, 0, 0, 0, 59, 58, 73, 73, 73, 73, 0, 0, 75, 0, 0, 0, 0, 0, 0, 86, 0, 115, 115, 115, 115, 115, 115, 115, 0, 0, 117, 0, 0, 0, 0, 126, 120, 0, 0, 17, 41, 41, 41, 41, 41, 0, 0, 41, 41, 41, 0, 16, 0, 57, 0, 88, 56, 0, 0, 48, 47, 0, 0, 73, 0, 0, 73, 73, 73, 0, 89, 0, 80, 79, 0, 0, 0, 0, 0, 115, 115, 0, 0, 115, 0, 0, 0, 0, 0, 0, 0, 12, 0, 123, 124, 17, 41, 0, 0, 41, 0, 0, 41, 0, 29, 28, 41, 0, 41, 0, 41, 0, 16, 46, 0, 88, 45, 0, 0, 0, 0, 0, 63, 62, 73, 73, 73, 0, 74, 89, 0, 0, 91, 90, 0, 0, 0, 115, 115, 0, 105, 104, 115, 0, 111, 110, 0, 0, 0, 0, 0, 11, 41, 0, 21, 20, 41, 0, 25, 24, 41, 41, 41, 0, 33, 32, 41, 0, 42, 46, 77, 45, 0, 78, 0, 61, 60, 73, 73, 73, 0, 0, 0, 0, 0, 115, 115, 115, 0, 116, 0, 0, 0, 0, 41, 41, 41, 0, 0, 41, 41, 77, 0, 53, 78, 73, 0, 0, 0, 0, 0, 0, 0, 0, 0, 115, 115, 115, 0, 0, 0, 0, 41, 41, 0, 0, 0, 31, 30, 41, 41, 73, 0, 67, 66, 0, 69, 68, 0, 85, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 131, 0, 0, 0, 41, 0, 0, 0, 27, 26, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23, 22, 0, 35, 34, 41, 0, 65, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 18, 41, 93, 0, 92, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 95, 94, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 99, 0, 98, 0, 103, 0, 102, 0, 109, 0, 108, 0, 0, 0, 0, 0, 0, 41, 97, 96, 101, 100, 107, 106, 0, 0, 0, 12, 0, 12, 0, 41, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 11, 0, 37, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 0 } ; static yyconst YY_CHAR yy_ec[256] = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 3, 1, 2, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 5, 6, 7, 1, 1, 8, 9, 1, 1, 1, 1, 1, 10, 11, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 15, 16, 17, 18, 19, 1, 20, 21, 22, 23, 24, 21, 25, 25, 25, 25, 25, 26, 27, 25, 28, 29, 25, 25, 30, 31, 25, 25, 25, 25, 32, 25, 33, 1, 34, 1, 25, 1, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 25, 57, 58, 59, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } ; static yyconst YY_CHAR yy_meta[60] = { 0, 1, 2, 2, 2, 1, 1, 1, 1, 1, 3, 3, 1, 4, 5, 1, 1, 1, 6, 1, 7, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 7, 7, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 } ; static yyconst flex_uint16_t yy_base[814] = { 0, 0, 0, 0, 3, 6, 9, 24, 27, 11, 14, 15, 17, 29, 38, 45, 52, 59, 61, 67, 70, 95, 0, 73, 76, 153, 156, 159, 162, 177, 180, 183, 186, 201, 204, 207, 210, 226, 278, 245, 248, 251, 254, 297, 300, 303, 306, 330, 0, 388, 391, 394, 397, 412, 415, 418, 421, 437, 489, 456, 459, 541, 0, 462, 465, 598, 600, 2053, 2135, 2135, 269, 272, 47, 62, 2135, 321, 172, 2135, 2135, 2033, 2135, 2135, 2022, 2135, 2033, 2031, 475, 2135, 2135, 2135, 2135, 2027, 2025, 1987, 468, 2135, 324, 1998, 0, 196, 2135, 8, 1968, 1958, 1952, 1863, 1858, 35, 2135, 481, 600, 608, 604, 612, 641, 2135, 509, 1878, 0, 258, 2135, 1855, 2135, 515, 660, 663, 682, 2135, 525, 1875, 0, 310, 2135, 1854, 1852, 1855, 1834, 1849, 2135, 528, 704, 701, 723, 2135, 531, 1869, 0, 407, 2135, 1847, 2135, 534, 606, 2135, 626, 1857, 0, 469, 2135, 1831, 1829, 1821, 1830, 1814, 32, 2135, 667, 621, 1851, 1820, 671, 628, 1784, 2135, 1827, 1818, 2135, 2135, 2135, 2, 44, 1781, 1780, 1778, 1814, 1793, 2135, 0, 1787, 1774, 1777, 1784, 1784, 1772, 1747, 1752, 1753, 1752, 1748, 1742, 1722, 1735, 1712, 1715, 1720, 2135, 0, 1713, 1695, 1693, 2135, 0, 686, 1670, 1671, 1668, 1660, 1666, 1656, 2135, 0, 1661, 1650, 2135, 0, 1652, 1644, 1635, 1646, 1627, 1648, 1632, 1626, 1634, 1648, 1646, 1649, 458, 1611, 2135, 72, 0, 1606, 1602, 1635, 1633, 1594, 2135, 1607, 1590, 1586, 1594, 1592, 1601, 1583, 1567, 1567, 1600, 759, 762, 1577, 1564, 1570, 1547, 1559, 1560, 1544, 727, 766, 772, 1534, 779, 783, 1545, 1536, 1520, 1519, 791, 795, 1528, 799, 802, 808, 1527, 1517, 1517, 1513, 1506, 1496, 1503, 815, 819, 1519, 2135, 1521, 2135, 2135, 17, 1488, 2135, 1514, 1513, 1473, 2135, 2135, 1464, 1475, 1477, 1459, 1490, 1453, 1450, 825, 1455, 1443, 1438, 828, 832, 2135, 836, 1449, 1442, 845, 1447, 1432, 849, 1437, 853, 856, 864, 871, 2135, 874, 1433, 877, 2135, 2135, 1425, 891, 1426, 1421, 894, 897, 2135, 900, 1410, 903, 907, 919, 922, 2135, 925, 928, 931, 1411, 1406, 947, 1404, 950, 953, 956, 2135, 966, 1422, 646, 976, 2135, 2135, 1429, 1427, 959, 1419, 979, 1386, 983, 1379, 986, 995, 1374, 1004, 1381, 1007, 989, 1379, 1010, 1367, 1013, 1020, 1369, 1024, 2135, 2135, 1032, 1357, 1035, 1038, 1054, 1352, 1353, 1347, 1041, 1062, 1065, 2135, 2135, 1073, 1077, 1080, 1093, 1096, 1333, 1325, 1104, 1111, 1334, 1120, 1123, 1131, 1352, 507, 1353, 1136, 2135, 1347, 2135, 2135, 1139, 1315, 1142, 1149, 1291, 1159, 1162, 1297, 1176, 2135, 2135, 1295, 1170, 1285, 1187, 1298, 1195, 1198, 1201, 1275, 1204, 1207, 1212, 1274, 1215, 1218, 1231, 2135, 2135, 1282, 1273, 1263, 1239, 2135, 1242, 1245, 1249, 2135, 2135, 1258, 124, 125, 1274, 1262, 1266, 2135, 2135, 1259, 1274, 2135, 2135, 1282, 1261, 491, 152, 5, 2135, 1235, 1285, 2135, 2135, 1234, 1293, 2135, 2135, 1227, 1301, 1226, 1304, 2135, 2135, 1235, 1312, 2135, 1317, 1320, 1323, 1329, 1332, 1335, 2135, 2135, 1236, 1346, 1349, 1352, 1223, 1211, 1217, 1204, 1210, 1202, 1187, 1355, 2135, 1218, 174, 158, 384, 1188, 1177, 1358, 1365, 1374, 1186, 1181, 1382, 1385, 2135, 1388, 1187, 1391, 1395, 1407, 1410, 1418, 1175, 1169, 1162, 1141, 1423, 1426, 1429, 1161, 1435, 176, 461, 1134, 1445, 1448, 1452, 1464, 2135, 2135, 1472, 1151, 1475, 1478, 2135, 2135, 1491, 2135, 2135, 1499, 2135, 1128, 1131, 1122, 1120, 1502, 1505, 1518, 1521, 1529, 1534, 2135, 1545, 228, 179, 1548, 1551, 1555, 1567, 2135, 2135, 1575, 1579, 1143, 1587, 1591, 1113, 1105, 1103, 1101, 1599, 567, 570, 1607, 582, 626, 1615, 668, 671, 280, 222, 643, 1623, 1626, 1635, 2135, 2135, 1643, 2135, 2135, 1101, 1651, 2135, 2135, 1142, 1093, 1127, 1077, 1095, 1076, 1084, 1066, 1081, 1059, 1074, 1052, 1068, 1049, 1058, 1040, 715, 244, 385, 1659, 2135, 2135, 1053, 2135, 1082, 2135, 1078, 1026, 1015, 1016, 1006, 1008, 992, 1000, 974, 973, 921, 929, 918, 388, 391, 1667, 937, 2135, 2135, 912, 907, 889, 902, 861, 851, 835, 848, 832, 830, 792, 776, 1670, 1673, 1676, 1693, 753, 714, 746, 695, 712, 666, 678, 644, 648, 603, 609, 501, 471, 1701, 1704, 1711, 1721, 1739, 1773, 408, 429, 2135, 421, 2135, 421, 2135, 402, 2135, 402, 2135, 396, 2135, 725, 1791, 675, 506, 1809, 1843, 272, 2135, 2135, 2135, 2135, 2135, 2135, 684, 706, 200, 311, 731, 260, 1731, 220, 668, 218, 202, 207, 1765, 1836, 296, 153, 1861, 1757, 728, 2135, 1799, 2135, 2135, 1734, 1864, 439, 709, 753, 773, 806, 807, 761, 873, 823, 901, 948, 1031, 1013, 1035, 1014, 929, 1877, 1881, 2135, 2135, 1899, 1906, 1913, 1920, 1927, 1934, 1941, 1948, 1955, 1962, 1969, 1976, 1983, 1990, 1995, 2000, 2005, 2010, 2015, 2022, 2025, 2028, 2031, 2034, 2037, 2044, 2048, 2054, 2060, 2066, 2072, 2078, 2085, 2092, 2099, 2106, 2113, 2120, 2127 } ; static yyconst flex_int16_t yy_def[814] = { 0, 775, 775, 776, 776, 776, 776, 777, 777, 778, 778, 779, 779, 780, 780, 780, 780, 781, 781, 775, 775, 774, 21, 782, 782, 782, 782, 775, 775, 782, 782, 775, 775, 782, 782, 782, 782, 783, 783, 784, 784, 775, 775, 784, 784, 784, 784, 774, 47, 785, 785, 775, 775, 785, 785, 785, 785, 786, 786, 787, 787, 774, 61, 788, 788, 780, 780, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 789, 789, 789, 789, 789, 789, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 790, 774, 774, 790, 774, 774, 774, 774, 774, 774, 774, 774, 791, 774, 774, 791, 791, 791, 791, 791, 774, 774, 774, 774, 774, 774, 774, 774, 792, 774, 774, 792, 774, 774, 774, 774, 774, 774, 793, 774, 774, 793, 793, 793, 793, 793, 793, 774, 774, 774, 774, 774, 774, 794, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 789, 795, 774, 774, 774, 774, 774, 774, 774, 774, 790, 790, 796, 774, 774, 791, 791, 791, 791, 791, 791, 797, 774, 774, 792, 792, 798, 774, 793, 793, 793, 793, 793, 793, 793, 793, 799, 774, 774, 800, 774, 800, 774, 774, 774, 801, 774, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 802, 802, 774, 774, 774, 774, 774, 774, 774, 790, 803, 803, 774, 774, 774, 791, 791, 791, 791, 804, 804, 774, 792, 805, 805, 793, 793, 793, 793, 793, 793, 793, 806, 806, 774, 774, 800, 774, 774, 800, 774, 774, 801, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 774, 802, 774, 802, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 803, 774, 803, 774, 774, 774, 774, 791, 791, 791, 791, 774, 804, 774, 804, 774, 774, 774, 774, 805, 774, 805, 793, 793, 793, 793, 793, 793, 793, 774, 806, 774, 806, 774, 800, 807, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 774, 774, 789, 789, 789, 802, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 803, 774, 791, 774, 774, 791, 791, 791, 804, 774, 774, 774, 774, 805, 774, 774, 774, 774, 793, 793, 774, 774, 793, 774, 774, 806, 774, 800, 807, 807, 774, 807, 774, 774, 774, 789, 774, 774, 789, 774, 774, 789, 774, 774, 774, 789, 774, 789, 774, 789, 802, 774, 774, 774, 774, 774, 803, 774, 774, 774, 774, 774, 774, 791, 791, 791, 774, 774, 774, 805, 774, 774, 774, 774, 774, 774, 793, 793, 774, 774, 774, 793, 774, 774, 774, 806, 774, 800, 807, 807, 774, 789, 774, 774, 774, 789, 774, 774, 774, 789, 789, 789, 774, 774, 774, 789, 774, 774, 774, 774, 774, 803, 774, 774, 774, 774, 791, 791, 791, 805, 774, 774, 774, 774, 793, 793, 793, 774, 774, 774, 800, 807, 807, 789, 789, 789, 774, 774, 789, 789, 774, 774, 774, 774, 791, 774, 774, 774, 774, 805, 774, 774, 774, 774, 793, 793, 793, 774, 800, 807, 807, 789, 789, 774, 774, 774, 774, 774, 789, 789, 791, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 774, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 807, 808, 809, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 810, 811, 808, 809, 789, 774, 774, 774, 774, 774, 774, 800, 810, 807, 812, 811, 813, 807, 789, 800, 812, 774, 813, 807, 789, 800, 774, 774, 774, 800, 774, 774, 774, 774, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 774, 0, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; static yyconst flex_uint16_t yy_nxt[2195] = { 0, 774, 70, 71, 70, 70, 71, 70, 70, 71, 70, 70, 71, 70, 78, 240, 72, 78, 81, 72, 81, 79, 73, 422, 79, 73, 75, 71, 75, 75, 71, 75, 84, 85, 82, 295, 82, 86, 87, 364, 76, 84, 85, 76, 524, 88, 86, 87, 84, 85, 188, 89, 171, 86, 88, 84, 85, 189, 89, 241, 86, 88, 91, 92, 91, 92, 172, 171, 88, 71, 71, 71, 71, 71, 71, 109, 71, 109, 109, 71, 109, 173, 195, 94, 230, 240, 94, 299, 196, 110, 231, 242, 110, 93, 243, 93, 95, 96, 71, 96, 95, 95, 95, 95, 95, 95, 95, 97, 95, 98, 95, 99, 95, 100, 95, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 95, 95, 98, 98, 101, 98, 98, 102, 98, 98, 103, 104, 98, 98, 98, 105, 98, 98, 98, 98, 98, 98, 98, 106, 107, 98, 98, 109, 71, 109, 109, 71, 109, 71, 71, 71, 71, 71, 71, 512, 514, 111, 422, 749, 111, 513, 515, 112, 422, 174, 112, 109, 71, 109, 109, 71, 109, 71, 71, 71, 71, 71, 71, 173, 295, 110, 422, 551, 110, 422, 550, 113, 523, 174, 113, 109, 71, 109, 109, 71, 109, 109, 71, 109, 109, 71, 109, 173, 740, 111, 422, 423, 111, 745, 611, 114, 740, 583, 114, 115, 116, 71, 116, 115, 115, 115, 115, 115, 115, 115, 117, 115, 422, 115, 119, 115, 120, 115, 422, 123, 71, 123, 123, 71, 123, 71, 71, 71, 71, 71, 71, 115, 115, 124, 422, 174, 124, 641, 610, 125, 743, 740, 125, 170, 170, 170, 170, 170, 170, 173, 121, 115, 116, 71, 116, 115, 115, 115, 115, 115, 115, 115, 117, 115, 664, 115, 119, 115, 120, 115, 295, 123, 71, 123, 123, 71, 123, 123, 71, 123, 123, 71, 123, 115, 115, 124, 295, 174, 124, 740, 640, 126, 748, 737, 126, 170, 170, 170, 170, 170, 170, 173, 121, 127, 128, 71, 128, 127, 127, 127, 127, 127, 127, 127, 129, 127, 130, 127, 131, 127, 132, 127, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 127, 127, 130, 130, 130, 130, 130, 130, 130, 130, 133, 130, 130, 134, 130, 135, 130, 130, 130, 136, 130, 130, 130, 137, 130, 130, 130, 139, 71, 139, 139, 71, 139, 71, 71, 71, 71, 71, 71, 422, 422, 140, 729, 295, 140, 728, 422, 141, 727, 174, 141, 139, 71, 139, 139, 71, 139, 139, 71, 139, 139, 71, 139, 173, 726, 140, 681, 725, 140, 682, 665, 142, 724, 552, 142, 143, 144, 71, 144, 143, 143, 143, 143, 143, 143, 143, 145, 143, 723, 143, 147, 143, 148, 143, 295, 151, 71, 151, 151, 71, 151, 166, 71, 166, 166, 71, 166, 143, 143, 152, 174, 174, 152, 295, 757, 167, 422, 716, 167, 179, 170, 170, 170, 297, 173, 173, 149, 143, 144, 71, 144, 143, 143, 143, 143, 143, 143, 143, 145, 143, 185, 143, 147, 143, 148, 143, 295, 180, 170, 170, 170, 584, 732, 181, 170, 170, 170, 522, 182, 143, 143, 735, 295, 183, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 479, 715, 149, 153, 154, 71, 154, 153, 153, 153, 153, 153, 153, 153, 155, 153, 156, 153, 157, 153, 158, 153, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 153, 153, 156, 156, 156, 156, 156, 159, 156, 156, 156, 156, 156, 160, 156, 156, 161, 156, 156, 162, 163, 164, 156, 156, 156, 156, 156, 84, 85, 84, 85, 174, 86, 628, 86, 174, 630, 174, 197, 174, 168, 714, 168, 174, 222, 173, 197, 629, 632, 173, 631, 173, 174, 173, 170, 170, 170, 173, 169, 232, 169, 198, 633, 199, 236, 198, 173, 204, 713, 198, 200, 199, 174, 198, 200, 199, 201, 237, 203, 197, 201, 202, 203, 712, 201, 202, 173, 422, 201, 202, 295, 174, 634, 202, 174, 170, 170, 170, 208, 170, 170, 170, 198, 419, 204, 173, 635, 732, 173, 711, 710, 203, 295, 174, 271, 271, 271, 201, 642, 733, 208, 198, 202, 204, 198, 744, 204, 173, 295, 272, 203, 709, 174, 203, 636, 174, 209, 638, 732, 209, 738, 202, 217, 198, 202, 204, 173, 708, 637, 173, 733, 639, 203, 295, 174, 327, 327, 327, 201, 295, 707, 217, 198, 202, 204, 198, 732, 204, 173, 295, 328, 203, 295, 758, 203, 735, 663, 201, 706, 705, 201, 753, 218, 730, 198, 218, 204, 316, 316, 316, 316, 316, 316, 203, 329, 329, 329, 295, 763, 201, 329, 329, 329, 318, 202, 295, 318, 271, 271, 271, 331, 334, 334, 334, 759, 335, 331, 295, 336, 341, 341, 341, 272, 341, 341, 341, 319, 346, 346, 346, 348, 348, 348, 704, 760, 343, 348, 348, 348, 343, 332, 697, 347, 359, 359, 359, 350, 359, 359, 359, 295, 295, 350, 376, 376, 376, 316, 316, 316, 361, 316, 316, 316, 361, 316, 316, 316, 295, 377, 351, 344, 696, 318, 384, 384, 384, 318, 387, 387, 387, 318, 327, 327, 327, 389, 389, 389, 765, 390, 761, 762, 391, 329, 329, 329, 695, 328, 381, 362, 329, 329, 329, 329, 329, 329, 334, 334, 334, 331, 335, 764, 694, 336, 693, 692, 331, 691, 295, 331, 395, 395, 395, 341, 341, 341, 341, 341, 341, 341, 341, 341, 346, 346, 346, 396, 402, 402, 402, 343, 403, 690, 343, 404, 392, 343, 295, 347, 348, 348, 348, 348, 348, 348, 348, 348, 348, 406, 406, 406, 408, 408, 408, 400, 350, 771, 766, 350, 689, 688, 350, 397, 407, 687, 295, 409, 412, 412, 412, 415, 415, 415, 359, 359, 359, 359, 359, 359, 426, 426, 426, 413, 686, 295, 416, 359, 359, 359, 361, 685, 680, 361, 679, 678, 405, 421, 421, 421, 428, 428, 428, 361, 431, 431, 431, 376, 376, 376, 443, 443, 443, 422, 423, 429, 434, 434, 434, 432, 435, 767, 377, 436, 417, 438, 438, 438, 316, 316, 316, 384, 384, 384, 446, 446, 446, 439, 677, 771, 440, 387, 387, 387, 318, 389, 389, 389, 676, 390, 295, 295, 391, 329, 329, 329, 450, 450, 450, 395, 395, 395, 458, 458, 458, 675, 674, 442, 295, 331, 769, 451, 295, 673, 396, 452, 452, 452, 459, 453, 672, 671, 454, 460, 460, 460, 402, 402, 402, 670, 403, 669, 770, 404, 348, 348, 348, 448, 406, 406, 406, 462, 462, 462, 768, 463, 668, 667, 464, 666, 350, 662, 661, 407, 408, 408, 408, 465, 465, 465, 660, 466, 659, 658, 467, 412, 412, 412, 657, 409, 656, 461, 470, 470, 470, 655, 471, 654, 653, 472, 413, 415, 415, 415, 474, 474, 474, 652, 475, 651, 650, 476, 359, 359, 359, 649, 416, 421, 421, 421, 426, 426, 426, 428, 428, 428, 648, 647, 361, 646, 484, 484, 484, 422, 485, 627, 626, 486, 429, 625, 431, 431, 431, 488, 488, 488, 624, 489, 620, 599, 490, 438, 438, 438, 480, 432, 598, 434, 434, 434, 597, 435, 596, 477, 436, 593, 440, 585, 494, 494, 494, 481, 495, 581, 574, 496, 498, 498, 498, 443, 443, 443, 500, 500, 500, 446, 446, 446, 502, 502, 502, 573, 499, 329, 329, 329, 450, 450, 450, 505, 505, 505, 572, 506, 571, 562, 507, 561, 560, 331, 554, 451, 452, 452, 452, 553, 453, 549, 548, 454, 458, 458, 458, 460, 460, 460, 348, 348, 348, 503, 462, 462, 462, 547, 463, 546, 459, 464, 545, 465, 465, 465, 350, 466, 544, 543, 467, 470, 470, 470, 542, 471, 536, 531, 472, 474, 474, 474, 530, 475, 527, 526, 476, 519, 519, 519, 484, 484, 484, 525, 485, 521, 511, 486, 488, 488, 488, 518, 489, 520, 517, 490, 528, 528, 528, 494, 494, 494, 516, 495, 510, 509, 496, 498, 498, 498, 508, 529, 500, 500, 500, 532, 532, 532, 502, 502, 502, 504, 501, 499, 533, 533, 533, 535, 535, 535, 505, 505, 505, 497, 506, 493, 492, 507, 491, 487, 534, 537, 537, 537, 539, 539, 539, 348, 348, 348, 519, 519, 519, 555, 555, 555, 538, 483, 482, 540, 528, 528, 528, 350, 422, 478, 520, 473, 556, 557, 557, 557, 469, 558, 468, 529, 559, 532, 532, 532, 533, 533, 533, 535, 535, 535, 537, 537, 537, 457, 563, 563, 563, 456, 564, 455, 534, 565, 449, 541, 447, 538, 539, 539, 539, 566, 566, 566, 445, 567, 444, 441, 568, 569, 569, 569, 437, 540, 575, 575, 575, 577, 577, 577, 579, 579, 579, 433, 430, 570, 582, 582, 582, 576, 427, 425, 578, 424, 418, 580, 586, 586, 586, 555, 555, 555, 295, 588, 588, 588, 414, 589, 411, 410, 590, 587, 401, 399, 556, 557, 557, 557, 398, 558, 394, 393, 559, 591, 591, 591, 594, 594, 594, 563, 563, 563, 388, 564, 386, 385, 565, 383, 592, 382, 380, 595, 566, 566, 566, 379, 567, 378, 375, 568, 569, 569, 569, 575, 575, 575, 600, 600, 600, 374, 601, 373, 372, 602, 371, 370, 570, 369, 576, 577, 577, 577, 603, 603, 603, 368, 604, 367, 366, 605, 579, 579, 579, 365, 578, 606, 606, 606, 295, 607, 363, 358, 608, 357, 356, 580, 582, 582, 582, 612, 612, 612, 586, 586, 586, 355, 614, 614, 614, 354, 615, 353, 295, 616, 613, 352, 345, 587, 588, 588, 588, 340, 589, 339, 338, 590, 591, 591, 591, 609, 617, 617, 617, 337, 618, 333, 326, 619, 594, 594, 594, 592, 621, 621, 621, 325, 622, 324, 323, 623, 600, 600, 600, 595, 601, 322, 321, 602, 603, 603, 603, 320, 604, 315, 314, 605, 606, 606, 606, 313, 607, 312, 311, 608, 612, 612, 612, 643, 643, 643, 310, 644, 309, 308, 645, 307, 614, 614, 614, 613, 615, 306, 305, 616, 617, 617, 617, 304, 618, 303, 302, 619, 621, 621, 621, 301, 622, 298, 296, 623, 643, 643, 643, 295, 644, 293, 292, 645, 683, 683, 683, 698, 698, 698, 699, 699, 699, 683, 683, 683, 291, 289, 288, 684, 422, 287, 286, 295, 285, 700, 422, 284, 684, 422, 701, 701, 701, 283, 702, 282, 280, 703, 698, 698, 698, 699, 699, 699, 279, 278, 422, 276, 718, 718, 718, 275, 719, 274, 295, 720, 700, 422, 701, 701, 701, 273, 702, 270, 422, 703, 717, 269, 742, 742, 742, 754, 754, 754, 422, 420, 420, 420, 420, 420, 420, 420, 420, 420, 422, 423, 420, 295, 267, 420, 420, 420, 422, 420, 750, 750, 750, 266, 751, 265, 264, 752, 742, 742, 742, 263, 262, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 422, 261, 420, 260, 259, 420, 420, 420, 422, 420, 718, 718, 718, 257, 719, 256, 255, 720, 750, 750, 750, 480, 751, 420, 420, 752, 422, 420, 420, 420, 420, 420, 736, 420, 420, 420, 254, 253, 420, 252, 251, 420, 420, 420, 422, 420, 250, 249, 248, 247, 246, 245, 244, 239, 236, 746, 746, 746, 238, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 736, 747, 234, 420, 233, 229, 420, 420, 420, 422, 420, 746, 746, 746, 754, 754, 754, 228, 755, 227, 226, 756, 225, 223, 420, 420, 747, 772, 772, 772, 295, 772, 772, 772, 221, 219, 216, 215, 214, 213, 212, 210, 207, 773, 205, 194, 193, 773, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 74, 74, 74, 74, 74, 74, 74, 77, 77, 77, 77, 77, 77, 77, 80, 80, 80, 80, 80, 80, 80, 83, 83, 83, 83, 83, 83, 83, 90, 90, 90, 90, 90, 90, 90, 108, 108, 108, 108, 108, 108, 108, 118, 118, 118, 118, 118, 118, 118, 122, 122, 122, 122, 122, 122, 122, 138, 138, 138, 138, 138, 138, 138, 146, 146, 146, 146, 146, 146, 146, 150, 150, 150, 150, 150, 150, 150, 165, 165, 165, 165, 165, 165, 165, 187, 187, 187, 192, 187, 206, 206, 206, 191, 206, 211, 211, 211, 190, 211, 220, 220, 220, 186, 220, 224, 224, 224, 184, 224, 235, 235, 235, 235, 235, 178, 235, 258, 177, 258, 268, 178, 268, 277, 177, 277, 281, 176, 281, 290, 175, 290, 294, 294, 294, 294, 294, 294, 294, 300, 774, 774, 300, 317, 317, 317, 317, 317, 317, 330, 330, 330, 330, 330, 330, 342, 342, 342, 342, 342, 342, 349, 349, 349, 349, 349, 349, 360, 360, 360, 360, 360, 360, 420, 420, 420, 420, 420, 420, 420, 721, 721, 721, 721, 721, 721, 721, 722, 722, 722, 722, 722, 722, 722, 731, 731, 731, 731, 731, 731, 731, 734, 734, 734, 734, 734, 734, 734, 739, 739, 739, 739, 739, 739, 739, 741, 741, 741, 741, 741, 741, 741, 67, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; static yyconst flex_int16_t yy_chk[2195] = { 0, 0, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 9, 179, 3, 10, 11, 4, 12, 9, 5, 481, 10, 6, 7, 7, 7, 8, 8, 8, 13, 13, 11, 297, 12, 13, 13, 297, 7, 14, 14, 8, 481, 13, 14, 14, 15, 15, 101, 15, 72, 15, 14, 16, 16, 101, 16, 179, 16, 15, 17, 17, 18, 18, 72, 73, 16, 19, 19, 19, 20, 20, 20, 23, 23, 23, 24, 24, 24, 73, 107, 19, 164, 240, 20, 240, 107, 23, 164, 180, 24, 17, 180, 18, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 466, 467, 25, 480, 745, 26, 466, 467, 27, 523, 76, 28, 29, 29, 29, 30, 30, 30, 31, 31, 31, 32, 32, 32, 76, 522, 29, 551, 523, 30, 584, 522, 31, 480, 99, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 99, 741, 33, 732, 732, 34, 740, 584, 35, 739, 551, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 610, 37, 37, 37, 37, 37, 583, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 37, 37, 39, 641, 119, 40, 610, 583, 41, 737, 735, 42, 70, 70, 70, 71, 71, 71, 119, 37, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 641, 38, 38, 38, 38, 38, 609, 43, 43, 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 38, 38, 43, 744, 131, 44, 733, 609, 45, 744, 723, 46, 75, 75, 75, 96, 96, 96, 131, 38, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 49, 49, 49, 50, 50, 50, 51, 51, 51, 52, 52, 52, 524, 642, 49, 715, 663, 50, 713, 664, 51, 711, 147, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56, 56, 56, 147, 709, 53, 663, 707, 54, 664, 642, 55, 705, 524, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 704, 57, 57, 57, 57, 57, 755, 59, 59, 59, 60, 60, 60, 63, 63, 63, 64, 64, 64, 57, 57, 59, 94, 157, 60, 237, 755, 63, 552, 697, 64, 86, 109, 109, 109, 237, 94, 157, 57, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 94, 58, 58, 58, 58, 58, 479, 86, 116, 116, 116, 552, 720, 86, 123, 123, 123, 479, 86, 58, 58, 720, 419, 86, 128, 128, 128, 139, 139, 139, 144, 144, 144, 151, 151, 151, 419, 696, 58, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 65, 65, 66, 66, 110, 65, 601, 66, 112, 602, 152, 110, 111, 65, 695, 66, 113, 152, 110, 111, 601, 604, 112, 602, 152, 167, 111, 154, 154, 154, 113, 65, 167, 66, 110, 604, 110, 171, 112, 167, 112, 694, 111, 110, 111, 114, 113, 112, 113, 110, 171, 111, 114, 112, 110, 113, 693, 111, 112, 114, 611, 113, 111, 364, 124, 605, 113, 125, 166, 166, 166, 124, 170, 170, 170, 114, 364, 114, 124, 605, 719, 125, 692, 691, 114, 738, 126, 212, 212, 212, 114, 611, 719, 126, 124, 114, 124, 125, 738, 125, 126, 730, 212, 124, 690, 141, 125, 607, 140, 124, 608, 731, 125, 730, 124, 140, 126, 125, 126, 141, 689, 607, 140, 731, 608, 126, 756, 142, 267, 267, 267, 126, 640, 688, 142, 141, 126, 141, 140, 734, 140, 142, 717, 267, 141, 748, 756, 140, 734, 640, 141, 687, 686, 140, 748, 141, 717, 142, 140, 142, 258, 258, 258, 259, 259, 259, 142, 268, 268, 268, 757, 761, 142, 269, 269, 269, 258, 142, 761, 259, 271, 271, 271, 268, 272, 272, 272, 757, 272, 269, 758, 272, 277, 277, 277, 271, 278, 278, 278, 259, 280, 280, 280, 281, 281, 281, 685, 758, 277, 282, 282, 282, 278, 269, 680, 280, 290, 290, 290, 281, 291, 291, 291, 759, 760, 282, 312, 312, 312, 316, 316, 316, 290, 317, 317, 317, 291, 319, 319, 319, 763, 312, 282, 278, 679, 316, 322, 322, 322, 317, 325, 325, 325, 319, 327, 327, 327, 328, 328, 328, 763, 328, 759, 760, 328, 329, 329, 329, 678, 327, 319, 291, 330, 330, 330, 332, 332, 332, 334, 334, 334, 329, 334, 762, 677, 334, 676, 675, 330, 674, 762, 332, 338, 338, 338, 341, 341, 341, 342, 342, 342, 344, 344, 344, 346, 346, 346, 338, 347, 347, 347, 341, 347, 673, 342, 347, 332, 344, 764, 346, 348, 348, 348, 349, 349, 349, 351, 351, 351, 352, 352, 352, 353, 353, 353, 344, 348, 770, 764, 349, 672, 671, 351, 338, 352, 670, 770, 353, 356, 356, 356, 358, 358, 358, 359, 359, 359, 360, 360, 360, 370, 370, 370, 356, 669, 765, 358, 362, 362, 362, 359, 666, 662, 360, 661, 660, 351, 365, 365, 365, 372, 372, 372, 362, 374, 374, 374, 376, 376, 376, 382, 382, 382, 365, 365, 372, 377, 377, 377, 374, 377, 765, 376, 377, 362, 379, 379, 379, 381, 381, 381, 384, 384, 384, 386, 386, 386, 379, 659, 769, 379, 387, 387, 387, 381, 389, 389, 389, 658, 389, 767, 769, 389, 392, 392, 392, 394, 394, 394, 395, 395, 395, 400, 400, 400, 657, 656, 381, 766, 392, 767, 394, 768, 655, 395, 396, 396, 396, 400, 396, 654, 653, 396, 401, 401, 401, 402, 402, 402, 652, 402, 651, 768, 402, 405, 405, 405, 392, 406, 406, 406, 407, 407, 407, 766, 407, 650, 648, 407, 646, 405, 639, 638, 406, 408, 408, 408, 409, 409, 409, 637, 409, 636, 635, 409, 412, 412, 412, 634, 408, 633, 405, 413, 413, 413, 632, 413, 631, 630, 413, 412, 415, 415, 415, 416, 416, 416, 629, 416, 628, 627, 416, 417, 417, 417, 626, 415, 421, 421, 421, 426, 426, 426, 428, 428, 428, 625, 624, 417, 620, 429, 429, 429, 421, 429, 599, 598, 429, 428, 597, 431, 431, 431, 432, 432, 432, 596, 432, 593, 574, 432, 438, 438, 438, 421, 431, 573, 434, 434, 434, 572, 434, 571, 417, 434, 561, 438, 553, 440, 440, 440, 421, 440, 549, 545, 440, 442, 442, 442, 443, 443, 443, 444, 444, 444, 446, 446, 446, 447, 447, 447, 544, 442, 448, 448, 448, 450, 450, 450, 451, 451, 451, 543, 451, 542, 536, 451, 531, 530, 448, 526, 450, 452, 452, 452, 525, 452, 521, 518, 452, 458, 458, 458, 460, 460, 460, 461, 461, 461, 448, 462, 462, 462, 517, 462, 516, 458, 462, 515, 465, 465, 465, 461, 465, 514, 513, 465, 470, 470, 470, 512, 470, 508, 497, 470, 474, 474, 474, 493, 474, 491, 487, 474, 477, 477, 477, 484, 484, 484, 483, 484, 478, 461, 484, 488, 488, 488, 473, 488, 477, 469, 488, 492, 492, 492, 494, 494, 494, 468, 494, 457, 456, 494, 498, 498, 498, 455, 492, 500, 500, 500, 501, 501, 501, 502, 502, 502, 449, 445, 498, 503, 503, 503, 504, 504, 504, 505, 505, 505, 441, 505, 439, 437, 505, 433, 430, 503, 509, 509, 509, 510, 510, 510, 511, 511, 511, 519, 519, 519, 527, 527, 527, 509, 427, 423, 510, 528, 528, 528, 511, 420, 418, 519, 414, 527, 529, 529, 529, 411, 529, 410, 528, 529, 532, 532, 532, 533, 533, 533, 535, 535, 535, 537, 537, 537, 399, 538, 538, 538, 398, 538, 397, 533, 538, 393, 511, 388, 537, 539, 539, 539, 540, 540, 540, 385, 540, 383, 380, 540, 541, 541, 541, 378, 539, 546, 546, 546, 547, 547, 547, 548, 548, 548, 375, 373, 541, 550, 550, 550, 546, 371, 369, 547, 368, 363, 548, 554, 554, 554, 555, 555, 555, 550, 556, 556, 556, 357, 556, 355, 354, 556, 554, 345, 340, 555, 557, 557, 557, 339, 557, 337, 333, 557, 560, 560, 560, 562, 562, 562, 563, 563, 563, 326, 563, 324, 323, 563, 321, 560, 320, 315, 562, 566, 566, 566, 314, 566, 313, 311, 566, 569, 569, 569, 575, 575, 575, 576, 576, 576, 310, 576, 309, 308, 576, 307, 306, 569, 305, 575, 577, 577, 577, 578, 578, 578, 302, 578, 301, 300, 578, 579, 579, 579, 298, 577, 580, 580, 580, 294, 580, 292, 289, 580, 288, 287, 579, 582, 582, 582, 585, 585, 585, 586, 586, 586, 286, 587, 587, 587, 285, 587, 284, 582, 587, 585, 283, 279, 586, 588, 588, 588, 276, 588, 275, 274, 588, 591, 591, 591, 582, 592, 592, 592, 273, 592, 270, 266, 592, 594, 594, 594, 591, 595, 595, 595, 265, 595, 264, 263, 595, 600, 600, 600, 594, 600, 262, 261, 600, 603, 603, 603, 260, 603, 257, 256, 603, 606, 606, 606, 255, 606, 254, 253, 606, 612, 612, 612, 613, 613, 613, 252, 613, 251, 250, 613, 249, 614, 614, 614, 612, 614, 248, 246, 614, 617, 617, 617, 245, 617, 244, 243, 617, 621, 621, 621, 242, 621, 238, 236, 621, 643, 643, 643, 235, 643, 234, 233, 643, 665, 665, 665, 681, 681, 681, 682, 682, 682, 683, 683, 683, 232, 231, 230, 665, 665, 229, 228, 681, 227, 682, 682, 226, 683, 683, 684, 684, 684, 225, 684, 222, 221, 684, 698, 698, 698, 699, 699, 699, 218, 217, 684, 216, 700, 700, 700, 215, 700, 214, 698, 700, 699, 699, 701, 701, 701, 213, 701, 209, 700, 701, 698, 208, 736, 736, 736, 753, 753, 753, 701, 702, 702, 702, 702, 702, 702, 702, 702, 702, 736, 736, 702, 753, 207, 702, 702, 702, 702, 702, 747, 747, 747, 204, 747, 203, 202, 747, 742, 742, 742, 201, 200, 702, 702, 703, 703, 703, 703, 703, 703, 703, 703, 703, 742, 199, 703, 198, 197, 703, 703, 703, 703, 703, 718, 718, 718, 196, 718, 195, 194, 718, 750, 750, 750, 742, 750, 703, 703, 750, 718, 721, 721, 721, 721, 721, 721, 721, 721, 721, 193, 192, 721, 191, 190, 721, 721, 721, 721, 721, 189, 188, 185, 184, 183, 182, 181, 175, 174, 743, 743, 743, 172, 721, 721, 722, 722, 722, 722, 722, 722, 722, 722, 722, 743, 169, 722, 168, 163, 722, 722, 722, 722, 722, 746, 746, 746, 754, 754, 754, 162, 754, 161, 160, 754, 159, 155, 722, 722, 746, 771, 771, 771, 754, 772, 772, 772, 149, 145, 137, 136, 135, 134, 133, 129, 121, 771, 117, 106, 105, 772, 775, 775, 775, 775, 775, 775, 775, 776, 776, 776, 776, 776, 776, 776, 777, 777, 777, 777, 777, 777, 777, 778, 778, 778, 778, 778, 778, 778, 779, 779, 779, 779, 779, 779, 779, 780, 780, 780, 780, 780, 780, 780, 781, 781, 781, 781, 781, 781, 781, 782, 782, 782, 782, 782, 782, 782, 783, 783, 783, 783, 783, 783, 783, 784, 784, 784, 784, 784, 784, 784, 785, 785, 785, 785, 785, 785, 785, 786, 786, 786, 786, 786, 786, 786, 787, 787, 787, 787, 787, 787, 787, 788, 788, 788, 788, 788, 788, 788, 789, 789, 789, 104, 789, 790, 790, 790, 103, 790, 791, 791, 791, 102, 791, 792, 792, 792, 97, 792, 793, 793, 793, 93, 793, 794, 794, 794, 794, 794, 92, 794, 795, 91, 795, 796, 85, 796, 797, 84, 797, 798, 82, 798, 799, 79, 799, 800, 800, 800, 800, 800, 800, 800, 801, 67, 0, 801, 802, 802, 802, 802, 802, 802, 803, 803, 803, 803, 803, 803, 804, 804, 804, 804, 804, 804, 805, 805, 805, 805, 805, 805, 806, 806, 806, 806, 806, 806, 807, 807, 807, 807, 807, 807, 807, 808, 808, 808, 808, 808, 808, 808, 809, 809, 809, 809, 809, 809, 809, 810, 810, 810, 810, 810, 810, 810, 811, 811, 811, 811, 811, 811, 811, 812, 812, 812, 812, 812, 812, 812, 813, 813, 813, 813, 813, 813, 813, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; /* Table of booleans, true if rule could match eol. */ static yyconst flex_int32_t yy_rule_can_match_eol[141] = { 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, }; static yy_state_type yy_last_accepting_state; static char *yy_last_accepting_cpos; extern int dax__flex_debug; int dax__flex_debug = 0; /* The intent behind this definition is that it'll catch * any uses of REJECT which flex missed. */ #define REJECT reject_used_but_not_detected #define yymore() yymore_used_but_not_detected #define YY_MORE_ADJ 0 #define YY_RESTORE_YY_MORE_OFFSET char *dax_text; /* Validating XML processor for src/simdag/dax.dtd. * * This program was generated with the FleXML XML processor generator. * FleXML is Copyright (C) 1999-2005 Kristoffer Rose. All rights reserved. * FleXML is Copyright (C) 2003-2013 Martin Quinson. All rights reserved. * (1.9.6). * * There are two, intertwined parts to this program, part A and part B. * * Part A * ------ * * Some parts, here collectively called "Part A", are found in the * FleXML package. They are Copyright (C) 1999-2005 Kristoffer Rose * and Copyright (C) 2003-2013 Martin Quinson. All rights reserved. * * You can redistribute, use, perform, display and/or modify "Part A" * provided the following two conditions hold: * * 1. The program is distributed WITHOUT ANY WARRANTY from the author of * FleXML; without even the implied warranty of MERCHANTABILITY or * FITNESS FOR A PARTICULAR PURPOSE. * * 2. The program distribution conditions do not in any way affect the * distribution conditions of the FleXML system used to generate this * file or any version of FleXML derived from that system. * * Notice that these are explicit rights granted to you for files * generated by the FleXML system. For your rights in connection with * the FleXML system itself please consult the GNU General Public License. * * Part B * ------ * * The other parts, here collectively called "Part B", and which came * from the DTD used by FleXML to generate this program, can be * distributed (or not, as the case may be) under the terms of whoever * wrote them, provided these terms respect and obey the two conditions * above under the heading "Part A". * * The author of and contributors to FleXML specifically disclaim * any copyright interest in "Part B", unless "Part B" was written * by the author of or contributors to FleXML. * */ /* Version strings. */ const char dax__flexml_version[] = "1.9.6"; /* ANSI headers. */ #include /* for realloc() -- needed here when using flex 2.5.4 */ #include #include #include #include #include #if defined(_WIN32) || defined(__WIN32__) || defined(WIN32) || defined(__TOS_WIN__) # ifndef __STRICT_ANSI__ # include # include # endif #else # include #endif #ifndef FLEXML_INDEXSTACKSIZE #define FLEXML_INDEXSTACKSIZE 1000 #endif /* Generated definitions. */ #define FLEXML_yylineno #ifndef FLEXML_BUFFERSTACKSIZE #define FLEXML_BUFFERSTACKSIZE 1000000 #endif #define FLEXML_NEED_BUFFERLIT /* XML processor api. */ /* FleXML-provided data. */ int dax__pcdata_ix; extern char *dax__bufferstack; #define dax__pcdata (dax__bufferstack + dax__pcdata_ix) AT_dax__adag_childCount AX_dax__adag_childCount; #define A_dax__adag_childCount (dax__bufferstack + AX_dax__adag_childCount) short int dax__adag_childCount_isset; AT_dax__adag_count AX_dax__adag_count; #define A_dax__adag_count (dax__bufferstack + AX_dax__adag_count) short int dax__adag_count_isset; AT_dax__adag_fileCount AX_dax__adag_fileCount; #define A_dax__adag_fileCount (dax__bufferstack + AX_dax__adag_fileCount) short int dax__adag_fileCount_isset; AT_dax__adag_index AX_dax__adag_index; #define A_dax__adag_index (dax__bufferstack + AX_dax__adag_index) short int dax__adag_index_isset; AT_dax__adag_jobCount AX_dax__adag_jobCount; #define A_dax__adag_jobCount (dax__bufferstack + AX_dax__adag_jobCount) short int dax__adag_jobCount_isset; AT_dax__adag_name AX_dax__adag_name; #define A_dax__adag_name (dax__bufferstack + AX_dax__adag_name) short int dax__adag_name_isset; AT_dax__adag_version AX_dax__adag_version; #define A_dax__adag_version (dax__bufferstack + AX_dax__adag_version) short int dax__adag_version_isset; AT_dax__adag_xmlns AX_dax__adag_xmlns; #define A_dax__adag_xmlns (dax__bufferstack + AX_dax__adag_xmlns) short int dax__adag_xmlns_isset; AT_dax__adag_xmlns_c_xsi AX_dax__adag_xmlns_c_xsi; #define A_dax__adag_xmlns_c_xsi (dax__bufferstack + AX_dax__adag_xmlns_c_xsi) short int dax__adag_xmlns_c_xsi_isset; AT_dax__adag_xsi_c_schemaLocation AX_dax__adag_xsi_c_schemaLocation; #define A_dax__adag_xsi_c_schemaLocation (dax__bufferstack + AX_dax__adag_xsi_c_schemaLocation) short int dax__adag_xsi_c_schemaLocation_isset; AT_dax__child_ref AX_dax__child_ref; #define A_dax__child_ref (dax__bufferstack + AX_dax__child_ref) short int dax__child_ref_isset; AT_dax__job_id AX_dax__job_id; #define A_dax__job_id (dax__bufferstack + AX_dax__job_id) short int dax__job_id_isset; AT_dax__job_level AX_dax__job_level; #define A_dax__job_level (dax__bufferstack + AX_dax__job_level) short int dax__job_level_isset; AT_dax__job_name AX_dax__job_name; #define A_dax__job_name (dax__bufferstack + AX_dax__job_name) short int dax__job_name_isset; AT_dax__job_namespace AX_dax__job_namespace; #define A_dax__job_namespace (dax__bufferstack + AX_dax__job_namespace) short int dax__job_namespace_isset; AT_dax__job_runtime AX_dax__job_runtime; #define A_dax__job_runtime (dax__bufferstack + AX_dax__job_runtime) short int dax__job_runtime_isset; AT_dax__job_version AX_dax__job_version; #define A_dax__job_version (dax__bufferstack + AX_dax__job_version) short int dax__job_version_isset; AT_dax__parent_ref AX_dax__parent_ref; #define A_dax__parent_ref (dax__bufferstack + AX_dax__parent_ref) short int dax__parent_ref_isset; AT_dax__uses_file AX_dax__uses_file; #define A_dax__uses_file (dax__bufferstack + AX_dax__uses_file) short int dax__uses_file_isset; AT_dax__uses_link AX_dax__uses_link; #define A_dax__uses_link AX_dax__uses_link short int dax__uses_link_isset; AT_dax__uses_optional AX_dax__uses_optional; #define A_dax__uses_optional AX_dax__uses_optional short int dax__uses_optional_isset; AT_dax__uses_register AX_dax__uses_register; #define A_dax__uses_register AX_dax__uses_register short int dax__uses_register_isset; AT_dax__uses_size AX_dax__uses_size; #define A_dax__uses_size (dax__bufferstack + AX_dax__uses_size) short int dax__uses_size_isset; AT_dax__uses_transfer AX_dax__uses_transfer; #define A_dax__uses_transfer AX_dax__uses_transfer short int dax__uses_transfer_isset; AT_dax__uses_type AX_dax__uses_type; #define A_dax__uses_type (dax__bufferstack + AX_dax__uses_type) short int dax__uses_type_isset; /* XML state. */ #ifdef FLEX_DEBUG # define ENTER(state) debug_enter(state,#state) # define LEAVE debug_leave() # define SET(state) debug_set(state,#state) static void debug_enter(int, const char*); static void debug_leave(void); static void debug_set(int, const char*); #else # define ENTER(state) (yy_push_state(state)) # define LEAVE (yy_pop_state()) # define SET(state) BEGIN(state) #endif /* Generic actions. */ #define SKIP /*skip*/ #define SUCCEED CLEANUP; return 0 #define FAIL return fail static int fail(const char*, ...); enum {flexml_max_err_msg_size = 512}; static char flexml_err_msg[flexml_max_err_msg_size]; const char * dax__parse_err_msg() { return flexml_err_msg; } static void reset_dax__parse_err_msg() { flexml_err_msg[0] = '\0'; } /* Cleanup */ static void cleanup(void); #define CLEANUP cleanup() /* Text buffer stack handling. */ char *dax__bufferstack = NULL; static int blimit = FLEXML_BUFFERSTACKSIZE; static int bnext = 1; static int *indexstack = NULL; static int ilimit = FLEXML_INDEXSTACKSIZE; static int inext = 1; #define BUFFERSET(P) (P = bnext) #define BUFFERPUTC(C) (ck_blimit(), dax__bufferstack[bnext++] = (C)) #define BUFFERDONE (BUFFERPUTC('\0')) #define BUFFERLITERAL(C, P) dax__bufferliteral(C, &(P), dax_text) /* after this is called, there are at least 2 slots left in the stack */ static int ck_blimit() { if (bnext >= blimit) { blimit += FLEXML_BUFFERSTACKSIZE + 2; { char *temp = (char *) realloc(dax__bufferstack, blimit); assert(temp); dax__bufferstack = temp; } } return 0; } /* after this is called, there are at least 2 slots left in the stack */ static int ck_ilimit() { if (inext >= ilimit) { ilimit += FLEXML_INDEXSTACKSIZE + 2; { int *temp = (int *) realloc(indexstack, ilimit); assert(temp); indexstack = temp; } } return 0; } #ifdef FLEXML_NEED_BUFFERLIT static void dax__bufferliteral(char c, int* pp, const char* text) { BUFFERSET(*pp); if (c) { const char *s = strchr(text, c), *e = strrchr(text, c); assert(s && e && s <= e); ++s; while (s < e) { if (isspace(*s)) { BUFFERPUTC(' '); do ++s; while (s < e && isspace(*s)); } else BUFFERPUTC(*s++); } } else { const char *s = text; while (*s) BUFFERPUTC(*s++); } BUFFERDONE; } #endif static void pushbuffer(int p) { ck_ilimit(); indexstack[inext++] = p; indexstack[inext++] = bnext; } static int popbuffer(void) { assert(inext >= 2); bnext = indexstack[--inext]; return indexstack[--inext]; } /* General internal entities are `unput' back onto the input stream... */ #define ENTITYTEXT(T) \ { char *s = (T), *e = s+strlen(s);\ while (--e >= s) { unput(*e); }} /* Flex standard options. */ #define YY_NO_INPUT 1 /* Flex user-requested options. */ /* XML character classes (currently restricted to ASCII). */ /* "Common syntactic structures." */ /* "Names and Tokens." */ /* Miscellaneous. */ /* Parser states (flex `exclusive start conditions'): * * PROLOG the XML prolog of the document before * DOCTYPE the XML prolog of the document after * EPILOG after the root element * INCOMMENT inside an XML comment * INPI inside an XML PI * VALUE1 inside a '...'-delimited literal * VALUE2 inside a "..."-delimited literal * CDATA inside a section. * ROOT_ expect root element * AL_ inside the attribute list for * IN_ inside a with element contents (ready for end tag) * IMPOSSIBLE dummy to permit disabling rules; must be last */ /* State names. */ const char* *dax__statenames=NULL; #define INITIAL 0 #define PROLOG 1 #define DOCTYPE 2 #define EPILOG 3 #define INCOMMENT 4 #define INPI 5 #define VALUE1 6 #define VALUE2 7 #define CDATA 8 #define ROOT_dax__adag 9 #define AL_dax__adag 10 #define S_dax__adag 11 #define S_dax__adag_1 12 #define S_dax__adag_2 13 #define S_dax__adag_3 14 #define S_dax__adag_4 15 #define S_dax__adag_5 16 #define E_dax__adag 17 #define AL_dax__child 18 #define S_dax__child 19 #define S_dax__child_1 20 #define S_dax__child_2 21 #define E_dax__child 22 #define AL_dax__job 23 #define S_dax__job 24 #define S_dax__job_1 25 #define S_dax__job_2 26 #define E_dax__job 27 #define AL_dax__parent 28 #define E_dax__parent 29 #define AL_dax__uses 30 #define E_dax__uses 31 #define IMPOSSIBLE 32 #ifndef YY_NO_UNISTD_H /* Special case for "unistd.h", since it is non-ANSI. We include it way * down here because we want the user's section 1 to have been scanned first. * The user has a chance to override it with an option. */ #if defined(_WIN32) # ifndef __STRICT_ANSI__ # include # include # endif #else # include #endif #endif #ifndef YY_EXTRA_TYPE #define YY_EXTRA_TYPE void * #endif static int yy_init_globals (void ); /* Accessor methods to globals. These are made visible to non-reentrant scanners for convenience. */ int dax_lex_destroy (void ); int dax_get_debug (void ); void dax_set_debug (int debug_flag ); YY_EXTRA_TYPE dax_get_extra (void ); void dax_set_extra (YY_EXTRA_TYPE user_defined ); FILE *dax_get_in (void ); void dax_set_in (FILE * _in_str ); FILE *dax_get_out (void ); void dax_set_out (FILE * _out_str ); int dax_get_leng (void ); char *dax_get_text (void ); int dax_get_lineno (void ); void dax_set_lineno (int _line_number ); /* Macros after this point can all be overridden by user definitions in * section 1. */ #ifndef YY_SKIP_YYWRAP #ifdef __cplusplus extern "C" int dax_wrap (void ); #else extern int dax_wrap (void ); #endif #endif #ifndef YY_NO_UNPUT #endif #ifndef yytext_ptr static void yy_flex_strncpy (char *,yyconst char *,int ); #endif #ifdef YY_NEED_STRLEN static int yy_flex_strlen (yyconst char * ); #endif #ifndef YY_NO_INPUT #ifdef __cplusplus static int yyinput (void ); #else static int input (void ); #endif #endif static int yy_start_stack_ptr = 0; static int yy_start_stack_depth = 0; static int *yy_start_stack = NULL; static void yy_push_state (int _new_state ); static void yy_pop_state (void ); /* Amount of stuff to slurp up with each read. */ #ifndef YY_READ_BUF_SIZE #ifdef __ia64__ /* On IA-64, the buffer size is 16k, not 8k */ #define YY_READ_BUF_SIZE 16384 #else #define YY_READ_BUF_SIZE 8192 #endif /* __ia64__ */ #endif /* Copy whatever the last rule matched to the standard output. */ #ifndef ECHO /* This used to be an fputs(), but since the string might contain NUL's, * we now use fwrite(). */ #define ECHO do { if (fwrite( dax_text, (size_t) dax_leng, 1, dax_out )) {} } while (0) #endif /* Gets input and stuffs it into "buf". number of characters read, or YY_NULL, * is returned in "result". */ #ifndef YY_INPUT #define YY_INPUT(buf,result,max_size) \ if ( YY_CURRENT_BUFFER_LVALUE->yy_is_interactive ) \ { \ int c = '*'; \ ssize_t n; \ for ( n = 0; n < max_size && \ (c = getc( dax_in )) != EOF && c != '\n'; ++n ) \ buf[n] = (char) c; \ if ( c == '\n' ) \ buf[n++] = (char) c; \ if ( c == EOF && ferror( dax_in ) ) \ YY_FATAL_ERROR( "input in flex scanner failed" ); \ result = n; \ } \ else \ { \ errno=0; \ while ( (result = (int) fread(buf, 1, max_size, dax_in))==0 && ferror(dax_in)) \ { \ if( errno != EINTR) \ { \ YY_FATAL_ERROR( "input in flex scanner failed" ); \ break; \ } \ errno=0; \ clearerr(dax_in); \ } \ }\ \ #endif /* No semi-colon after return; correct usage is to write "yyterminate();" - * we don't want an extra ';' after the "return" because that will cause * some compilers to complain about unreachable statements. */ #ifndef yyterminate #define yyterminate() return YY_NULL #endif /* Number of entries by which start-condition stack grows. */ #ifndef YY_START_STACK_INCR #define YY_START_STACK_INCR 25 #endif /* Report a fatal error. */ #ifndef YY_FATAL_ERROR #define YY_FATAL_ERROR(msg) yy_fatal_error( msg ) #endif /* end tables serialization structures and prototypes */ /* Default declaration of generated scanner - a define so the user can * easily add parameters. */ #ifndef YY_DECL #define YY_DECL_IS_OURS 1 extern int dax_lex (void); #define YY_DECL int dax_lex (void) #endif /* !YY_DECL */ /* Code executed at the beginning of each rule, after dax_text and dax_leng * have been set up. */ #ifndef YY_USER_ACTION #define YY_USER_ACTION #endif /* Code executed at the end of each rule. */ #ifndef YY_BREAK #define YY_BREAK /*LINTED*/break; #endif #define YY_RULE_SETUP \ YY_USER_ACTION /** The main scanner function which does all the work. */ YY_DECL { yy_state_type yy_current_state; char *yy_cp, *yy_bp; int yy_act; if ( !(yy_init) ) { (yy_init) = 1; #ifdef YY_USER_INIT YY_USER_INIT; #endif if ( ! (yy_start) ) (yy_start) = 1; /* first start state */ if ( ! dax_in ) dax_in = stdin; if ( ! dax_out ) dax_out = stdout; if ( ! YY_CURRENT_BUFFER ) { dax_ensure_buffer_stack (); YY_CURRENT_BUFFER_LVALUE = dax__create_buffer(dax_in,YY_BUF_SIZE ); } dax__load_buffer_state( ); } { /* Bypass Flex's default INITIAL state and begin by parsing the XML prolog. */ SET(PROLOG); reset_dax__parse_err_msg(); dax__bufferstack = (char *) malloc(FLEXML_BUFFERSTACKSIZE); assert(dax__bufferstack); #ifdef FLEX_DEBUG { int i; for (i = 0; i < blimit; i++) { dax__bufferstack[i] = '\377'; } } #endif dax__bufferstack[0] = '\0'; indexstack = (int *) malloc(FLEXML_INDEXSTACKSIZE * sizeof(int)); assert(indexstack); indexstack[0] = 0; /* FleXML_init */ bnext = inext = 1; dax__bufferliteral('\0', &bnext, "http://pegasus.isi.edu/schema/DAX"); dax__bufferliteral('\0', &bnext, "http://www.w3.org/2001/XMLSchema-instance"); dax__bufferliteral('\0', &bnext, "http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-2.1.xsd"); dax__bufferliteral('\0', &bnext, "1.0"); dax__bufferliteral('\0', &bnext, "0.0"); dax__bufferliteral('\0', &bnext, "data"); if(!dax__statenames) {dax__statenames= (const char **)calloc(IMPOSSIBLE,sizeof(char*)); dax__statenames[PROLOG] = NULL; dax__statenames[DOCTYPE] = NULL; dax__statenames[EPILOG] = NULL; dax__statenames[INCOMMENT] = NULL; dax__statenames[INPI] = NULL; dax__statenames[VALUE1] = NULL; dax__statenames[VALUE2] = NULL; dax__statenames[CDATA] = NULL; dax__statenames[ROOT_dax__adag] = NULL; dax__statenames[AL_dax__adag] = NULL; dax__statenames[S_dax__adag] = "adag"; dax__statenames[S_dax__adag_1] = "adag"; dax__statenames[S_dax__adag_2] = "adag"; dax__statenames[S_dax__adag_3] = "adag"; dax__statenames[S_dax__adag_4] = "adag"; dax__statenames[S_dax__adag_5] = "adag"; dax__statenames[E_dax__adag] = "adag"; dax__statenames[AL_dax__child] = NULL; dax__statenames[S_dax__child] = "child"; dax__statenames[S_dax__child_1] = "child"; dax__statenames[S_dax__child_2] = "child"; dax__statenames[E_dax__child] = "child"; dax__statenames[AL_dax__job] = NULL; dax__statenames[S_dax__job] = "job"; dax__statenames[S_dax__job_1] = "job"; dax__statenames[S_dax__job_2] = "job"; dax__statenames[E_dax__job] = "job"; dax__statenames[AL_dax__parent] = NULL; dax__statenames[E_dax__parent] = "parent"; dax__statenames[AL_dax__uses] = NULL; dax__statenames[E_dax__uses] = "uses"; } /* COMMENTS and PIs: handled uniformly for efficiency. */ while ( /*CONSTCOND*/1 ) /* loops until end-of-file is reached */ { yy_cp = (yy_c_buf_p); /* Support of dax_text. */ *yy_cp = (yy_hold_char); /* yy_bp points to the position in yy_ch_buf of the start of * the current run. */ yy_bp = yy_cp; yy_current_state = (yy_start); yy_match: do { YY_CHAR yy_c = yy_ec[YY_SC_TO_UI(*yy_cp)] ; if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (flex_int16_t) yy_c]; ++yy_cp; } while ( yy_base[yy_current_state] != 2135 ); yy_find_action: yy_act = yy_accept[yy_current_state]; if ( yy_act == 0 ) { /* have to back up */ yy_cp = (yy_last_accepting_cpos); yy_current_state = (yy_last_accepting_state); yy_act = yy_accept[yy_current_state]; } YY_DO_BEFORE_ACTION; if ( yy_act != YY_END_OF_BUFFER && yy_rule_can_match_eol[yy_act] ) { unsigned int yyl; for ( yyl = 0; yyl < dax_leng; ++yyl ) if ( dax_text[yyl] == '\n' ) dax_lineno++; ; } do_action: /* This label is used only to access EOF actions. */ switch ( yy_act ) { /* beginning of action switch */ case 0: /* must back up */ /* undo the effects of YY_DO_BEFORE_ACTION */ *yy_cp = (yy_hold_char); yy_cp = (yy_last_accepting_cpos); yy_current_state = (yy_last_accepting_state); goto yy_find_action; case 1: YY_RULE_SETUP ENTER(INCOMMENT); YY_BREAK case 2: YY_RULE_SETUP ENTER(INPI); YY_BREAK case 3: YY_RULE_SETUP LEAVE; YY_BREAK case 4: case 5: case 6: /* rule 6 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK case YY_STATE_EOF(INCOMMENT): FAIL("EOF in comment."); YY_BREAK case 7: YY_RULE_SETUP LEAVE; YY_BREAK case 8: case 9: /* rule 9 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK case YY_STATE_EOF(INPI): FAIL("EOF in PI (processing instruction)."); YY_BREAK /* SPACES: skipped uniformly */ case 10: /* rule 10 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK /* PROLOG: determine root element and process it. */ case 11: /* rule 11 can match eol */ YY_RULE_SETUP SET(ROOT_dax__adag); YY_BREAK case 12: /* rule 12 can match eol */ YY_RULE_SETUP FAIL("Bad declaration %s.",dax_text); YY_BREAK case 13: /* rule 13 can match eol */ YY_RULE_SETUP SET(ROOT_dax__adag); YY_BREAK case 14: /* rule 14 can match eol */ YY_RULE_SETUP FAIL("Bad declaration %s.",dax_text); YY_BREAK case 15: YY_RULE_SETUP FAIL("Unexpected character `%c' in prolog.", dax_text[0]); YY_BREAK case YY_STATE_EOF(PROLOG): case YY_STATE_EOF(DOCTYPE): FAIL("EOF in prolog."); YY_BREAK /* RULES DERIVED FROM DTD. */ /* */ case 16: /* rule 16 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 17: /* rule 17 can match eol */ YY_RULE_SETUP { AX_dax__adag_childCount = 0; dax__adag_childCount_isset = 0; AX_dax__adag_count = 0; dax__adag_count_isset = 0; AX_dax__adag_fileCount = 0; dax__adag_fileCount_isset = 0; AX_dax__adag_index = 0; dax__adag_index_isset = 0; AX_dax__adag_jobCount = 0; dax__adag_jobCount_isset = 0; AX_dax__adag_name = 0; dax__adag_name_isset = 0; AX_dax__adag_version = 153; dax__adag_version_isset = 0; AX_dax__adag_xmlns = 1; dax__adag_xmlns_isset = 0; AX_dax__adag_xmlns_c_xsi = 35; dax__adag_xmlns_c_xsi_isset = 0; AX_dax__adag_xsi_c_schemaLocation = 77; dax__adag_xsi_c_schemaLocation_isset = 0; ENTER(AL_dax__adag); pushbuffer(0); } YY_BREAK case 18: /* rule 18 can match eol */ YY_RULE_SETUP if (dax__adag_childCount_isset != 0) {FAIL("Multiple definition of attribute childCount in ");} dax__adag_childCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_childCount); YY_BREAK case 19: /* rule 19 can match eol */ YY_RULE_SETUP if (dax__adag_childCount_isset != 0) {FAIL("Multiple definition of attribute childCount in ");} dax__adag_childCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_childCount); YY_BREAK case 20: /* rule 20 can match eol */ YY_RULE_SETUP if (dax__adag_count_isset != 0) {FAIL("Multiple definition of attribute count in ");} dax__adag_count_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_count); YY_BREAK case 21: /* rule 21 can match eol */ YY_RULE_SETUP if (dax__adag_count_isset != 0) {FAIL("Multiple definition of attribute count in ");} dax__adag_count_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_count); YY_BREAK case 22: /* rule 22 can match eol */ YY_RULE_SETUP if (dax__adag_fileCount_isset != 0) {FAIL("Multiple definition of attribute fileCount in ");} dax__adag_fileCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_fileCount); YY_BREAK case 23: /* rule 23 can match eol */ YY_RULE_SETUP if (dax__adag_fileCount_isset != 0) {FAIL("Multiple definition of attribute fileCount in ");} dax__adag_fileCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_fileCount); YY_BREAK case 24: /* rule 24 can match eol */ YY_RULE_SETUP if (dax__adag_index_isset != 0) {FAIL("Multiple definition of attribute index in ");} dax__adag_index_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_index); YY_BREAK case 25: /* rule 25 can match eol */ YY_RULE_SETUP if (dax__adag_index_isset != 0) {FAIL("Multiple definition of attribute index in ");} dax__adag_index_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_index); YY_BREAK case 26: /* rule 26 can match eol */ YY_RULE_SETUP if (dax__adag_jobCount_isset != 0) {FAIL("Multiple definition of attribute jobCount in ");} dax__adag_jobCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_jobCount); YY_BREAK case 27: /* rule 27 can match eol */ YY_RULE_SETUP if (dax__adag_jobCount_isset != 0) {FAIL("Multiple definition of attribute jobCount in ");} dax__adag_jobCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_jobCount); YY_BREAK case 28: /* rule 28 can match eol */ YY_RULE_SETUP if (dax__adag_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__adag_name_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_name); YY_BREAK case 29: /* rule 29 can match eol */ YY_RULE_SETUP if (dax__adag_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__adag_name_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_name); YY_BREAK case 30: /* rule 30 can match eol */ YY_RULE_SETUP if (dax__adag_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__adag_version_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_version); YY_BREAK case 31: /* rule 31 can match eol */ YY_RULE_SETUP if (dax__adag_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__adag_version_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_version); YY_BREAK case 32: /* rule 32 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_isset != 0) {FAIL("Multiple definition of attribute xmlns in ");} dax__adag_xmlns_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xmlns); YY_BREAK case 33: /* rule 33 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_isset != 0) {FAIL("Multiple definition of attribute xmlns in ");} dax__adag_xmlns_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xmlns); YY_BREAK case 34: /* rule 34 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_c_xsi_isset != 0) {FAIL("Multiple definition of attribute xmlns:xsi in ");} dax__adag_xmlns_c_xsi_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xmlns_c_xsi); YY_BREAK case 35: /* rule 35 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_c_xsi_isset != 0) {FAIL("Multiple definition of attribute xmlns:xsi in ");} dax__adag_xmlns_c_xsi_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xmlns_c_xsi); YY_BREAK case 36: /* rule 36 can match eol */ YY_RULE_SETUP if (dax__adag_xsi_c_schemaLocation_isset != 0) {FAIL("Multiple definition of attribute xsi:schemaLocation in ");} dax__adag_xsi_c_schemaLocation_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xsi_c_schemaLocation); YY_BREAK case 37: /* rule 37 can match eol */ YY_RULE_SETUP if (dax__adag_xsi_c_schemaLocation_isset != 0) {FAIL("Multiple definition of attribute xsi:schemaLocation in ");} dax__adag_xsi_c_schemaLocation_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xsi_c_schemaLocation); YY_BREAK case 38: YY_RULE_SETUP { LEAVE; STag_dax__adag();dax__pcdata_ix = 0; ENTER(S_dax__adag); } YY_BREAK case 39: YY_RULE_SETUP { LEAVE; STag_dax__adag(); dax__pcdata_ix = 0; ETag_dax__adag(); popbuffer(); /* attribute */ switch (YY_START) { case ROOT_dax__adag: SET(EPILOG); break; } } YY_BREAK case 40: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of adag element.", dax_text[0]); YY_BREAK case 41: YY_RULE_SETUP FAIL("Bad attribute `%s' in `adag' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__adag): FAIL("EOF in attribute list of `adag' element."); YY_BREAK case 42: /* rule 42 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__adag(); popbuffer(); /* attribute */ switch (YY_START) { case ROOT_dax__adag: SET(EPILOG); break; } } YY_BREAK case 43: /* rule 43 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 44: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__adag): case YY_STATE_EOF(S_dax__adag): case YY_STATE_EOF(S_dax__adag_1): case YY_STATE_EOF(S_dax__adag_3): case YY_STATE_EOF(S_dax__adag_5): FAIL("Premature EOF: `' expected."); YY_BREAK case 45: /* rule 45 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 46: /* rule 46 can match eol */ YY_RULE_SETUP { AX_dax__child_ref = 0; dax__child_ref_isset = 0; ENTER(AL_dax__child); pushbuffer(0); } YY_BREAK case 47: /* rule 47 can match eol */ YY_RULE_SETUP if (dax__child_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__child_ref_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__child_ref); YY_BREAK case 48: /* rule 48 can match eol */ YY_RULE_SETUP if (dax__child_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__child_ref_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__child_ref); YY_BREAK case 49: YY_RULE_SETUP { if (!AX_dax__child_ref) FAIL("Required attribute `ref' not set for `child' element."); LEAVE; STag_dax__child();dax__pcdata_ix = 0; ENTER(S_dax__child); } YY_BREAK case 50: YY_RULE_SETUP { if (!AX_dax__child_ref) FAIL("Required attribute `ref' not set for `child' element."); LEAVE; STag_dax__child(); dax__pcdata_ix = 0; ETag_dax__child(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_1: case S_dax__adag_3: case S_dax__adag_4: case S_dax__adag_5: SET(S_dax__adag_5); break; } } YY_BREAK case 51: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of child element.", dax_text[0]); YY_BREAK case 52: YY_RULE_SETUP FAIL("Bad attribute `%s' in `child' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__child): FAIL("EOF in attribute list of `child' element."); YY_BREAK case 53: /* rule 53 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__child(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_1: case S_dax__adag_3: case S_dax__adag_4: case S_dax__adag_5: SET(S_dax__adag_5); break; } } YY_BREAK case 54: /* rule 54 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 55: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__child): case YY_STATE_EOF(S_dax__child): case YY_STATE_EOF(S_dax__child_2): FAIL("Premature EOF: `' expected."); YY_BREAK case 56: /* rule 56 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 57: /* rule 57 can match eol */ YY_RULE_SETUP { AX_dax__job_id = 0; dax__job_id_isset = 0; AX_dax__job_level = 0; dax__job_level_isset = 0; AX_dax__job_name = 0; dax__job_name_isset = 0; AX_dax__job_namespace = 0; dax__job_namespace_isset = 0; AX_dax__job_runtime = 0; dax__job_runtime_isset = 0; AX_dax__job_version = 157; dax__job_version_isset = 0; ENTER(AL_dax__job); pushbuffer(0); } YY_BREAK case 58: /* rule 58 can match eol */ YY_RULE_SETUP if (dax__job_id_isset != 0) {FAIL("Multiple definition of attribute id in ");} dax__job_id_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_id); YY_BREAK case 59: /* rule 59 can match eol */ YY_RULE_SETUP if (dax__job_id_isset != 0) {FAIL("Multiple definition of attribute id in ");} dax__job_id_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_id); YY_BREAK case 60: /* rule 60 can match eol */ YY_RULE_SETUP if (dax__job_level_isset != 0) {FAIL("Multiple definition of attribute level in ");} dax__job_level_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_level); YY_BREAK case 61: /* rule 61 can match eol */ YY_RULE_SETUP if (dax__job_level_isset != 0) {FAIL("Multiple definition of attribute level in ");} dax__job_level_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_level); YY_BREAK case 62: /* rule 62 can match eol */ YY_RULE_SETUP if (dax__job_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__job_name_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_name); YY_BREAK case 63: /* rule 63 can match eol */ YY_RULE_SETUP if (dax__job_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__job_name_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_name); YY_BREAK case 64: /* rule 64 can match eol */ YY_RULE_SETUP if (dax__job_namespace_isset != 0) {FAIL("Multiple definition of attribute namespace in ");} dax__job_namespace_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_namespace); YY_BREAK case 65: /* rule 65 can match eol */ YY_RULE_SETUP if (dax__job_namespace_isset != 0) {FAIL("Multiple definition of attribute namespace in ");} dax__job_namespace_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_namespace); YY_BREAK case 66: /* rule 66 can match eol */ YY_RULE_SETUP if (dax__job_runtime_isset != 0) {FAIL("Multiple definition of attribute runtime in ");} dax__job_runtime_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_runtime); YY_BREAK case 67: /* rule 67 can match eol */ YY_RULE_SETUP if (dax__job_runtime_isset != 0) {FAIL("Multiple definition of attribute runtime in ");} dax__job_runtime_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_runtime); YY_BREAK case 68: /* rule 68 can match eol */ YY_RULE_SETUP if (dax__job_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__job_version_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_version); YY_BREAK case 69: /* rule 69 can match eol */ YY_RULE_SETUP if (dax__job_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__job_version_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_version); YY_BREAK case 70: YY_RULE_SETUP { if (!AX_dax__job_id) FAIL("Required attribute `id' not set for `job' element."); if (!AX_dax__job_name) FAIL("Required attribute `name' not set for `job' element."); if (!AX_dax__job_runtime) FAIL("Required attribute `runtime' not set for `job' element."); LEAVE; STag_dax__job();dax__pcdata_ix = 0; ENTER(S_dax__job); } YY_BREAK case 71: YY_RULE_SETUP { if (!AX_dax__job_id) FAIL("Required attribute `id' not set for `job' element."); if (!AX_dax__job_name) FAIL("Required attribute `name' not set for `job' element."); if (!AX_dax__job_runtime) FAIL("Required attribute `runtime' not set for `job' element."); LEAVE; STag_dax__job(); dax__pcdata_ix = 0; ETag_dax__job(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_2: case S_dax__adag_3: SET(S_dax__adag_3); break; } } YY_BREAK case 72: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of job element.", dax_text[0]); YY_BREAK case 73: YY_RULE_SETUP FAIL("Bad attribute `%s' in `job' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__job): FAIL("EOF in attribute list of `job' element."); YY_BREAK case 74: /* rule 74 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__job(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_2: case S_dax__adag_3: SET(S_dax__adag_3); break; } } YY_BREAK case 75: /* rule 75 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 76: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__job): case YY_STATE_EOF(S_dax__job): case YY_STATE_EOF(S_dax__job_2): FAIL("Premature EOF: `' expected."); YY_BREAK case 77: /* rule 77 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 78: /* rule 78 can match eol */ YY_RULE_SETUP { AX_dax__parent_ref = 0; dax__parent_ref_isset = 0; ENTER(AL_dax__parent); pushbuffer(0); } YY_BREAK case 79: /* rule 79 can match eol */ YY_RULE_SETUP if (dax__parent_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__parent_ref_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__parent_ref); YY_BREAK case 80: /* rule 80 can match eol */ YY_RULE_SETUP if (dax__parent_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__parent_ref_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__parent_ref); YY_BREAK case 81: YY_RULE_SETUP { if (!AX_dax__parent_ref) FAIL("Required attribute `ref' not set for `parent' element."); LEAVE; STag_dax__parent();dax__pcdata_ix = 0; ENTER(E_dax__parent); } YY_BREAK case 82: YY_RULE_SETUP { if (!AX_dax__parent_ref) FAIL("Required attribute `ref' not set for `parent' element."); LEAVE; STag_dax__parent(); dax__pcdata_ix = 0; ETag_dax__parent(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__child: case S_dax__child_1: case S_dax__child_2: SET(S_dax__child_2); break; } } YY_BREAK case 83: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of parent element.", dax_text[0]); YY_BREAK case 84: YY_RULE_SETUP FAIL("Bad attribute `%s' in `parent' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__parent): FAIL("EOF in attribute list of `parent' element."); YY_BREAK case 85: /* rule 85 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__parent(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__child: case S_dax__child_1: case S_dax__child_2: SET(S_dax__child_2); break; } } YY_BREAK case 86: /* rule 86 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 87: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__parent): FAIL("Premature EOF: `' expected."); YY_BREAK /* * */ case 88: /* rule 88 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 89: /* rule 89 can match eol */ YY_RULE_SETUP { AX_dax__uses_file = 0; dax__uses_file_isset = 0; AX_dax__uses_link = AU_dax__uses_link; dax__uses_link_isset = 0; AX_dax__uses_optional = A_dax__uses_optional_false; dax__uses_optional_isset = 0; AX_dax__uses_register = A_dax__uses_register_true; dax__uses_register_isset = 0; AX_dax__uses_size = 0; dax__uses_size_isset = 0; AX_dax__uses_transfer = A_dax__uses_transfer_true; dax__uses_transfer_isset = 0; AX_dax__uses_type = 161; dax__uses_type_isset = 0; ENTER(AL_dax__uses); pushbuffer(0); } YY_BREAK case 90: /* rule 90 can match eol */ YY_RULE_SETUP if (dax__uses_file_isset != 0) {FAIL("Multiple definition of attribute file in ");} dax__uses_file_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_file); YY_BREAK case 91: /* rule 91 can match eol */ YY_RULE_SETUP if (dax__uses_file_isset != 0) {FAIL("Multiple definition of attribute file in ");} dax__uses_file_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_file); YY_BREAK case 92: /* rule 92 can match eol */ case 93: /* rule 93 can match eol */ YY_RULE_SETUP A_dax__uses_link = A_dax__uses_link_input; YY_BREAK case 94: /* rule 94 can match eol */ case 95: /* rule 95 can match eol */ YY_RULE_SETUP A_dax__uses_link = A_dax__uses_link_output; YY_BREAK case 96: /* rule 96 can match eol */ case 97: /* rule 97 can match eol */ YY_RULE_SETUP A_dax__uses_optional = A_dax__uses_optional_false; YY_BREAK case 98: /* rule 98 can match eol */ case 99: /* rule 99 can match eol */ YY_RULE_SETUP A_dax__uses_optional = A_dax__uses_optional_true; YY_BREAK case 100: /* rule 100 can match eol */ case 101: /* rule 101 can match eol */ YY_RULE_SETUP A_dax__uses_register = A_dax__uses_register_false; YY_BREAK case 102: /* rule 102 can match eol */ case 103: /* rule 103 can match eol */ YY_RULE_SETUP A_dax__uses_register = A_dax__uses_register_true; YY_BREAK case 104: /* rule 104 can match eol */ YY_RULE_SETUP if (dax__uses_size_isset != 0) {FAIL("Multiple definition of attribute size in ");} dax__uses_size_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_size); YY_BREAK case 105: /* rule 105 can match eol */ YY_RULE_SETUP if (dax__uses_size_isset != 0) {FAIL("Multiple definition of attribute size in ");} dax__uses_size_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_size); YY_BREAK case 106: /* rule 106 can match eol */ case 107: /* rule 107 can match eol */ YY_RULE_SETUP A_dax__uses_transfer = A_dax__uses_transfer_false; YY_BREAK case 108: /* rule 108 can match eol */ case 109: /* rule 109 can match eol */ YY_RULE_SETUP A_dax__uses_transfer = A_dax__uses_transfer_true; YY_BREAK case 110: /* rule 110 can match eol */ YY_RULE_SETUP if (dax__uses_type_isset != 0) {FAIL("Multiple definition of attribute type in ");} dax__uses_type_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_type); YY_BREAK case 111: /* rule 111 can match eol */ YY_RULE_SETUP if (dax__uses_type_isset != 0) {FAIL("Multiple definition of attribute type in ");} dax__uses_type_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_type); YY_BREAK case 112: YY_RULE_SETUP { if (!AX_dax__uses_file) FAIL("Required attribute `file' not set for `uses' element."); if (!AX_dax__uses_size) FAIL("Required attribute `size' not set for `uses' element."); LEAVE; STag_dax__uses();dax__pcdata_ix = 0; ENTER(E_dax__uses); } YY_BREAK case 113: YY_RULE_SETUP { if (!AX_dax__uses_file) FAIL("Required attribute `file' not set for `uses' element."); if (!AX_dax__uses_size) FAIL("Required attribute `size' not set for `uses' element."); LEAVE; STag_dax__uses(); dax__pcdata_ix = 0; ETag_dax__uses(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__job: case S_dax__job_1: case S_dax__job_2: SET(S_dax__job_2); break; } } YY_BREAK case 114: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of uses element.", dax_text[0]); YY_BREAK case 115: YY_RULE_SETUP FAIL("Bad attribute `%s' in `uses' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__uses): FAIL("EOF in attribute list of `uses' element."); YY_BREAK case 116: /* rule 116 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__uses(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__job: case S_dax__job_1: case S_dax__job_2: SET(S_dax__job_2); break; } } YY_BREAK case 117: /* rule 117 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 118: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__uses): FAIL("Premature EOF: `' expected."); YY_BREAK /* EPILOG: after the root element. */ case 119: YY_RULE_SETUP {SET(PROLOG); yyless(0); CLEANUP; return -1;} YY_BREAK case YY_STATE_EOF(EPILOG): SUCCEED; YY_BREAK /* CHARACTER DATA. */ /* Non-defined standard entities... */ case 120: YY_RULE_SETUP BUFFERPUTC('&'); YY_BREAK case 121: YY_RULE_SETUP BUFFERPUTC('<'); YY_BREAK case 122: YY_RULE_SETUP BUFFERPUTC('>'); YY_BREAK case 123: YY_RULE_SETUP BUFFERPUTC('\''); YY_BREAK case 124: YY_RULE_SETUP BUFFERPUTC('"'); YY_BREAK /* Character entities. */ case 125: YY_RULE_SETUP BUFFERPUTC((unsigned char)atoi(dax_text+2)); YY_BREAK case 126: YY_RULE_SETUP BUFFERPUTC((unsigned char)strtol(dax_text+3,NULL,16)); YY_BREAK case 127: /* rule 127 can match eol */ case 128: /* rule 128 can match eol */ case 129: /* rule 129 can match eol */ case 130: /* rule 130 can match eol */ YY_RULE_SETUP BUFFERPUTC('\n'); YY_BREAK case 131: YY_RULE_SETUP ENTER(CDATA); YY_BREAK case 132: YY_RULE_SETUP FAIL("Unexpected `]""]>' in character data."); YY_BREAK case 133: YY_RULE_SETUP BUFFERDONE; LEAVE; YY_BREAK case YY_STATE_EOF(VALUE1): FAIL("EOF in literal (\"'\" expected)."); YY_BREAK case 134: YY_RULE_SETUP BUFFERDONE; LEAVE; YY_BREAK case YY_STATE_EOF(VALUE2): FAIL("EOF in literal (`\"' expected)."); YY_BREAK case 135: /* rule 135 can match eol */ YY_RULE_SETUP BUFFERPUTC(dax_text[0]); YY_BREAK case 136: YY_RULE_SETUP FAIL("Spurious `%c' in character data.",dax_text[0]); YY_BREAK case 137: YY_RULE_SETUP LEAVE; YY_BREAK /* "]""]" BUFFERPUTC(dax_text[0]); BUFFERPUTC(dax_text[1]); */ case 138: YY_RULE_SETUP BUFFERPUTC(dax_text[0]); YY_BREAK case YY_STATE_EOF(CDATA): FAIL("EOF in CDATA section."); YY_BREAK /* Impossible rules to avoid warnings from flex(1). */ /* Ideally, this should be replaced by code in flexml.pl that generates just the states not covered by other rules. */ case 139: /* rule 139 can match eol */ YY_RULE_SETUP FAIL("Syntax error on character `%c'.", dax_text[0]); YY_BREAK case 140: YY_RULE_SETUP ECHO; YY_BREAK case YY_STATE_EOF(INITIAL): case YY_STATE_EOF(ROOT_dax__adag): case YY_STATE_EOF(S_dax__adag_2): case YY_STATE_EOF(S_dax__adag_4): case YY_STATE_EOF(S_dax__child_1): case YY_STATE_EOF(S_dax__job_1): case YY_STATE_EOF(IMPOSSIBLE): yyterminate(); case YY_END_OF_BUFFER: { /* Amount of text matched not including the EOB char. */ int yy_amount_of_matched_text = (int) (yy_cp - (yytext_ptr)) - 1; /* Undo the effects of YY_DO_BEFORE_ACTION. */ *yy_cp = (yy_hold_char); YY_RESTORE_YY_MORE_OFFSET if ( YY_CURRENT_BUFFER_LVALUE->yy_buffer_status == YY_BUFFER_NEW ) { /* We're scanning a new file or input source. It's * possible that this happened because the user * just pointed dax_in at a new source and called * dax_lex(). If so, then we have to assure * consistency between YY_CURRENT_BUFFER and our * globals. Here is the right place to do so, because * this is the first action (other than possibly a * back-up) that will match for the new input source. */ (yy_n_chars) = YY_CURRENT_BUFFER_LVALUE->yy_n_chars; YY_CURRENT_BUFFER_LVALUE->yy_input_file = dax_in; YY_CURRENT_BUFFER_LVALUE->yy_buffer_status = YY_BUFFER_NORMAL; } /* Note that here we test for yy_c_buf_p "<=" to the position * of the first EOB in the buffer, since yy_c_buf_p will * already have been incremented past the NUL character * (since all states make transitions on EOB to the * end-of-buffer state). Contrast this with the test * in input(). */ if ( (yy_c_buf_p) <= &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] ) { /* This was really a NUL. */ yy_state_type yy_next_state; (yy_c_buf_p) = (yytext_ptr) + yy_amount_of_matched_text; yy_current_state = yy_get_previous_state( ); /* Okay, we're now positioned to make the NUL * transition. We couldn't have * yy_get_previous_state() go ahead and do it * for us because it doesn't know how to deal * with the possibility of jamming (and we don't * want to build jamming into it because then it * will run more slowly). */ yy_next_state = yy_try_NUL_trans( yy_current_state ); yy_bp = (yytext_ptr) + YY_MORE_ADJ; if ( yy_next_state ) { /* Consume the NUL. */ yy_cp = ++(yy_c_buf_p); yy_current_state = yy_next_state; goto yy_match; } else { yy_cp = (yy_c_buf_p); goto yy_find_action; } } else switch ( yy_get_next_buffer( ) ) { case EOB_ACT_END_OF_FILE: { (yy_did_buffer_switch_on_eof) = 0; if ( dax_wrap( ) ) { /* Note: because we've taken care in * yy_get_next_buffer() to have set up * dax_text, we can now set up * yy_c_buf_p so that if some total * hoser (like flex itself) wants to * call the scanner after we return the * YY_NULL, it'll still work - another * YY_NULL will get returned. */ (yy_c_buf_p) = (yytext_ptr) + YY_MORE_ADJ; yy_act = YY_STATE_EOF(YY_START); goto do_action; } else { if ( ! (yy_did_buffer_switch_on_eof) ) YY_NEW_FILE; } break; } case EOB_ACT_CONTINUE_SCAN: (yy_c_buf_p) = (yytext_ptr) + yy_amount_of_matched_text; yy_current_state = yy_get_previous_state( ); yy_cp = (yy_c_buf_p); yy_bp = (yytext_ptr) + YY_MORE_ADJ; goto yy_match; case EOB_ACT_LAST_MATCH: (yy_c_buf_p) = &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)]; yy_current_state = yy_get_previous_state( ); yy_cp = (yy_c_buf_p); yy_bp = (yytext_ptr) + YY_MORE_ADJ; goto yy_find_action; } break; } default: YY_FATAL_ERROR( "fatal flex scanner internal error--no action found" ); } /* end of action switch */ } /* end of scanning one token */ } /* end of user's declarations */ } /* end of dax_lex */ /* yy_get_next_buffer - try to read in a new buffer * * Returns a code representing an action: * EOB_ACT_LAST_MATCH - * EOB_ACT_CONTINUE_SCAN - continue scanning from current position * EOB_ACT_END_OF_FILE - end of file */ static int yy_get_next_buffer (void) { char *dest = YY_CURRENT_BUFFER_LVALUE->yy_ch_buf; char *source = (yytext_ptr); int number_to_move, i; int ret_val; if ( (yy_c_buf_p) > &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars) + 1] ) YY_FATAL_ERROR( "fatal flex scanner internal error--end of buffer missed" ); if ( YY_CURRENT_BUFFER_LVALUE->yy_fill_buffer == 0 ) { /* Don't try to fill the buffer, so this is an EOF. */ if ( (yy_c_buf_p) - (yytext_ptr) - YY_MORE_ADJ == 1 ) { /* We matched a single character, the EOB, so * treat this as a final EOF. */ return EOB_ACT_END_OF_FILE; } else { /* We matched some text prior to the EOB, first * process it. */ return EOB_ACT_LAST_MATCH; } } /* Try to read more data. */ /* First move last chars to start of buffer. */ number_to_move = (int) ((yy_c_buf_p) - (yytext_ptr) - 1); for ( i = 0; i < number_to_move; ++i ) *(dest++) = *(source++); if ( YY_CURRENT_BUFFER_LVALUE->yy_buffer_status == YY_BUFFER_EOF_PENDING ) /* don't do the read, it's not guaranteed to return an EOF, * just force an EOF */ YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars) = 0; else { int num_to_read = YY_CURRENT_BUFFER_LVALUE->yy_buf_size - number_to_move - 1; while ( num_to_read <= 0 ) { /* Not enough room in the buffer - grow it. */ /* just a shorter name for the current buffer */ YY_BUFFER_STATE b = YY_CURRENT_BUFFER_LVALUE; int yy_c_buf_p_offset = (int) ((yy_c_buf_p) - b->yy_ch_buf); if ( b->yy_is_our_buffer ) { int new_size = b->yy_buf_size * 2; if ( new_size <= 0 ) b->yy_buf_size += b->yy_buf_size / 8; else b->yy_buf_size *= 2; b->yy_ch_buf = (char *) /* Include room in for 2 EOB chars. */ dax_realloc((void *) b->yy_ch_buf,b->yy_buf_size + 2 ); } else /* Can't grow it, we don't own it. */ b->yy_ch_buf = NULL; if ( ! b->yy_ch_buf ) YY_FATAL_ERROR( "fatal error - scanner input buffer overflow" ); (yy_c_buf_p) = &b->yy_ch_buf[yy_c_buf_p_offset]; num_to_read = YY_CURRENT_BUFFER_LVALUE->yy_buf_size - number_to_move - 1; } if ( num_to_read > YY_READ_BUF_SIZE ) num_to_read = YY_READ_BUF_SIZE; /* Read in more data. */ YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]), (yy_n_chars), num_to_read ); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } if ( (yy_n_chars) == 0 ) { if ( number_to_move == YY_MORE_ADJ ) { ret_val = EOB_ACT_END_OF_FILE; dax_restart(dax_in ); } else { ret_val = EOB_ACT_LAST_MATCH; YY_CURRENT_BUFFER_LVALUE->yy_buffer_status = YY_BUFFER_EOF_PENDING; } } else ret_val = EOB_ACT_CONTINUE_SCAN; if (((yy_n_chars) + number_to_move) > YY_CURRENT_BUFFER_LVALUE->yy_buf_size) { /* Extend the array by 50%, plus the number we really need. */ int new_size = (yy_n_chars) + number_to_move + ((yy_n_chars) >> 1); YY_CURRENT_BUFFER_LVALUE->yy_ch_buf = (char *) dax_realloc((void *) YY_CURRENT_BUFFER_LVALUE->yy_ch_buf,new_size ); if ( ! YY_CURRENT_BUFFER_LVALUE->yy_ch_buf ) YY_FATAL_ERROR( "out of dynamic memory in yy_get_next_buffer()" ); } (yy_n_chars) += number_to_move; YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] = YY_END_OF_BUFFER_CHAR; YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars) + 1] = YY_END_OF_BUFFER_CHAR; (yytext_ptr) = &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[0]; return ret_val; } /* yy_get_previous_state - get the state just before the EOB char was reached */ static yy_state_type yy_get_previous_state (void) { yy_state_type yy_current_state; char *yy_cp; yy_current_state = (yy_start); for ( yy_cp = (yytext_ptr) + YY_MORE_ADJ; yy_cp < (yy_c_buf_p); ++yy_cp ) { YY_CHAR yy_c = (*yy_cp ? yy_ec[YY_SC_TO_UI(*yy_cp)] : 1); if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (flex_int16_t) yy_c]; } return yy_current_state; } /* yy_try_NUL_trans - try to make a transition on the NUL character * * synopsis * next_state = yy_try_NUL_trans( current_state ); */ static yy_state_type yy_try_NUL_trans (yy_state_type yy_current_state ) { int yy_is_jam; char *yy_cp = (yy_c_buf_p); YY_CHAR yy_c = 1; if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (flex_int16_t) yy_c]; yy_is_jam = (yy_current_state == 774); return yy_is_jam ? 0 : yy_current_state; } #ifndef YY_NO_UNPUT #endif #ifndef YY_NO_INPUT #ifdef __cplusplus static int yyinput (void) #else static int input (void) #endif { int c; *(yy_c_buf_p) = (yy_hold_char); if ( *(yy_c_buf_p) == YY_END_OF_BUFFER_CHAR ) { /* yy_c_buf_p now points to the character we want to return. * If this occurs *before* the EOB characters, then it's a * valid NUL; if not, then we've hit the end of the buffer. */ if ( (yy_c_buf_p) < &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] ) /* This was really a NUL. */ *(yy_c_buf_p) = '\0'; else { /* need more input */ int offset = (yy_c_buf_p) - (yytext_ptr); ++(yy_c_buf_p); switch ( yy_get_next_buffer( ) ) { case EOB_ACT_LAST_MATCH: /* This happens because yy_g_n_b() * sees that we've accumulated a * token and flags that we need to * try matching the token before * proceeding. But for input(), * there's no matching to consider. * So convert the EOB_ACT_LAST_MATCH * to EOB_ACT_END_OF_FILE. */ /* Reset buffer status. */ dax_restart(dax_in ); /*FALLTHROUGH*/ case EOB_ACT_END_OF_FILE: { if ( dax_wrap( ) ) return 0; if ( ! (yy_did_buffer_switch_on_eof) ) YY_NEW_FILE; #ifdef __cplusplus return yyinput(); #else return input(); #endif } case EOB_ACT_CONTINUE_SCAN: (yy_c_buf_p) = (yytext_ptr) + offset; break; } } } c = *(unsigned char *) (yy_c_buf_p); /* cast for 8-bit char's */ *(yy_c_buf_p) = '\0'; /* preserve dax_text */ (yy_hold_char) = *++(yy_c_buf_p); if ( c == '\n' ) dax_lineno++; ; return c; } #endif /* ifndef YY_NO_INPUT */ /** Immediately switch to a different input stream. * @param input_file A readable stream. * * @note This function does not reset the start condition to @c INITIAL . */ void dax_restart (FILE * input_file ) { if ( ! YY_CURRENT_BUFFER ){ dax_ensure_buffer_stack (); YY_CURRENT_BUFFER_LVALUE = dax__create_buffer(dax_in,YY_BUF_SIZE ); } dax__init_buffer(YY_CURRENT_BUFFER,input_file ); dax__load_buffer_state( ); } /** Switch to a different input buffer. * @param new_buffer The new input buffer. * */ void dax__switch_to_buffer (YY_BUFFER_STATE new_buffer ) { /* TODO. We should be able to replace this entire function body * with * dax_pop_buffer_state(); * dax_push_buffer_state(new_buffer); */ dax_ensure_buffer_stack (); if ( YY_CURRENT_BUFFER == new_buffer ) return; if ( YY_CURRENT_BUFFER ) { /* Flush out information for old buffer. */ *(yy_c_buf_p) = (yy_hold_char); YY_CURRENT_BUFFER_LVALUE->yy_buf_pos = (yy_c_buf_p); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } YY_CURRENT_BUFFER_LVALUE = new_buffer; dax__load_buffer_state( ); /* We don't actually know whether we did this switch during * EOF (dax_wrap()) processing, but the only time this flag * is looked at is after dax_wrap() is called, so it's safe * to go ahead and always set it. */ (yy_did_buffer_switch_on_eof) = 1; } static void dax__load_buffer_state (void) { (yy_n_chars) = YY_CURRENT_BUFFER_LVALUE->yy_n_chars; (yytext_ptr) = (yy_c_buf_p) = YY_CURRENT_BUFFER_LVALUE->yy_buf_pos; dax_in = YY_CURRENT_BUFFER_LVALUE->yy_input_file; (yy_hold_char) = *(yy_c_buf_p); } /** Allocate and initialize an input buffer state. * @param file A readable stream. * @param size The character buffer size in bytes. When in doubt, use @c YY_BUF_SIZE. * * @return the allocated buffer state. */ YY_BUFFER_STATE dax__create_buffer (FILE * file, int size ) { YY_BUFFER_STATE b; b = (YY_BUFFER_STATE) dax_alloc(sizeof( struct yy_buffer_state ) ); if ( ! b ) YY_FATAL_ERROR( "out of dynamic memory in dax__create_buffer()" ); b->yy_buf_size = (yy_size_t)size; /* yy_ch_buf has to be 2 characters longer than the size given because * we need to put in 2 end-of-buffer characters. */ b->yy_ch_buf = (char *) dax_alloc(b->yy_buf_size + 2 ); if ( ! b->yy_ch_buf ) YY_FATAL_ERROR( "out of dynamic memory in dax__create_buffer()" ); b->yy_is_our_buffer = 1; dax__init_buffer(b,file ); return b; } /** Destroy the buffer. * @param b a buffer created with dax__create_buffer() * */ void dax__delete_buffer (YY_BUFFER_STATE b ) { if ( ! b ) return; if ( b == YY_CURRENT_BUFFER ) /* Not sure if we should pop here. */ YY_CURRENT_BUFFER_LVALUE = (YY_BUFFER_STATE) 0; if ( b->yy_is_our_buffer ) dax_free((void *) b->yy_ch_buf ); dax_free((void *) b ); } /* Initializes or reinitializes a buffer. * This function is sometimes called more than once on the same buffer, * such as during a dax_restart() or at EOF. */ static void dax__init_buffer (YY_BUFFER_STATE b, FILE * file ) { int oerrno = errno; dax__flush_buffer(b ); b->yy_input_file = file; b->yy_fill_buffer = 1; /* If b is the current buffer, then dax__init_buffer was _probably_ * called from dax_restart() or through yy_get_next_buffer. * In that case, we don't want to reset the lineno or column. */ if (b != YY_CURRENT_BUFFER){ b->yy_bs_lineno = 1; b->yy_bs_column = 0; } b->yy_is_interactive = file ? (isatty( fileno(file) ) > 0) : 0; errno = oerrno; } /** Discard all buffered characters. On the next scan, YY_INPUT will be called. * @param b the buffer state to be flushed, usually @c YY_CURRENT_BUFFER. * */ void dax__flush_buffer (YY_BUFFER_STATE b ) { if ( ! b ) return; b->yy_n_chars = 0; /* We always need two end-of-buffer characters. The first causes * a transition to the end-of-buffer state. The second causes * a jam in that state. */ b->yy_ch_buf[0] = YY_END_OF_BUFFER_CHAR; b->yy_ch_buf[1] = YY_END_OF_BUFFER_CHAR; b->yy_buf_pos = &b->yy_ch_buf[0]; b->yy_at_bol = 1; b->yy_buffer_status = YY_BUFFER_NEW; if ( b == YY_CURRENT_BUFFER ) dax__load_buffer_state( ); } /** Pushes the new state onto the stack. The new state becomes * the current state. This function will allocate the stack * if necessary. * @param new_buffer The new state. * */ void dax_push_buffer_state (YY_BUFFER_STATE new_buffer ) { if (new_buffer == NULL) return; dax_ensure_buffer_stack(); /* This block is copied from dax__switch_to_buffer. */ if ( YY_CURRENT_BUFFER ) { /* Flush out information for old buffer. */ *(yy_c_buf_p) = (yy_hold_char); YY_CURRENT_BUFFER_LVALUE->yy_buf_pos = (yy_c_buf_p); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } /* Only push if top exists. Otherwise, replace top. */ if (YY_CURRENT_BUFFER) (yy_buffer_stack_top)++; YY_CURRENT_BUFFER_LVALUE = new_buffer; /* copied from dax__switch_to_buffer. */ dax__load_buffer_state( ); (yy_did_buffer_switch_on_eof) = 1; } /** Removes and deletes the top of the stack, if present. * The next element becomes the new top. * */ void dax_pop_buffer_state (void) { if (!YY_CURRENT_BUFFER) return; dax__delete_buffer(YY_CURRENT_BUFFER ); YY_CURRENT_BUFFER_LVALUE = NULL; if ((yy_buffer_stack_top) > 0) --(yy_buffer_stack_top); if (YY_CURRENT_BUFFER) { dax__load_buffer_state( ); (yy_did_buffer_switch_on_eof) = 1; } } /* Allocates the stack if it does not exist. * Guarantees space for at least one push. */ static void dax_ensure_buffer_stack (void) { int num_to_alloc; if (!(yy_buffer_stack)) { /* First allocation is just for 2 elements, since we don't know if this * scanner will even need a stack. We use 2 instead of 1 to avoid an * immediate realloc on the next call. */ num_to_alloc = 1; /* After all that talk, this was set to 1 anyways... */ (yy_buffer_stack) = (struct yy_buffer_state**)dax_alloc (num_to_alloc * sizeof(struct yy_buffer_state*) ); if ( ! (yy_buffer_stack) ) YY_FATAL_ERROR( "out of dynamic memory in dax_ensure_buffer_stack()" ); memset((yy_buffer_stack), 0, num_to_alloc * sizeof(struct yy_buffer_state*)); (yy_buffer_stack_max) = num_to_alloc; (yy_buffer_stack_top) = 0; return; } if ((yy_buffer_stack_top) >= ((yy_buffer_stack_max)) - 1){ /* Increase the buffer to prepare for a possible push. */ yy_size_t grow_size = 8 /* arbitrary grow size */; num_to_alloc = (yy_buffer_stack_max) + grow_size; (yy_buffer_stack) = (struct yy_buffer_state**)dax_realloc ((yy_buffer_stack), num_to_alloc * sizeof(struct yy_buffer_state*) ); if ( ! (yy_buffer_stack) ) YY_FATAL_ERROR( "out of dynamic memory in dax_ensure_buffer_stack()" ); /* zero only the new slots.*/ memset((yy_buffer_stack) + (yy_buffer_stack_max), 0, grow_size * sizeof(struct yy_buffer_state*)); (yy_buffer_stack_max) = num_to_alloc; } } /** Setup the input buffer state to scan directly from a user-specified character buffer. * @param base the character buffer * @param size the size in bytes of the character buffer * * @return the newly allocated buffer state object. */ YY_BUFFER_STATE dax__scan_buffer (char * base, yy_size_t size ) { YY_BUFFER_STATE b; if ( size < 2 || base[size-2] != YY_END_OF_BUFFER_CHAR || base[size-1] != YY_END_OF_BUFFER_CHAR ) /* They forgot to leave room for the EOB's. */ return NULL; b = (YY_BUFFER_STATE) dax_alloc(sizeof( struct yy_buffer_state ) ); if ( ! b ) YY_FATAL_ERROR( "out of dynamic memory in dax__scan_buffer()" ); b->yy_buf_size = size - 2; /* "- 2" to take care of EOB's */ b->yy_buf_pos = b->yy_ch_buf = base; b->yy_is_our_buffer = 0; b->yy_input_file = NULL; b->yy_n_chars = b->yy_buf_size; b->yy_is_interactive = 0; b->yy_at_bol = 1; b->yy_fill_buffer = 0; b->yy_buffer_status = YY_BUFFER_NEW; dax__switch_to_buffer(b ); return b; } /** Setup the input buffer state to scan a string. The next call to dax_lex() will * scan from a @e copy of @a str. * @param yystr a NUL-terminated string to scan * * @return the newly allocated buffer state object. * @note If you want to scan bytes that may contain NUL values, then use * dax__scan_bytes() instead. */ YY_BUFFER_STATE dax__scan_string (yyconst char * yystr ) { return dax__scan_bytes(yystr,(int) strlen(yystr) ); } /** Setup the input buffer state to scan the given bytes. The next call to dax_lex() will * scan from a @e copy of @a bytes. * @param yybytes the byte buffer to scan * @param _yybytes_len the number of bytes in the buffer pointed to by @a bytes. * * @return the newly allocated buffer state object. */ YY_BUFFER_STATE dax__scan_bytes (yyconst char * yybytes, int _yybytes_len ) { YY_BUFFER_STATE b; char *buf; yy_size_t n; int i; /* Get memory for full buffer, including space for trailing EOB's. */ n = (yy_size_t) (_yybytes_len + 2); buf = (char *) dax_alloc(n ); if ( ! buf ) YY_FATAL_ERROR( "out of dynamic memory in dax__scan_bytes()" ); for ( i = 0; i < _yybytes_len; ++i ) buf[i] = yybytes[i]; buf[_yybytes_len] = buf[_yybytes_len+1] = YY_END_OF_BUFFER_CHAR; b = dax__scan_buffer(buf,n ); if ( ! b ) YY_FATAL_ERROR( "bad buffer in dax__scan_bytes()" ); /* It's okay to grow etc. this buffer, and we should throw it * away when we're done. */ b->yy_is_our_buffer = 1; return b; } static void yy_push_state (int _new_state ) { if ( (yy_start_stack_ptr) >= (yy_start_stack_depth) ) { yy_size_t new_size; (yy_start_stack_depth) += YY_START_STACK_INCR; new_size = (yy_size_t) (yy_start_stack_depth) * sizeof( int ); if ( ! (yy_start_stack) ) (yy_start_stack) = (int *) dax_alloc(new_size ); else (yy_start_stack) = (int *) dax_realloc((void *) (yy_start_stack),new_size ); if ( ! (yy_start_stack) ) YY_FATAL_ERROR( "out of memory expanding start-condition stack" ); } (yy_start_stack)[(yy_start_stack_ptr)++] = YY_START; BEGIN(_new_state); } static void yy_pop_state (void) { if ( --(yy_start_stack_ptr) < 0 ) YY_FATAL_ERROR( "start-condition stack underflow" ); BEGIN((yy_start_stack)[(yy_start_stack_ptr)]); } #ifndef YY_EXIT_FAILURE #define YY_EXIT_FAILURE 2 #endif static void yynoreturn yy_fatal_error (yyconst char* msg ) { (void) fprintf( stderr, "%s\n", msg ); exit( YY_EXIT_FAILURE ); } /* Redefine yyless() so it works in section 3 code. */ #undef yyless #define yyless(n) \ do \ { \ /* Undo effects of setting up dax_text. */ \ int yyless_macro_arg = (n); \ YY_LESS_LINENO(yyless_macro_arg);\ dax_text[dax_leng] = (yy_hold_char); \ (yy_c_buf_p) = dax_text + yyless_macro_arg; \ (yy_hold_char) = *(yy_c_buf_p); \ *(yy_c_buf_p) = '\0'; \ dax_leng = yyless_macro_arg; \ } \ while ( 0 ) /* Accessor methods (get/set functions) to struct members. */ /** Get the current line number. * */ int dax_get_lineno (void) { return dax_lineno; } /** Get the input stream. * */ FILE *dax_get_in (void) { return dax_in; } /** Get the output stream. * */ FILE *dax_get_out (void) { return dax_out; } /** Get the length of the current token. * */ int dax_get_leng (void) { return dax_leng; } /** Get the current token. * */ char *dax_get_text (void) { return dax_text; } /** Set the current line number. * @param _line_number line number * */ void dax_set_lineno (int _line_number ) { dax_lineno = _line_number; } /** Set the input stream. This does not discard the current * input buffer. * @param _in_str A readable stream. * * @see dax__switch_to_buffer */ void dax_set_in (FILE * _in_str ) { dax_in = _in_str ; } void dax_set_out (FILE * _out_str ) { dax_out = _out_str ; } int dax_get_debug (void) { return dax__flex_debug; } void dax_set_debug (int _bdebug ) { dax__flex_debug = _bdebug ; } static int yy_init_globals (void) { /* Initialization is the same as for the non-reentrant scanner. * This function is called from dax_lex_destroy(), so don't allocate here. */ /* We do not touch dax_lineno unless the option is enabled. */ dax_lineno = 1; (yy_buffer_stack) = NULL; (yy_buffer_stack_top) = 0; (yy_buffer_stack_max) = 0; (yy_c_buf_p) = NULL; (yy_init) = 0; (yy_start) = 0; (yy_start_stack_ptr) = 0; (yy_start_stack_depth) = 0; (yy_start_stack) = NULL; /* Defined in main.c */ #ifdef YY_STDINIT dax_in = stdin; dax_out = stdout; #else dax_in = NULL; dax_out = NULL; #endif /* For future reference: Set errno on error, since we are called by * dax_lex_init() */ return 0; } /* dax_lex_destroy is for both reentrant and non-reentrant scanners. */ int dax_lex_destroy (void) { /* Pop the buffer stack, destroying each element. */ while(YY_CURRENT_BUFFER){ dax__delete_buffer(YY_CURRENT_BUFFER ); YY_CURRENT_BUFFER_LVALUE = NULL; dax_pop_buffer_state(); } /* Destroy the stack itself. */ dax_free((yy_buffer_stack) ); (yy_buffer_stack) = NULL; /* Destroy the start condition stack. */ dax_free((yy_start_stack) ); (yy_start_stack) = NULL; /* Reset the globals. This is important in a non-reentrant scanner so the next time * dax_lex() is called, initialization will occur. */ yy_init_globals( ); return 0; } /* * Internal utility routines. */ #ifndef yytext_ptr static void yy_flex_strncpy (char* s1, yyconst char * s2, int n ) { int i; for ( i = 0; i < n; ++i ) s1[i] = s2[i]; } #endif #ifdef YY_NEED_STRLEN static int yy_flex_strlen (yyconst char * s ) { int n; for ( n = 0; s[n]; ++n ) ; return n; } #endif void *dax_alloc (yy_size_t size ) { return malloc(size); } void *dax_realloc (void * ptr, yy_size_t size ) { /* The cast to (char *) in the following accommodates both * implementations that use char* generic pointers, and those * that use void* generic pointers. It works with the latter * because both ANSI C and C++ allow castless assignment from * any pointer type to void*, and deal with argument conversions * as though doing an assignment. */ return realloc(ptr, size); } void dax_free (void * ptr ) { free( (char *) ptr ); /* see dax_realloc() for (char *) cast */ } #define YYTABLES_NAME "yytables" /* Element context stack lookup. */ int dax__element_context(int i) { return (0(); runnable_tasks = new std::set(); completed_tasks = new std::set(); return_set = new std::set(); } Global::~Global(){ delete initial_tasks; delete runnable_tasks; delete completed_tasks; delete return_set; } std::set* simulate(double how_long){ XBT_VERB("Run simulation for %f seconds", how_long); sd_global->watch_point_reached = false; sd_global->return_set->clear(); /* explore the runnable tasks */ while (not sd_global->runnable_tasks->empty()) SD_task_run(*(sd_global->runnable_tasks->begin())); double elapsed_time = 0.0; double total_time = 0.0; /* main loop */ while (elapsed_time >= 0 && (how_long < 0 || 0.00001 < (how_long - total_time)) && not sd_global->watch_point_reached) { XBT_DEBUG("Total time: %f", total_time); elapsed_time = surf_solve(how_long > 0 ? surf_get_clock() + how_long - total_time: -1.0); XBT_DEBUG("surf_solve() returns %f", elapsed_time); if (elapsed_time > 0.0) total_time += elapsed_time; /* let's see which tasks are done */ for (auto const& model : *all_existing_models) { surf_action_t action = surf_model_extract_done_action_set(model); while (action != nullptr) { SD_task_t task = static_cast(action->getData()); XBT_VERB("Task '%s' done", SD_task_get_name(task)); SD_task_set_state(task, SD_DONE); /* the state has changed. Add it only if it's the first change */ if (sd_global->return_set->find(task) == sd_global->return_set->end()) sd_global->return_set->insert(task); /* remove the dependencies after this task */ for (auto const& succ : *task->successors) { succ->predecessors->erase(task); succ->inputs->erase(task); XBT_DEBUG("Release dependency on %s: %zu remain(s). Becomes schedulable if %zu=0", SD_task_get_name(succ), succ->predecessors->size()+succ->inputs->size(), succ->predecessors->size()); if (SD_task_get_state(succ) == SD_NOT_SCHEDULED && succ->predecessors->empty()) SD_task_set_state(succ, SD_SCHEDULABLE); if (SD_task_get_state(succ) == SD_SCHEDULED && succ->predecessors->empty() && succ->inputs->empty()) SD_task_set_state(succ, SD_RUNNABLE); if (SD_task_get_state(succ) == SD_RUNNABLE && not sd_global->watch_point_reached) SD_task_run(succ); } task->successors->clear(); for (auto const& output : *task->outputs) { output->start_time = task->finish_time; output->predecessors->erase(task); if (SD_task_get_state(output) == SD_SCHEDULED) SD_task_set_state(output, SD_RUNNABLE); else SD_task_set_state(output, SD_SCHEDULABLE); SD_task_t comm_dst = *(output->successors->begin()); if (SD_task_get_state(comm_dst) == SD_NOT_SCHEDULED && comm_dst->predecessors->empty()){ XBT_DEBUG("%s is a transfer, %s may be ready now if %zu=0", SD_task_get_name(output), SD_task_get_name(comm_dst), comm_dst->predecessors->size()); SD_task_set_state(comm_dst, SD_SCHEDULABLE); } if (SD_task_get_state(output) == SD_RUNNABLE && not sd_global->watch_point_reached) SD_task_run(output); } task->outputs->clear(); action = surf_model_extract_done_action_set(model); } /* let's see which tasks have just failed */ action = surf_model_extract_failed_action_set(model); while (action != nullptr) { SD_task_t task = static_cast(action->getData()); XBT_VERB("Task '%s' failed", SD_task_get_name(task)); SD_task_set_state(task, SD_FAILED); sd_global->return_set->insert(task); action = surf_model_extract_failed_action_set(model); } } } if (not sd_global->watch_point_reached && how_long < 0 && not sd_global->initial_tasks->empty()) { XBT_WARN("Simulation is finished but %zu tasks are still not done", sd_global->initial_tasks->size()); for (auto const& t : *sd_global->initial_tasks) XBT_WARN("%s is in %s state", SD_task_get_name(t), __get_state_name(SD_task_get_state(t))); } XBT_DEBUG("elapsed_time = %f, total_time = %f, watch_point_reached = %d", elapsed_time, total_time, sd_global->watch_point_reached); XBT_DEBUG("current time = %f", surf_get_clock()); return sd_global->return_set; } } } /** * \brief helper for pretty printing of task state * \param state the state of a task * \return the equivalent as a readable string */ const char *__get_state_name(e_SD_task_state_t state){ static std::string state_names[7] = { "not scheduled", "schedulable", "scheduled", "runnable","running", "done", "failed" }; return state_names[static_cast(log2(static_cast(state)))].data(); } /** * \brief Initializes SD internal data * * This function must be called before any other SD function. Then you should call SD_create_environment(). * * \param argc argument number * \param argv argument list * \see SD_create_environment(), SD_exit() */ void SD_init_nocheck(int *argc, char **argv) { xbt_assert(sd_global == nullptr, "SD_init() already called"); sd_global = new simgrid::sd::Global(); surf_init(argc, argv); xbt_cfg_setdefault_string("host/model", "ptask_L07"); if(xbt_cfg_get_boolean("clean-atexit")) atexit(SD_exit); if (_sg_cfg_exit_asap) { exit(0); } } /** \brief set a configuration variable * * Do --help on any simgrid binary to see the list of currently existing configuration variables, and * see Section @ref options. * * Example: SD_config("host/model","default") */ void SD_config(const char *key, const char *value){ xbt_assert(sd_global,"ERROR: Please call SD_init() before using SD_config()"); xbt_cfg_set_as_string(key, value); } /** * \brief Creates the environment * * The environment (i.e. the \ref SD_host_api "hosts" and the \ref SD_link_api "links") is created with * the data stored in the given XML platform file. * * \param platform_file name of an XML file describing the environment to create * \see SD_host_api, SD_link_api * * The XML file follows this DTD: * * \include simgrid.dtd * * Here is a small example of such a platform: * * \include small_platform.xml */ void SD_create_environment(const char *platform_file) { simgrid::s4u::Engine::getInstance()->loadPlatform(platform_file); XBT_DEBUG("Host number: %zu, link number: %d", sg_host_count(), sg_link_count()); #if SIMGRID_HAVE_JEDULE jedule_sd_init(); #endif XBT_VERB("Starting simulation..."); surf_presolve(); /* Takes traces into account */ } /** * \brief Launches the simulation. * * The function will execute the \ref SD_RUNNABLE runnable tasks. * If \a how_long is positive, then the simulation will be stopped either when time reaches \a how_long or when a watch * point is reached. * A non-positive value for \a how_long means no time limit, in which case the simulation will be stopped either when a * watch point is reached or when no more task can be executed. * Then you can call SD_simulate() again. * * \param how_long maximum duration of the simulation (a negative value means no time limit) * \return a dynar of \ref SD_task_t whose state has changed. * \see SD_task_schedule(), SD_task_watch() */ void SD_simulate(double how_long) { simgrid::sd::simulate(how_long); } void SD_simulate_with_update(double how_long, xbt_dynar_t changed_tasks_dynar) { std::set *changed_tasks = simgrid::sd::simulate(how_long); for (auto const& task : *changed_tasks) xbt_dynar_push(changed_tasks_dynar, &task); } /** @brief Returns the current clock, in seconds */ double SD_get_clock() { return surf_get_clock(); } /** * \brief Destroys all SD internal data * This function should be called when the simulation is over. Don't forget to destroy too. * \see SD_init(), SD_task_destroy() */ void SD_exit() { TRACE_end(); #if SIMGRID_HAVE_JEDULE jedule_sd_exit(); #endif delete sd_global; } SimGrid-3.18/src/simdag/dax.dtd0000644000175000017500000000233213217757321016623 0ustar mquinsonmquinson SimGrid-3.18/src/simdag/simdag_private.hpp0000644000175000017500000000410413217757321021060 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "surf/surf.hpp" #include #include #include #ifndef SIMDAG_PRIVATE_HPP #define SIMDAG_PRIVATE_HPP #if SIMGRID_HAVE_JEDULE #include "simgrid/jedule/jedule_sd_binding.h" #endif namespace simgrid{ namespace sd{ class Global { public: explicit Global(); Global(const Global&) = delete; ~Global(); bool watch_point_reached; /* has a task just reached a watch point? */ std::set *initial_tasks; std::set *runnable_tasks; std::set *completed_tasks; std::set *return_set; }; std::set* simulate (double how_long); } } extern "C" { extern XBT_PRIVATE simgrid::sd::Global *sd_global; /* Task */ struct s_SD_task_t { e_SD_task_state_t state; void *data; /* user data */ char *name; e_SD_task_kind_t kind; double amount; double alpha; /* used by typed parallel tasks */ double start_time; double finish_time; surf_action_t surf_action; unsigned short watch_points; /* bit field xor()ed with masks */ int marked; /* used to check if the task DAG has some cycle*/ /* dependencies */ std::set *inputs; std::set *outputs; std::set *predecessors; std::set *successors; /* scheduling parameters (only exist in state SD_SCHEDULED) */ std::vector *allocation; double *flops_amount; double *bytes_amount; double rate; }; /* SimDag private functions */ XBT_PRIVATE void SD_task_set_state(SD_task_t task, e_SD_task_state_t new_state); XBT_PRIVATE void SD_task_run(SD_task_t task); XBT_PRIVATE bool acyclic_graph_detail(xbt_dynar_t dag); XBT_PRIVATE void uniq_transfer_task_name(SD_task_t task); XBT_PRIVATE const char *__get_state_name(e_SD_task_state_t state); } #endif SimGrid-3.18/src/simdag/sd_daxloader.cpp0000644000175000017500000002675513217757316020532 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simdag_private.hpp" #include "simgrid/simdag.h" #include "xbt/file.hpp" #include "xbt/log.h" #include "xbt/misc.h" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_daxparse, sd, "Parsing DAX files"); extern "C" { #include "dax_dtd.h" #include "dax_dtd.c" } /* Ensure that transfer tasks have unique names even though a file is used several times */ void uniq_transfer_task_name(SD_task_t task) { SD_task_t child = *(task->successors->begin()); SD_task_t parent = *(task->predecessors->begin()); std::string new_name = std::string(SD_task_get_name(parent)) + "_" + SD_task_get_name(task) + "_" + SD_task_get_name(child); SD_task_set_name(task, new_name.c_str()); } static bool children_are_marked(SD_task_t task){ for (SD_task_t const& it : *task->successors) if (it->marked == 0) return false; for (SD_task_t const& it : *task->outputs) if (it->marked == 0) return false; return true; } static bool parents_are_marked(SD_task_t task){ for (SD_task_t const& it : *task->predecessors) if (it->marked == 0) return false; for (SD_task_t const& it : *task->inputs) if (it->marked == 0) return false; return true; } bool acyclic_graph_detail(xbt_dynar_t dag){ unsigned int count; bool all_marked = true; SD_task_t task = nullptr; std::vector current; xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E){ task->marked = 0; if(task->successors->empty() && task->outputs->empty()) current.push_back(task); } } while (not current.empty()) { std::vector next; for (auto const& t : current) { //Mark task t->marked = 1; for (SD_task_t const& input : *t->inputs) { input->marked=1; // Inputs are communication, hence they can have only one predecessor SD_task_t input_pred = *(input->predecessors->begin()); if (children_are_marked(input_pred)) next.push_back(input_pred); } for (SD_task_t const& pred : *t->predecessors) { if (children_are_marked(pred)) next.push_back(pred); } } current.clear(); current = next; } all_marked = true; //test if all tasks are marked xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->marked == 0){ XBT_WARN("the task %s is not marked",task->name); all_marked = false; break; } } if (not all_marked) { XBT_VERB("there is at least one cycle in your task graph"); xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->predecessors->empty() && task->inputs->empty()){ task->marked = 1; current.push_back(task); } } //test if something has to be done for the next iteration while (not current.empty()) { std::vector next; //test if the current iteration is done for (auto const& t : current) { t->marked = 1; for (SD_task_t const& output : *t->outputs) { output->marked = 1; // outputs are communication, hence they can have only one successor SD_task_t output_succ = *(output->successors->begin()); if (parents_are_marked(output_succ)) next.push_back(output_succ); } for (SD_task_t const& succ : *t->successors) { if (parents_are_marked(succ)) next.push_back(succ); } } current.clear(); current = next; } all_marked = true; xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->marked == 0){ XBT_WARN("the task %s is in a cycle",task->name); all_marked = false; } } } return all_marked; } static YY_BUFFER_STATE input_buffer; static xbt_dynar_t result; static std::map jobs; static std::map files; static SD_task_t current_job; static SD_task_t root_task; static SD_task_t end_task; static void dax_task_free(void *task) { SD_task_destroy(static_cast(task)); } /** @brief loads a DAX file describing a DAG * * See https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator for more details. */ xbt_dynar_t SD_daxload(const char *filename) { SD_task_t file; FILE* in_file = fopen(filename, "r"); xbt_assert(in_file, "Unable to open \"%s\"\n", filename); input_buffer = dax__create_buffer(in_file, 10); dax__switch_to_buffer(input_buffer); dax_lineno = 1; result = xbt_dynar_new(sizeof(SD_task_t), dax_task_free); root_task = SD_task_create_comp_seq("root", nullptr, 0); /* by design the root task is always SCHEDULABLE */ SD_task_set_state(root_task, SD_SCHEDULABLE); xbt_dynar_push(result, &root_task); end_task = SD_task_create_comp_seq("end", nullptr, 0); int res = dax_lex(); if (res != 0) xbt_die("Parse error in %s: %s", filename, dax__parse_err_msg()); dax__delete_buffer(input_buffer); fclose(in_file); dax_lex_destroy(); /* And now, post-process the files. * We want a file task per pair of computation tasks exchanging the file. Duplicate on need * Files not produced in the system are said to be produced by root task (top of DAG). * Files not consumed in the system are said to be consumed by end task (bottom of DAG). */ for (auto const& elm : files) { file = elm.second; SD_task_t newfile; if (file->predecessors->empty()) { for (SD_task_t const& it : *file->successors) { newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, root_task, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, it); xbt_dynar_push(result, &newfile); } } else if (file->successors->empty()) { for (SD_task_t const& it : *file->predecessors) { newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, it, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, end_task); xbt_dynar_push(result, &newfile); } } else { for (SD_task_t const& it : *file->predecessors) { for (SD_task_t const& it2 : *file->successors) { if (it == it2) { XBT_WARN ("File %s is produced and consumed by task %s." "This loop dependency will prevent the execution of the task.", file->name, it->name); } newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, it, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, it2); xbt_dynar_push(result, &newfile); } } } } /* Push end task last */ xbt_dynar_push(result, &end_task); /* Free previous copy of the files */ for (auto const& elm : files) SD_task_destroy(elm.second); unsigned int cpt; xbt_dynar_foreach(result, cpt, file) { if (SD_task_get_kind(file) == SD_TASK_COMM_E2E) { uniq_transfer_task_name(file); } else if (SD_task_get_kind(file) == SD_TASK_COMP_SEQ){ /* If some tasks do not take files as input, connect them to the root * if they don't produce files, connect them to the end node. */ if ((file != root_task) && (file != end_task)) { if (file->inputs->empty()) SD_task_dependency_add(nullptr, nullptr, root_task, file); if (file->outputs->empty()) SD_task_dependency_add(nullptr, nullptr, file, end_task); } } else { THROW_IMPOSSIBLE; } } if (not acyclic_graph_detail(result)) { std::string base = simgrid::xbt::Path(filename).getBasename(); XBT_ERROR("The DAX described in %s is not a DAG. It contains a cycle.", base.c_str()); xbt_dynar_foreach(result, cpt, file) SD_task_destroy(file); xbt_dynar_free_container(&result); return nullptr; } else { return result; } } void STag_dax__adag() { try { double version = std::stod(std::string(A_dax__adag_version)); xbt_assert(version == 2.1, "Expected version 2.1 in tag, got %f. Fix the parser or your file", version); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Parse error: ") + A_dax__adag_version + " is not a double"); } } void STag_dax__job() { try { double runtime = std::stod(std::string(A_dax__job_runtime)); std::string name = std::string(A_dax__job_id) + "@" + A_dax__job_name; runtime *= 4200000000.; /* Assume that timings were done on a 4.2GFlops machine. I mean, why not? */ XBT_DEBUG("See ", A_dax__job_id, A_dax__job_runtime, runtime); current_job = SD_task_create_comp_seq(name.c_str(), nullptr, runtime); jobs.insert({A_dax__job_id, current_job}); xbt_dynar_push(result, ¤t_job); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Parse error: ") + A_dax__job_runtime + " is not a double"); } } void STag_dax__uses() { double size; try { size = std::stod(std::string(A_dax__uses_size)); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Parse error: ") + A_dax__uses_size + " is not a double"); } bool is_input = (A_dax__uses_link == A_dax__uses_link_input); XBT_DEBUG("See ",A_dax__uses_file,(is_input?"in":"out")); auto it = files.find(A_dax__uses_file); SD_task_t file; if (it == files.end()) { file = SD_task_create_comm_e2e(A_dax__uses_file, nullptr, size); sd_global->initial_tasks->erase(file); files[A_dax__uses_file] = file; } else { file = it->second; if (file->amount < size || file->amount > size) { XBT_WARN("Ignore file %s size redefinition from %.0f to %.0f", A_dax__uses_file, SD_task_get_amount(file), size); } } if (is_input) { SD_task_dependency_add(nullptr, nullptr, file, current_job); } else { SD_task_dependency_add(nullptr, nullptr, current_job, file); if ((file->predecessors->size() + file->inputs->size()) > 1) { XBT_WARN("File %s created at more than one location...", file->name); } } } static SD_task_t current_child; void STag_dax__child() { auto job = jobs.find(A_dax__child_ref); if (job != jobs.end()) { current_child = job->second; } else { throw std::out_of_range(std::string("Parse error on line ") + std::to_string(dax_lineno) + ": Asked to add dependencies to the non-existent " + A_dax__child_ref + "task"); } } void ETag_dax__child() { current_child = nullptr; } void STag_dax__parent() { auto job = jobs.find(A_dax__parent_ref); if (job != jobs.end()) { SD_task_t parent = job->second; SD_task_dependency_add(nullptr, nullptr, parent, current_child); XBT_DEBUG("Control-flow dependency from %s to %s", current_child->name, parent->name); } else { throw std::out_of_range(std::string("Parse error on line ") + std::to_string(dax_lineno) + ": Asked to add a dependency from " + current_child->name + " to " + A_dax__parent_ref + ", but " + A_dax__parent_ref + " does not exist"); } } void ETag_dax__adag() { XBT_DEBUG("See "); } void ETag_dax__job() { current_job = nullptr; XBT_DEBUG("See "); } void ETag_dax__parent() { XBT_DEBUG("See "); } void ETag_dax__uses() { XBT_DEBUG("See "); } SimGrid-3.18/src/simdag/sd_task.cpp0000644000175000017500000010316013217757316017513 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simdag_private.hpp" #include "src/surf/HostImpl.hpp" #include "src/surf/surf_interface.hpp" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_task, sd, "Logging specific to SimDag (task)"); /* Destroys the data memorized by SD_task_schedule. Task state must be SD_SCHEDULED or SD_RUNNABLE. */ static void __SD_task_destroy_scheduling_data(SD_task_t task) { if (task->state != SD_SCHEDULED && task->state != SD_RUNNABLE) THROWF(arg_error, 0, "Task '%s' must be SD_SCHEDULED or SD_RUNNABLE", SD_task_get_name(task)); xbt_free(task->flops_amount); xbt_free(task->bytes_amount); task->bytes_amount = nullptr; task->flops_amount = nullptr; } /** * \brief Creates a new task. * * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param amount amount of the task * \return the new task * \see SD_task_destroy() */ SD_task_t SD_task_create(const char *name, void *data, double amount) { SD_task_t task = xbt_new0(s_SD_task_t, 1); task->kind = SD_TASK_NOT_TYPED; task->state= SD_NOT_SCHEDULED; sd_global->initial_tasks->insert(task); task->marked = 0; task->start_time = -1.0; task->finish_time = -1.0; task->surf_action = nullptr; task->watch_points = 0; task->inputs = new std::set(); task->outputs = new std::set(); task->predecessors = new std::set(); task->successors = new std::set(); task->data = data; task->name = xbt_strdup(name); task->amount = amount; task->allocation = new std::vector(); task->rate = -1; return task; } static inline SD_task_t SD_task_create_sized(const char *name, void *data, double amount, int count) { SD_task_t task = SD_task_create(name, data, amount); task->bytes_amount = xbt_new0(double, count * count); task->flops_amount = xbt_new0(double, count); return task; } /** @brief create a end-to-end communication task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A end-to-end communication must be scheduled on 2 hosts, and the amount specified at creation is sent from hosts[0] * to hosts[1]. */ SD_task_t SD_task_create_comm_e2e(const char *name, void *data, double amount) { SD_task_t res = SD_task_create_sized(name, data, amount, 2); res->bytes_amount[2] = amount; res->kind = SD_TASK_COMM_E2E; return res; } /** @brief create a sequential computation task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A sequential computation must be scheduled on 1 host, and the amount specified at creation to be run on hosts[0]. * * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param flops_amount amount of compute work to be done by the task * \return the new SD_TASK_COMP_SEQ typed task */ SD_task_t SD_task_create_comp_seq(const char *name, void *data, double flops_amount) { SD_task_t res = SD_task_create_sized(name, data, flops_amount, 1); res->flops_amount[0] = flops_amount; res->kind = SD_TASK_COMP_SEQ; return res; } /** @brief create a parallel computation task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A parallel computation can be scheduled on any number of host. * The underlying speedup model is Amdahl's law. * To be auto-scheduled, \see SD_task_distribute_comp_amdahl has to be called first. * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param flops_amount amount of compute work to be done by the task * \param alpha purely serial fraction of the work to be done (in [0.;1.[) * \return the new task */ SD_task_t SD_task_create_comp_par_amdahl(const char *name, void *data, double flops_amount, double alpha) { xbt_assert(alpha < 1. && alpha >= 0., "Invalid parameter: alpha must be in [0.;1.["); SD_task_t res = SD_task_create(name, data, flops_amount); res->alpha = alpha; res->kind = SD_TASK_COMP_PAR_AMDAHL; return res; } /** @brief create a complex data redistribution task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). * This allows to specify the task costs at creation, and decouple them from the scheduling process where you just * specify which resource should communicate. * * A data redistribution can be scheduled on any number of host. * The assumed distribution is a 1D block distribution. Each host owns the same share of the \see amount. * To be auto-scheduled, \see SD_task_distribute_comm_mxn_1d_block has to be called first. * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param amount amount of data to redistribute by the task * \return the new task */ SD_task_t SD_task_create_comm_par_mxn_1d_block(const char *name, void *data, double amount) { SD_task_t res = SD_task_create(name, data, amount); res->kind = SD_TASK_COMM_PAR_MXN_1D_BLOCK; return res; } /** * \brief Destroys a task. * * The user data (if any) should have been destroyed first. * * \param task the task you want to destroy * \see SD_task_create() */ void SD_task_destroy(SD_task_t task) { XBT_DEBUG("Destroying task %s...", SD_task_get_name(task)); /* First Remove all dependencies associated with the task. */ while (not task->predecessors->empty()) SD_task_dependency_remove(*(task->predecessors->begin()), task); while (not task->inputs->empty()) SD_task_dependency_remove(*(task->inputs->begin()), task); while (not task->successors->empty()) SD_task_dependency_remove(task, *(task->successors->begin())); while (not task->outputs->empty()) SD_task_dependency_remove(task, *(task->outputs->begin())); if (task->state == SD_SCHEDULED || task->state == SD_RUNNABLE) __SD_task_destroy_scheduling_data(task); xbt_free(task->name); if (task->surf_action != nullptr) task->surf_action->unref(); delete task->allocation; xbt_free(task->bytes_amount); xbt_free(task->flops_amount); delete task->inputs; delete task->outputs; delete task->predecessors; delete task->successors; xbt_free(task); XBT_DEBUG("Task destroyed."); } /** * \brief Returns the user data of a task * * \param task a task * \return the user data associated with this task (can be \c nullptr) * \see SD_task_set_data() */ void *SD_task_get_data(SD_task_t task) { return task->data; } /** * \brief Sets the user data of a task * * The new data can be \c nullptr. The old data should have been freed first, if it was not \c nullptr. * * \param task a task * \param data the new data you want to associate with this task * \see SD_task_get_data() */ void SD_task_set_data(SD_task_t task, void *data) { task->data = data; } /** * \brief Sets the rate of a task * * This will change the network bandwidth a task can use. This rate cannot be dynamically changed. Once the task has * started, this call is ineffective. This rate depends on both the nominal bandwidth on the route onto which the task * is scheduled (\see SD_task_get_current_bandwidth) and the amount of data to transfer. * * To divide the nominal bandwidth by 2, the rate then has to be : * rate = bandwidth/(2*amount) * * \param task a \see SD_TASK_COMM_E2E task (end-to-end communication) * \param rate the new rate you want to associate with this task. */ void SD_task_set_rate(SD_task_t task, double rate) { xbt_assert(task->kind == SD_TASK_COMM_E2E, "The rate can be modified for end-to-end communications only."); if(task->state < SD_RUNNING) { task->rate = rate; } else { XBT_WARN("Task %p has started. Changing rate is ineffective.", task); } } /** * \brief Returns the state of a task * * \param task a task * \return the current \ref e_SD_task_state_t "state" of this task: * #SD_NOT_SCHEDULED, #SD_SCHEDULED, #SD_RUNNABLE, #SD_RUNNING, #SD_DONE or #SD_FAILED * \see e_SD_task_state_t */ e_SD_task_state_t SD_task_get_state(SD_task_t task) { return task->state; } /* Changes the state of a task. Updates the sd_global->watch_point_reached flag. */ void SD_task_set_state(SD_task_t task, e_SD_task_state_t new_state) { std::set::iterator idx; XBT_DEBUG("Set state of '%s' to %d", task->name, new_state); if ((new_state == SD_NOT_SCHEDULED || new_state == SD_SCHEDULABLE) && task->state == SD_FAILED){ sd_global->completed_tasks->erase(task); sd_global->initial_tasks->insert(task); } if (new_state == SD_SCHEDULED && task->state == SD_RUNNABLE){ sd_global->initial_tasks->insert(task); sd_global->runnable_tasks->erase(task); } if (new_state == SD_RUNNABLE){ idx = sd_global->initial_tasks->find(task); if (idx != sd_global->initial_tasks->end()) { sd_global->runnable_tasks->insert(*idx); sd_global->initial_tasks->erase(idx); } } if (new_state == SD_RUNNING) sd_global->runnable_tasks->erase(task); if (new_state == SD_DONE || new_state == SD_FAILED){ sd_global->completed_tasks->insert(task); task->start_time = task->surf_action->getStartTime(); if (new_state == SD_DONE){ task->finish_time = task->surf_action->getFinishTime(); #if SIMGRID_HAVE_JEDULE jedule_log_sd_event(task); #endif } else task->finish_time = surf_get_clock(); task->surf_action->unref(); task->surf_action = nullptr; task->allocation->clear(); } task->state = new_state; if (task->watch_points & new_state) { XBT_VERB("Watch point reached with task '%s'!", task->name); sd_global->watch_point_reached = true; SD_task_unwatch(task, new_state); /* remove the watch point */ } } /** * \brief Returns the name of a task * * \param task a task * \return the name of this task (can be \c nullptr) */ const char *SD_task_get_name(SD_task_t task) { return task->name; } /** @brief Allows to change the name of a task */ void SD_task_set_name(SD_task_t task, const char *name) { xbt_free(task->name); task->name = xbt_strdup(name); } /** @brief Returns the dynar of the parents of a task * * \param task a task * \return a newly allocated dynar comprising the parents of this task */ xbt_dynar_t SD_task_get_parents(SD_task_t task) { xbt_dynar_t parents = xbt_dynar_new(sizeof(SD_task_t), nullptr); for (auto const& it : *task->predecessors) xbt_dynar_push(parents, &it); for (auto const& it : *task->inputs) xbt_dynar_push(parents, &it); return parents; } /** @brief Returns the dynar of the parents of a task * * \param task a task * \return a newly allocated dynar comprising the parents of this task */ xbt_dynar_t SD_task_get_children(SD_task_t task) { xbt_dynar_t children = xbt_dynar_new(sizeof(SD_task_t), nullptr); for (auto const& it : *task->successors) xbt_dynar_push(children, &it); for (auto const& it : *task->outputs) xbt_dynar_push(children, &it); return children; } /** * \brief Returns the number of workstations involved in a task * * Only call this on already scheduled tasks! * \param task a task */ int SD_task_get_workstation_count(SD_task_t task) { return task->allocation->size(); } /** * \brief Returns the list of workstations involved in a task * * Only call this on already scheduled tasks! * \param task a task */ sg_host_t *SD_task_get_workstation_list(SD_task_t task) { return task->allocation->data(); } /** * \brief Returns the total amount of work contained in a task * * \param task a task * \return the total amount of work (computation or data transfer) for this task * \see SD_task_get_remaining_amount() */ double SD_task_get_amount(SD_task_t task) { return task->amount; } /** @brief Sets the total amount of work of a task * For sequential typed tasks (COMP_SEQ and COMM_E2E), it also sets the appropriate values in the flops_amount and * bytes_amount arrays respectively. Nothing more than modifying task->amount is done for parallel typed tasks * (COMP_PAR_AMDAHL and COMM_PAR_MXN_1D_BLOCK) as the distribution of the amount of work is done at scheduling time. * * \param task a task * \param amount the new amount of work to execute */ void SD_task_set_amount(SD_task_t task, double amount) { task->amount = amount; if (task->kind == SD_TASK_COMP_SEQ) task->flops_amount[0] = amount; if (task->kind == SD_TASK_COMM_E2E) task->bytes_amount[2] = amount; } /** * \brief Returns the alpha parameter of a SD_TASK_COMP_PAR_AMDAHL task * * \param task a parallel task assuming Amdahl's law as speedup model * \return the alpha parameter (serial part of a task in percent) for this task */ double SD_task_get_alpha(SD_task_t task) { xbt_assert(SD_task_get_kind(task) == SD_TASK_COMP_PAR_AMDAHL, "Alpha parameter is not defined for this kind of task"); return task->alpha; } /** * \brief Returns the remaining amount work to do till the completion of a task * * \param task a task * \return the remaining amount of work (computation or data transfer) of this task * \see SD_task_get_amount() */ double SD_task_get_remaining_amount(SD_task_t task) { if (task->surf_action) return task->surf_action->getRemains(); else return (task->state == SD_DONE) ? 0 : task->amount; } e_SD_task_kind_t SD_task_get_kind(SD_task_t task) { return task->kind; } /** @brief Displays debugging information about a task */ void SD_task_dump(SD_task_t task) { XBT_INFO("Displaying task %s", SD_task_get_name(task)); if (task->state == SD_RUNNABLE) XBT_INFO(" - state: runnable"); else if (task->state < SD_RUNNABLE) XBT_INFO(" - state: %s not runnable", __get_state_name(task->state)); else XBT_INFO(" - state: not runnable %s", __get_state_name(task->state)); if (task->kind != 0) { switch (task->kind) { case SD_TASK_COMM_E2E: XBT_INFO(" - kind: end-to-end communication"); break; case SD_TASK_COMP_SEQ: XBT_INFO(" - kind: sequential computation"); break; case SD_TASK_COMP_PAR_AMDAHL: XBT_INFO(" - kind: parallel computation following Amdahl's law"); break; case SD_TASK_COMM_PAR_MXN_1D_BLOCK: XBT_INFO(" - kind: MxN data redistribution assuming 1D block distribution"); break; default: XBT_INFO(" - (unknown kind %d)", task->kind); } } XBT_INFO(" - amount: %.0f", SD_task_get_amount(task)); if (task->kind == SD_TASK_COMP_PAR_AMDAHL) XBT_INFO(" - alpha: %.2f", task->alpha); XBT_INFO(" - Dependencies to satisfy: %zu", task->inputs->size()+ task->predecessors->size()); if ((task->inputs->size()+ task->predecessors->size()) > 0) { XBT_INFO(" - pre-dependencies:"); for (auto const& it : *task->predecessors) XBT_INFO(" %s", it->name); for (auto const& it : *task->inputs) XBT_INFO(" %s", it->name); } if ((task->outputs->size() + task->successors->size()) > 0) { XBT_INFO(" - post-dependencies:"); for (auto const& it : *task->successors) XBT_INFO(" %s", it->name); for (auto const& it : *task->outputs) XBT_INFO(" %s", it->name); } } /** @brief Dumps the task in dotty formalism into the FILE* passed as second argument */ void SD_task_dotty(SD_task_t task, void *out) { FILE *fout = static_cast(out); fprintf(fout, " T%p [label=\"%.20s\"", task, task->name); switch (task->kind) { case SD_TASK_COMM_E2E: case SD_TASK_COMM_PAR_MXN_1D_BLOCK: fprintf(fout, ", shape=box"); break; case SD_TASK_COMP_SEQ: case SD_TASK_COMP_PAR_AMDAHL: fprintf(fout, ", shape=circle"); break; default: xbt_die("Unknown task type!"); } fprintf(fout, "];\n"); for (auto const& it : *task->predecessors) fprintf(fout, " T%p -> T%p;\n", it, task); for (auto const& it : *task->inputs) fprintf(fout, " T%p -> T%p;\n", it, task); } /** * \brief Adds a dependency between two tasks * * \a dst will depend on \a src, ie \a dst will not start before \a src is finished. * Their \ref e_SD_task_state_t "state" must be #SD_NOT_SCHEDULED, #SD_SCHEDULED or #SD_RUNNABLE. * * \param name the name of the new dependency (can be \c nullptr) * \param data the user data you want to associate with this dependency (can be \c nullptr) * \param src the task which must be executed first * \param dst the task you want to make depend on \a src * \see SD_task_dependency_remove() */ void SD_task_dependency_add(const char *name, void *data, SD_task_t src, SD_task_t dst) { if (src == dst) THROWF(arg_error, 0, "Cannot add a dependency between task '%s' and itself", SD_task_get_name(src)); if (src->state == SD_DONE || src->state == SD_FAILED) THROWF(arg_error, 0, "Task '%s' must be SD_NOT_SCHEDULED, SD_SCHEDULABLE, SD_SCHEDULED, SD_RUNNABLE, or SD_RUNNING", src->name); if (dst->state == SD_DONE || dst->state == SD_FAILED || dst->state == SD_RUNNING) THROWF(arg_error, 0, "Task '%s' must be SD_NOT_SCHEDULED, SD_SCHEDULABLE, SD_SCHEDULED, or SD_RUNNABLE", dst->name); if (dst->inputs->find(src) != dst->inputs->end() || src->outputs->find(dst) != src->outputs->end() || src->successors->find(dst) != src->successors->end() || dst->predecessors->find(src) != dst->predecessors->end()) THROWF(arg_error, 0, "A dependency already exists between task '%s' and task '%s'", src->name, dst->name); XBT_DEBUG("SD_task_dependency_add: src = %s, dst = %s", src->name, dst->name); if (src->kind == SD_TASK_COMM_E2E || src->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK){ if (dst->kind == SD_TASK_COMP_SEQ || dst->kind == SD_TASK_COMP_PAR_AMDAHL) dst->inputs->insert(src); else dst->predecessors->insert(src); src->successors->insert(dst); } else { if (dst->kind == SD_TASK_COMM_E2E|| dst->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK) src->outputs->insert(dst); else src->successors->insert(dst); dst->predecessors->insert(src); } /* if the task was runnable, the task goes back to SD_SCHEDULED because of the new dependency*/ if (dst->state == SD_RUNNABLE) { XBT_DEBUG("SD_task_dependency_add: %s was runnable and becomes scheduled!", dst->name); SD_task_set_state(dst, SD_SCHEDULED); } } /** * \brief Indicates whether there is a dependency between two tasks. * * \param src a task * \param dst a task depending on \a src * * If src is nullptr, checks whether dst has any pre-dependency. * If dst is nullptr, checks whether src has any post-dependency. */ int SD_task_dependency_exists(SD_task_t src, SD_task_t dst) { xbt_assert(src != nullptr || dst != nullptr, "Invalid parameter: both src and dst are nullptr"); if (src) { if (dst) { return (src->successors->find(dst) != src->successors->end() || src->outputs->find(dst) != src->outputs->end()); } else { return src->successors->size() + src->outputs->size(); } } else { return dst->predecessors->size() + dst->inputs->size(); } return 0; } /** * \brief Remove a dependency between two tasks * * \param src a task * \param dst a task depending on \a src * \see SD_task_dependency_add() */ void SD_task_dependency_remove(SD_task_t src, SD_task_t dst) { XBT_DEBUG("SD_task_dependency_remove: src = %s, dst = %s", SD_task_get_name(src), SD_task_get_name(dst)); if (src->successors->find(dst) == src->successors->end() && src->outputs->find(dst) == src->outputs->end()) THROWF(arg_error, 0, "No dependency found between task '%s' and '%s': task '%s' is not a successor of task '%s'", src->name, dst->name, dst->name, src->name); if (src->kind == SD_TASK_COMM_E2E || src->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK){ if (dst->kind == SD_TASK_COMP_SEQ || dst->kind == SD_TASK_COMP_PAR_AMDAHL) dst->inputs->erase(src); else dst->predecessors->erase(src); src->successors->erase(dst); } else { if (dst->kind == SD_TASK_COMM_E2E|| dst->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK) src->outputs->erase(dst); else src->successors->erase(dst); dst->predecessors->erase(src); } /* if the task was scheduled and dependencies are satisfied, we can make it runnable */ if (dst->predecessors->empty() && dst->inputs->empty() && dst->state == SD_SCHEDULED) SD_task_set_state(dst, SD_RUNNABLE); } /** * \brief Adds a watch point to a task * * SD_simulate() will stop as soon as the \ref e_SD_task_state_t "state" of this task becomes the one given in argument. * The watch point is then automatically removed. * * \param task a task * \param state the \ref e_SD_task_state_t "state" you want to watch (cannot be #SD_NOT_SCHEDULED) * \see SD_task_unwatch() */ void SD_task_watch(SD_task_t task, e_SD_task_state_t state) { if (state & SD_NOT_SCHEDULED) THROWF(arg_error, 0, "Cannot add a watch point for state SD_NOT_SCHEDULED"); task->watch_points = task->watch_points | state; } /** * \brief Removes a watch point from a task * * \param task a task * \param state the \ref e_SD_task_state_t "state" you no longer want to watch * \see SD_task_watch() */ void SD_task_unwatch(SD_task_t task, e_SD_task_state_t state) { xbt_assert(state != SD_NOT_SCHEDULED, "SimDag error: Cannot have a watch point for state SD_NOT_SCHEDULED"); task->watch_points = task->watch_points & ~state; } /** * \brief Returns an approximative estimation of the execution time of a task. * * The estimation is very approximative because the value returned is the time the task would take if it was executed * now and if it was the only task. * * \param task the task to evaluate * \param host_count number of hosts on which the task would be executed * \param host_list the hosts on which the task would be executed * \param flops_amount computation amount for each host(i.e., an array of host_count doubles) * \param bytes_amount communication amount between each pair of hosts (i.e., a matrix of host_count*host_count doubles) * \see SD_schedule() */ double SD_task_get_execution_time(SD_task_t task, int host_count, const sg_host_t *host_list, const double *flops_amount, const double *bytes_amount) { xbt_assert(host_count > 0, "Invalid parameter"); double max_time = 0.0; /* the task execution time is the maximum execution time of the parallel tasks */ for (int i = 0; i < host_count; i++) { double time = 0.0; if (flops_amount != nullptr) time = flops_amount[i] / host_list[i]->getSpeed(); if (bytes_amount != nullptr) for (int j = 0; j < host_count; j++) if (bytes_amount[i * host_count + j] != 0) time += (sg_host_route_latency(host_list[i], host_list[j]) + bytes_amount[i * host_count + j] / sg_host_route_bandwidth(host_list[i], host_list[j])); if (time > max_time) max_time = time; } return max_time; } static inline void SD_task_do_schedule(SD_task_t task) { if (SD_task_get_state(task) > SD_SCHEDULABLE) THROWF(arg_error, 0, "Task '%s' has already been scheduled", SD_task_get_name(task)); if (task->predecessors->empty() && task->inputs->empty()) SD_task_set_state(task, SD_RUNNABLE); else SD_task_set_state(task, SD_SCHEDULED); } /** * \brief Schedules a task * * The task state must be #SD_NOT_SCHEDULED. * Once scheduled, a task is executed as soon as possible in \see SD_simulate, i.e. when its dependencies are satisfied. * * \param task the task you want to schedule * \param host_count number of hosts on which the task will be executed * \param host_list the hosts on which the task will be executed * \param flops_amount computation amount for each hosts (i.e., an array of host_count doubles) * \param bytes_amount communication amount between each pair of hosts (i.e., a matrix of host_count*host_count doubles) * \param rate task execution speed rate * \see SD_task_unschedule() */ void SD_task_schedule(SD_task_t task, int host_count, const sg_host_t * host_list, const double *flops_amount, const double *bytes_amount, double rate) { xbt_assert(host_count > 0, "host_count must be positive"); task->rate = rate; if (flops_amount) { task->flops_amount = static_cast(xbt_realloc(task->flops_amount, sizeof(double) * host_count)); memcpy(task->flops_amount, flops_amount, sizeof(double) * host_count); } else { xbt_free(task->flops_amount); task->flops_amount = nullptr; } int communication_nb = host_count * host_count; if (bytes_amount) { task->bytes_amount = static_cast(xbt_realloc(task->bytes_amount, sizeof(double) * communication_nb)); memcpy(task->bytes_amount, bytes_amount, sizeof(double) * communication_nb); } else { xbt_free(task->bytes_amount); task->bytes_amount = nullptr; } for(int i =0; iallocation->push_back(host_list[i]); SD_task_do_schedule(task); } /** * \brief Unschedules a task * * The task state must be #SD_SCHEDULED, #SD_RUNNABLE, #SD_RUNNING or #SD_FAILED. * If you call this function, the task state becomes #SD_NOT_SCHEDULED. * Call SD_task_schedule() to schedule it again. * * \param task the task you want to unschedule * \see SD_task_schedule() */ void SD_task_unschedule(SD_task_t task) { if (task->state == SD_NOT_SCHEDULED || task->state == SD_SCHEDULABLE) THROWF(arg_error, 0, "Task %s: the state must be SD_SCHEDULED, SD_RUNNABLE, SD_RUNNING or SD_FAILED", task->name); if ((task->state == SD_SCHEDULED || task->state == SD_RUNNABLE) /* if the task is scheduled or runnable */ && ((task->kind == SD_TASK_COMP_PAR_AMDAHL) || (task->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK))) { /* Don't free scheduling data for typed tasks */ __SD_task_destroy_scheduling_data(task); task->allocation->clear(); } if (SD_task_get_state(task) == SD_RUNNING) /* the task should become SD_FAILED */ task->surf_action->cancel(); else { if (task->predecessors->empty() && task->inputs->empty()) SD_task_set_state(task, SD_SCHEDULABLE); else SD_task_set_state(task, SD_NOT_SCHEDULED); } task->start_time = -1.0; } /* Runs a task. */ void SD_task_run(SD_task_t task) { xbt_assert(task->state == SD_RUNNABLE, "Task '%s' is not runnable! Task state: %d", task->name, (int) task->state); xbt_assert(task->allocation != nullptr, "Task '%s': host_list is nullptr!", task->name); XBT_VERB("Executing task '%s'", task->name); /* Copy the elements of the task into the action */ int host_nb = task->allocation->size(); sg_host_t* hosts = new sg_host_t[host_nb]; std::copy_n(task->allocation->begin(), host_nb, hosts); double* flops_amount = new double[host_nb](); double* bytes_amount = new double[host_nb * host_nb](); if(task->flops_amount) std::copy_n(task->flops_amount, host_nb, flops_amount); if(task->bytes_amount) std::copy_n(task->bytes_amount, host_nb * host_nb, bytes_amount); task->surf_action = surf_host_model->executeParallelTask(host_nb, hosts, flops_amount, bytes_amount, task->rate); task->surf_action->setData(task); XBT_DEBUG("surf_action = %p", task->surf_action); __SD_task_destroy_scheduling_data(task); /* now the scheduling data are not useful anymore */ SD_task_set_state(task, SD_RUNNING); sd_global->return_set->insert(task); } /** * \brief Returns the start time of a task * * The task state must be SD_RUNNING, SD_DONE or SD_FAILED. * * \param task: a task * \return the start time of this task */ double SD_task_get_start_time(SD_task_t task) { if (task->surf_action) return task->surf_action->getStartTime(); else return task->start_time; } /** * \brief Returns the finish time of a task * * The task state must be SD_RUNNING, SD_DONE or SD_FAILED. * If the state is not completed yet, the returned value is an estimation of the task finish time. This value can * vary until the task is completed. * * \param task: a task * \return the start time of this task */ double SD_task_get_finish_time(SD_task_t task) { if (task->surf_action) /* should never happen as actions are destroyed right after their completion */ return task->surf_action->getFinishTime(); else return task->finish_time; } void SD_task_distribute_comp_amdahl(SD_task_t task, int count) { xbt_assert(task->kind == SD_TASK_COMP_PAR_AMDAHL, "Task %s is not a SD_TASK_COMP_PAR_AMDAHL typed task." "Cannot use this function.", task->name); task->flops_amount = xbt_new0(double, count); task->bytes_amount = xbt_new0(double, count * count); for (int i=0; iflops_amount[i] = (task->alpha + (1 - task->alpha)/count) * task->amount; } } void SD_task_build_MxN_1D_block_matrix(SD_task_t task, int src_nb, int dst_nb){ xbt_assert(task->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK, "Task %s is not a SD_TASK_COMM_PAR_MXN_1D_BLOCK typed task." "Cannot use this function.", task->name); xbt_free(task->bytes_amount); task->bytes_amount = xbt_new0(double,task->allocation->size() * task->allocation->size()); for (int i=0; iamount/src_nb; double src_end = src_start + task->amount/src_nb; for (int j=0; jamount/dst_nb; double dst_end = dst_start + task->amount/dst_nb; XBT_VERB("(%d->%d): (%.2f, %.2f)-> (%.2f, %.2f)", i, j, src_start, src_end, dst_start, dst_end); task->bytes_amount[i*(src_nb+dst_nb)+src_nb+j]=0.0; if ((src_end > dst_start) && (dst_end > src_start)) { /* There is something to send */ task->bytes_amount[i * (src_nb + dst_nb) + src_nb + j] = std::min(src_end, dst_end) - std::max(src_start, dst_start); XBT_VERB("==> %.2f", task->bytes_amount[i*(src_nb+dst_nb)+src_nb+j]); } } } } /** @brief Auto-schedules a task. * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * To be auto-schedulable, a task must be a typed computation SD_TASK_COMP_SEQ or SD_TASK_COMP_PAR_AMDAHL. */ void SD_task_schedulev(SD_task_t task, int count, const sg_host_t * list) { xbt_assert(task->kind == SD_TASK_COMP_SEQ || task->kind == SD_TASK_COMP_PAR_AMDAHL, "Task %s is not typed. Cannot automatically schedule it.", SD_task_get_name(task)); for(int i =0; iallocation->push_back(list[i]); XBT_VERB("Schedule computation task %s on %zu host(s)", task->name, task->allocation->size()); if (task->kind == SD_TASK_COMP_SEQ) { if (not task->flops_amount) { /*This task has failed and is rescheduled. Reset the flops_amount*/ task->flops_amount = xbt_new0(double, 1); task->flops_amount[0] = task->amount; } XBT_VERB("It costs %.f flops", task->flops_amount[0]); } if (task->kind == SD_TASK_COMP_PAR_AMDAHL) { SD_task_distribute_comp_amdahl(task, count); XBT_VERB("%.f flops will be distributed following Amdahl's Law", task->flops_amount[0]); } SD_task_do_schedule(task); /* Iterate over all inputs and outputs to say where I am located (and start them if runnable) */ for (auto const& input : *task->inputs) { int src_nb = input->allocation->size(); int dst_nb = count; if (input->allocation->empty()) XBT_VERB("Sender side of '%s' not scheduled. Set receiver side to '%s''s allocation", input->name, task->name); for (int i=0; iallocation->push_back(task->allocation->at(i)); if (input->allocation->size () > task->allocation->size()) { if (task->kind == SD_TASK_COMP_PAR_AMDAHL) SD_task_build_MxN_1D_block_matrix(input, src_nb, dst_nb); SD_task_do_schedule(input); XBT_VERB ("Auto-Schedule Communication task '%s'. Send %.f bytes from %d hosts to %d hosts.", input->name,input->amount, src_nb, dst_nb); } } for (auto const& output : *task->outputs) { int src_nb = count; int dst_nb = output->allocation->size(); if (output->allocation->empty()) XBT_VERB("Receiver side of '%s' not scheduled. Set sender side to '%s''s allocation", output->name, task->name); for (int i=0; iallocation->insert(output->allocation->begin()+i, task->allocation->at(i)); if (output->allocation->size () > task->allocation->size()) { if (task->kind == SD_TASK_COMP_PAR_AMDAHL) SD_task_build_MxN_1D_block_matrix(output, src_nb, dst_nb); SD_task_do_schedule(output); XBT_VERB ("Auto-Schedule Communication task %s. Send %.f bytes from %d hosts to %d hosts.", output->name, output->amount, src_nb, dst_nb); } } } /** @brief autoschedule a task on a list of hosts * * This function is similar to SD_task_schedulev(), but takes the list of hosts to schedule onto as separate parameters. * It builds a proper vector of hosts and then call SD_task_schedulev() */ void SD_task_schedulel(SD_task_t task, int count, ...) { va_list ap; sg_host_t* list = new sg_host_t[count]; va_start(ap, count); for (int i=0; i ${TMPFILE} echo "#line 1 \"${ARG}\"" >> ${TMPFILE} fi sed 's/[[:space:]]\{6\}[[:space:]]*\([eE][nN][dD] \)\{0,1\}[pP][rR][oO][gG][rR][aA][mM][[:space:]]*\([a-zA-Z0-9\-\_]*\)/ \1subroutine user_main /g;s/[[:space:]]*[uU][sS][eE][[:space:]]*[mM][pP][iI]/\include \"mpif\.h\" /g' "${ARG}" >> "${TMPFILE}" SRCFILE="${TMPFILE}" list_add CMDLINE "${SRCFILE}" } TRACE_CALL_LOCATION=0 NEEDS_OUTPUT=1 list_set CMDLINE "${F77}" list_add_not_empty CMDLINE "${FFLAGS}" while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDLINE "-c" ;; *.f) TMPFILE=$(mymktemp "${ARG}" ".f") ORIGFILE="${ARG%.f}" filter_and_compile ;; *.F) TMPFILE=$(mymktemp "${ARG}" ".F") ORIGFILE="${ARG%.F}" filter_and_compile ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${F77} --version ;; '-trace-call-location') TRACE_CALL_LOCATION=1 # This should be list_add FFLAGS but it's not possible # anymore: FFLAGS was already moved into CMDLINE above. list_add_not_empty CMDLINE "-ffixed-line-length-none" "-cpp" ;; -o) NEEDS_OUTPUT=0 list_add CMDLINE "-o$1" shift ;; *) list_add CMDLINE "${ARG}" ;; esac done if [ $NEEDS_OUTPUT -ne 0 ]; then list_add CMDLINE "-o${ORIGFILE}.o" fi list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.18/src/smpi/smpif90.in0000644000175000017500000000415613217757325016707 0ustar mquinsonmquinson#! /bin/sh # Copyright (c) 2012-2017. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" F90=@SMPI_Fortran_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set FFLAGS @SMPI_Fortran_FLAGS@ list_set LINKARGS "-shared" "-lsimgrid" @SMPI_Fortran_LIBS@ "-lm" list_set TMPFILES main_name=main cleanup () { eval $(list_get TMPFILES) rm -f "$@" } trap 'cleanup' EXIT filter_and_compile() { list_add TMPFILES "${TMPFILE}" #replace "program main_name by subroutine user\_main (and the end clause as well)" sed 's/[[:space:]]*[pP][rR][oO][gG][rR][aA][mM][[:space:]]*\([a-zA-Z0-9\-\_]*\)/ subroutine user\_main /g;s/[[:space:]]*[uU][sS][eE][[:space:]]*[mM][pP][iI]/\include \"mpif\.h\" /g' "${ARG}" > "${TMPFILE}" SRCFILE="${TMPFILE}" list_add CMDLINE "${SRCFILE}" } list_set CMDLINE "${F90}" list_add_not_empty CMDLINE "${FFLAGS}" while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDLINE "-c" ;; *.f90) TMPFILE=$(mymktemp "${ARG}" ".f90") filter_and_compile ;; *.F90) TMPFILE=$(mymktemp "${ARG}" ".F90") filter_and_compile ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${F90} --version ;; -o) list_add CMDLINE "-o$1" shift ;; *) list_add CMDLINE "${ARG}" ;; esac done list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.18/src/smpi/smpicxx.in0000755000175000017500000000341413217757325017112 0ustar mquinsonmquinson#! /bin/sh # Copyright (c) 2014-2017. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" CXX=@CMAKE_CXX_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set CXXFLAGS list_set LINKARGS if [ "@WIN32@" != "1" ]; then # list_add CXXFLAGS "-Dmain=smpi_simulated_main_" list_add CXXFLAGS "-fpic" list_add LINKARGS "-shared" "-lsimgrid" else list_add CXXFLAGS "-include" "@includedir@/smpi/smpi_main.h" list_add LINKARGS "@libdir@\libsimgrid.dll" fi list_set CMDARGS while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDARGS "-c" ;; *.c) SRCFILE="$(readlink -f ${ARG} 2>/dev/null)" if [ -z "$SRCFILE" ] ; then SRCFILE="$ARG" fi list_add CMDARGS "${SRCFILE}" ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${CXX} --version ;; *) list_add CMDARGS "${ARG}" ;; esac done list_set CMDLINE "${CXX}" list_add_not_empty CMDLINE "${CXXFLAGS}" list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${CMDARGS}" list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.18/src/smpi/smpicc.in0000755000175000017500000000371713217757325016703 0ustar mquinsonmquinson#! /bin/sh # Copyright (c) 2007-2017. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" CC=@CMAKE_C_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set CFLAGS list_set LINKARGS if [ "x@WIN32@" = "x1" ]; then list_add CFLAGS "-include" "@includedir@/smpi/smpi_main.h" list_add LINKARGS "@libdir@\libsimgrid.dll" elif [ "x@APPLE@" = "x1" ]; then list_add CFLAGS "-fpic" list_add LINKARGS "-shared" "-lsimgrid" "-Wl,-undefined,error" else list_add CFLAGS "-fpic" list_add LINKARGS "-shared" "-lsimgrid" "-Wl,-z,defs" fi list_set CMDARGS while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDARGS "-c" ;; *.c) SRCFILE="$(readlink -f ${ARG} 2>/dev/null)" if [ -z "$SRCFILE" ] ; then SRCFILE="$ARG" fi list_add CMDARGS "${SRCFILE}" ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-trace-call-location') list_add_not_empty CMDARGS "-DTRACE_CALL_LOCATION" ;; '-compiler-version' | '--compiler-version') ${CC} --version ;; *) list_add CMDARGS "${ARG}" ;; esac done list_set CMDLINE "${CC}" list_add_not_empty CMDLINE "${CFLAGS}" list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${CMDARGS}" list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.18/src/smpi/smpi_main.c0000644000175000017500000000074213217757317017206 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include int main(int argc, char **argv) { if (argc < 2) { fprintf(stderr, "Usage: smpi_main \n"); exit(1); } return smpi_main(argv[1], argc - 1, argv + 1); } SimGrid-3.18/src/smpi/colls/0000755000175000017500000000000013217757321016172 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/alltoall/0000755000175000017500000000000013217757317020003 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/alltoall/alltoall-pair-light-barrier.cpp0000644000175000017500000000525113217757317026000 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j exchanges data with node i ^ j. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_pair_light_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_partner; int tag = COLL_TAG_ALLTOALL; /*, failure = 0; */ char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; Request::sendrecv(send_ptr + rank * send_chunk, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_partner = rank ^ (i + 1); Request::sendrecv(&send_sync, 1, MPI_CHAR, next_partner, tag, &recv_sync, 1, MPI_CHAR, next_partner, tag, comm, &s); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-ring-light-barrier.cpp0000644000175000017500000000517213217757317026006 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_ring_light_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_dst, next_src; int tag = COLL_TAG_ALLTOALL; char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; Request::sendrecv(send_ptr + rank * send_chunk, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_src = (rank - (i + 1) + num_procs) % num_procs; next_dst = (rank + (i + 1) + num_procs) % num_procs; Request::sendrecv(&send_sync, 1, MPI_CHAR, next_src, tag, &recv_sync, 1, MPI_CHAR, next_dst, tag, comm, &s); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-ring-mpi-barrier.cpp0000644000175000017500000000403513217757317025461 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. MPI barriers are added between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_ring_mpi_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Colls::barrier(comm); Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-3dmesh.cpp0000644000175000017500000001304313217757317023475 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include /***************************************************************************** * Function: alltoall_3dmesh_shoot * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the alltoall operation using the 3dmesh algorithm. It actually performs allgather operation in x dimension, y dimension, then in z dimension. Each node then extracts the needed data. The communication in all dimension is simple. * Auther: Ahmad Faraj ****************************************************************************/ static int alltoall_check_is_3dmesh(int num, int *i, int *j, int *k) { int x, max = num / 3; x = cbrt(num); *i = *j = *k = 0; while (x <= max) { if ((num % (x * x)) == 0) { *i = *j = x; *k = num / (x * x); return 1; } x++; } return 0; } namespace simgrid{ namespace smpi{ int Coll_alltoall_3dmesh::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *reqs, *req_ptr; MPI_Aint extent; MPI_Status status, *statuses; int i, j, src, dst, rank, num_procs, num_reqs, X, Y, Z, block_size, count; int my_z, two_dsize, my_row_base, my_col_base, my_z_base, src_row_base; int src_z_base, send_offset, recv_offset, tag = COLL_TAG_ALLTOALL; char *tmp_buff1, *tmp_buff2; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); if (not alltoall_check_is_3dmesh(num_procs, &X, &Y, &Z)) return MPI_ERR_OTHER; num_reqs = X; if (Y > X) num_reqs = Y; if (Z > Y) num_reqs = Z; two_dsize = X * Y; my_z = rank / two_dsize; my_row_base = (rank / X) * X; my_col_base = (rank % Y) + (my_z * two_dsize); my_z_base = my_z * two_dsize; block_size = extent * send_count; tmp_buff1 = (char *) smpi_get_tmp_sendbuffer(block_size * num_procs * two_dsize); tmp_buff2 = (char *) smpi_get_tmp_recvbuffer(block_size * two_dsize); statuses = (MPI_Status *) xbt_malloc(num_reqs * sizeof(MPI_Status)); reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = reqs; recv_offset = (rank % two_dsize) * block_size * num_procs; Request::sendrecv(send_buff, send_count * num_procs, send_type, rank, tag, tmp_buff1 + recv_offset, num_procs * recv_count, recv_type, rank, tag, comm, &status); count = send_count * num_procs; for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = (src % two_dsize) * block_size * num_procs; *(req_ptr++) = Request::irecv(tmp_buff1 + recv_offset, count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; Request::send(send_buff, count, send_type, dst, tag, comm); } Request::waitall(Y - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / X) * X; recv_offset = (src_row_base % two_dsize) * block_size * num_procs; *(req_ptr++) = Request::irecv(tmp_buff1 + recv_offset, recv_count * num_procs * Y, recv_type, src, tag, comm); } send_offset = (my_row_base % two_dsize) * block_size * num_procs; for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; Request::send(tmp_buff1 + send_offset, send_count * num_procs * Y, send_type, dst, tag, comm); } Request::waitall(X - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < two_dsize; i++) { send_offset = (rank * block_size) + (i * block_size * num_procs); recv_offset = (my_z_base * block_size) + (i * block_size); Request::sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &status); } for (i = 1; i < Z; i++) { src = (rank + i * two_dsize) % num_procs; src_z_base = (src / two_dsize) * two_dsize; recv_offset = (src_z_base * block_size); *(req_ptr++) = Request::irecv((char *) recv_buff + recv_offset, recv_count * two_dsize, recv_type, src, tag, comm); } for (i = 1; i < Z; i++) { dst = (rank + i * two_dsize) % num_procs; recv_offset = 0; for (j = 0; j < two_dsize; j++) { send_offset = (dst + j * num_procs) * block_size; Request::sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &status); recv_offset += block_size; } Request::send(tmp_buff2, send_count * two_dsize, send_type, dst, tag, comm); } Request::waitall(Z - 1, reqs, statuses); free(reqs); free(statuses); smpi_free_tmp_buffer(tmp_buff1); smpi_free_tmp_buffer(tmp_buff2); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-basic-linear.cpp0000644000175000017500000000477413217757317024656 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /*Naive and simple basic alltoall implementation. */ namespace simgrid{ namespace smpi{ int Coll_alltoall_basic_linear::alltoall(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = 888; int i; int count; MPI_Aint lb = 0, sendext = 0, recvext = 0; MPI_Request *requests; /* Initialize. */ int rank = comm->rank(); int size = comm->size(); XBT_DEBUG("<%d> algorithm alltoall_basic_linear() called.", rank); sendtype->extent(&lb, &sendext); recvtype->extent(&lb, &recvext); /* simple optimization */ int err = Datatype::copy(static_cast(sendbuf) + rank * sendcount * sendext, sendcount, sendtype, static_cast(recvbuf) + rank * recvcount * recvext, recvcount, recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ requests = xbt_new(MPI_Request, 2 * (size - 1)); /* Post all receives first -- a simple optimization */ count = 0; for (i = (rank + 1) % size; i != rank; i = (i + 1) % size) { requests[count] = Request::irecv_init(static_cast(recvbuf) + i * recvcount * recvext, recvcount, recvtype, i, system_tag, comm); count++; } /* Now post all sends in reverse order * - We would like to minimize the search time through message queue * when messages actually arrive in the order in which they were posted. * TODO: check the previous assertion */ for (i = (rank + size - 1) % size; i != rank; i = (i + size - 1) % size) { requests[count] = Request::isend_init(static_cast(sendbuf) + i * sendcount * sendext, sendcount, sendtype, i, system_tag, comm); count++; } /* Wait for them all. */ Request::startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); Request::waitall(count, requests, MPI_STATUS_IGNORE); for(i = 0; i < count; i++) { if(requests[i]!=MPI_REQUEST_NULL) Request::unref(&requests[i]); } xbt_free(requests); } return err; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-2dmesh.cpp0000644000175000017500000001175313217757317023502 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include /***************************************************************************** * Function: alltoall_2dmesh_shoot * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the alltoall operation using the 2dmesh algorithm. It actually performs allgather operation in x dimension then in the y dimension. Each node then extracts the needed data. The communication in each dimension follows "simple." * Auther: Ahmad Faraj ****************************************************************************/ static int alltoall_check_is_2dmesh(int num, int *i, int *j) { int x, max = num / 2; x = sqrt(num); while (x <= max) { if ((num % x) == 0) { *i = x; *j = num / x; if (*i > *j) { x = *i; *i = *j; *j = x; } return 1; } x++; } return 0; } namespace simgrid{ namespace smpi{ int Coll_alltoall_2dmesh::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status *statuses, s; MPI_Request *reqs, *req_ptr;; MPI_Aint extent; char *tmp_buff1, *tmp_buff2; int i, j, src, dst, rank, num_procs, count, num_reqs; int X, Y, send_offset, recv_offset; int my_row_base, my_col_base, src_row_base, block_size; int tag = COLL_TAG_ALLTOALL; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); if (not alltoall_check_is_2dmesh(num_procs, &X, &Y)) return MPI_ERR_OTHER; my_row_base = (rank / Y) * Y; my_col_base = rank % Y; block_size = extent * send_count; tmp_buff1 = (char *) smpi_get_tmp_sendbuffer(block_size * num_procs * Y); tmp_buff2 = (char *) smpi_get_tmp_recvbuffer(block_size * Y); num_reqs = X; if (Y > X) num_reqs = Y; statuses = (MPI_Status *) xbt_malloc(num_reqs * sizeof(MPI_Status)); reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = reqs; count = send_count * num_procs; for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = (src % Y) * block_size * num_procs; *(req_ptr++) = Request::irecv(tmp_buff1 + recv_offset, count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; Request::send(send_buff, count, send_type, dst, tag, comm); } Request::waitall(Y - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < Y; i++) { send_offset = (rank * block_size) + (i * block_size * num_procs); recv_offset = (my_row_base * block_size) + (i * block_size); if (i + my_row_base == rank) Request::sendrecv((char *) send_buff + recv_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &s); else Request::sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &s); } for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / Y) * Y; *(req_ptr++) = Request::irecv((char *) recv_buff + src_row_base * block_size, recv_count * Y, recv_type, src, tag, comm); } for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; recv_offset = 0; for (j = 0; j < Y; j++) { send_offset = (dst + j * num_procs) * block_size; if (j + my_row_base == rank) Request::sendrecv((char *) send_buff + dst * block_size, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &s); else Request::sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &s); recv_offset += block_size; } Request::send(tmp_buff2, send_count * Y, send_type, dst, tag, comm); } Request::waitall(X - 1, reqs, statuses); free(reqs); free(statuses); smpi_free_tmp_buffer(tmp_buff1); smpi_free_tmp_buffer(tmp_buff2); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-mvapich-scatter-dest.cpp0000644000175000017500000001217513217757317026346 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2012 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ //correct on stampede #define MV2_ALLTOALL_THROTTLE_FACTOR 4 #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_alltoall_mvapich2_scatter_dest::alltoall( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int comm_size, i, j; MPI_Aint sendtype_extent = 0, recvtype_extent = 0; int mpi_errno=MPI_SUCCESS; int dst, rank; MPI_Request *reqarray; MPI_Status *starray; if (recvcount == 0) return MPI_SUCCESS; comm_size = comm->size(); rank = comm->rank(); /* Get extent of send and recv types */ recvtype_extent = recvtype->get_extent(); sendtype_extent = sendtype->get_extent(); /* Medium-size message. Use isend/irecv with scattered destinations. Use Tony Ladd's modification to post only a small number of isends/irecvs at a time. */ /* FIXME: This converts the Alltoall to a set of blocking phases. Two alternatives should be considered: 1) the choice of communication pattern could try to avoid contending routes in each phase 2) rather than wait for all communication to finish (waitall), we could maintain constant queue size by using waitsome and posting new isend/irecv as others complete. This avoids synchronization delays at the end of each block (when there are only a few isend/irecvs left) */ int ii, ss, bblock; //Stampede is configured with bblock = MV2_ALLTOALL_THROTTLE_FACTOR;//mv2_coll_param.alltoall_throttle_factor; if (bblock >= comm_size) bblock = comm_size; /* If throttle_factor is n, each process posts n pairs of isend/irecv in each iteration. */ /* FIXME: This should use the memory macros (there are storage leaks here if there is an error, for example) */ reqarray= (MPI_Request*)xbt_malloc(2*bblock*sizeof(MPI_Request)); starray=(MPI_Status *)xbt_malloc(2*bblock*sizeof(MPI_Status)); for (ii=0; iisize(); rank = comm->rank(); extent = recv_type->get_extent(); tmp_buff = (char *) smpi_get_tmp_sendbuffer(num_procs * recv_count * extent); disps = (int *) xbt_malloc(sizeof(int) * num_procs); blocks_length = (int *) xbt_malloc(sizeof(int) * num_procs); Request::sendrecv(send_ptr + rank * send_count * extent, (num_procs - rank) * send_count, send_type, rank, tag, recv_ptr, (num_procs - rank) * recv_count, recv_type, rank, tag, comm, &status); Request::sendrecv(send_ptr, rank * send_count, send_type, rank, tag, recv_ptr + (num_procs - rank) * recv_count * extent, rank * recv_count, recv_type, rank, tag, comm, &status); MPI_Pack_size(send_count * num_procs, send_type, comm, &pack_size); while (pof2 < num_procs) { dst = (rank + pof2) % num_procs; src = (rank - pof2 + num_procs) % num_procs; count = 0; for (block = 1; block < num_procs; block++) if (block & pof2) { blocks_length[count] = send_count; disps[count] = block * send_count; count++; } MPI_Type_indexed(count, blocks_length, disps, recv_type, &new_type); new_type->commit(); position = 0; MPI_Pack(recv_buff, 1, new_type, tmp_buff, pack_size, &position, comm); Request::sendrecv(tmp_buff, position, MPI_PACKED, dst, tag, recv_buff, 1, new_type, src, tag, comm, &status); Datatype::unref(new_type); pof2 *= 2; } free(disps); free(blocks_length); Request::sendrecv(recv_ptr + (rank + 1) * recv_count * extent, (num_procs - rank - 1) * recv_count, send_type, rank, tag, tmp_buff, (num_procs - rank - 1) * recv_count, recv_type, rank, tag, comm, &status); Request::sendrecv(recv_ptr, (rank + 1) * recv_count, send_type, rank, tag, tmp_buff + (num_procs - rank - 1) * recv_count * extent, (rank + 1) * recv_count, recv_type, rank, tag, comm, &status); for (i = 0; i < num_procs; i++) Request::sendrecv(tmp_buff + i * recv_count * extent, recv_count, send_type, rank, tag, recv_ptr + (num_procs - i - 1) * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); smpi_free_tmp_buffer(tmp_buff); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-pair-mpi-barrier.cpp0000644000175000017500000000425313217757317025457 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. MPI barriers are inserted between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_pair_mpi_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Colls::barrier(comm); Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-ring.cpp0000644000175000017500000000361313217757317023253 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_ring::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-ring-one-barrier.cpp0000644000175000017500000000372313217757317025460 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_ring_one_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; Colls::barrier(comm); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-rdb.cpp0000644000175000017500000001167213217757317023067 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include "smpi_status.hpp" /***************************************************************************** * Function: alltoall_rdb * Return: int * inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the allgather operation using the recursive doubling algorithm. * Auther: MPICH / slightly modified by Ahmad Faraj. ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_rdb::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { /* MPI variables */ MPI_Status status; MPI_Aint send_increment, recv_increment, extent; int dst_tree_root, rank_tree_root, send_offset, recv_offset; int rank, num_procs, j, k, dst, curr_size, max_size; int last_recv_count = 0, tmp_mask, tree_root, num_procs_completed; int tag = COLL_TAG_ALLTOALL, mask = 1, i = 0; char *tmp_buff; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; num_procs = comm->size(); rank = comm->rank(); send_increment = send_type->get_extent(); recv_increment = recv_type->get_extent(); extent = recv_type->get_extent(); send_increment *= (send_count * num_procs); recv_increment *= (recv_count * num_procs); max_size = num_procs * recv_increment; tmp_buff = (char *) smpi_get_tmp_sendbuffer(max_size); curr_size = send_count * num_procs; Request::sendrecv(send_ptr, curr_size, send_type, rank, tag, tmp_buff + (rank * recv_increment), curr_size, recv_type, rank, tag, comm, &status); while (mask < num_procs) { dst = rank ^ mask; dst_tree_root = dst >> i; dst_tree_root <<= i; rank_tree_root = rank >> i; rank_tree_root <<= i; send_offset = rank_tree_root * send_increment; recv_offset = dst_tree_root * recv_increment; if (dst < num_procs) { Request::sendrecv(tmp_buff + send_offset, curr_size, send_type, dst, tag, tmp_buff + recv_offset, mask * recv_count * num_procs, recv_type, dst, tag, comm, &status); last_recv_count = Status::get_count(&status, recv_type); curr_size += last_recv_count; } if (dst_tree_root + mask > num_procs) { num_procs_completed = num_procs - rank_tree_root - mask; /* num_procs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + num_procs_completed) && (dst >= tree_root + num_procs_completed)) { Request::send(tmp_buff + dst_tree_root * send_increment, last_recv_count, send_type, dst, tag, comm); } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + num_procs_completed) && (rank >= tree_root + num_procs_completed)) { Request::recv(tmp_buff + dst_tree_root * send_increment, mask * num_procs * send_count, send_type, dst, tag, comm, &status); last_recv_count = Status::get_count(&status, send_type); curr_size += last_recv_count; } tmp_mask >>= 1; k--; } } mask <<= 1; i++; } for (i = 0; i < num_procs; i++) Request::sendrecv(tmp_buff + (rank + i * num_procs) * send_count * extent, send_count, send_type, rank, tag, recv_ptr + (i * recv_count * extent), recv_count, recv_type, rank, tag, comm, &status); smpi_free_tmp_buffer(tmp_buff); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-pair.cpp0000644000175000017500000000566413217757317023257 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include "smpi_win.hpp" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_pair_rma::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Win win; int assert = 0; int i, dst, rank, num_procs; char *send_ptr = (char *) send_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); win=new Win(recv_buff, num_procs * recv_chunk * send_count, recv_chunk, 0, comm); send_chunk *= send_count; recv_chunk *= recv_count; win->fence(assert); for (i = 0; i < num_procs; i++) { dst = rank ^ i; win->put(send_ptr + dst * send_chunk, send_count, send_type, dst, rank /* send_chunk*/, send_count, send_type); } win->fence(assert); delete win; return 0; } int Coll_alltoall_pair::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoall/alltoall-pair-one-barrier.cpp0000644000175000017500000000414213217757317025450 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoall_pair_one_barrier::alltoall(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); send_chunk *= send_count; recv_chunk *= recv_count; Colls::barrier(comm); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/colls_private.hpp0000644000175000017500000000104213217757321021546 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_COLLS_PRIVATE_HPP #define SMPI_COLLS_PRIVATE_HPP #include "private.hpp" #include "smpi/mpi.h" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" #include "smpi_request.hpp" #include XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_colls); #endif SimGrid-3.18/src/smpi/colls/alltoallv/0000755000175000017500000000000013217757317020171 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-ring.cpp0000644000175000017500000000411513217757317023625 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_ring::alltoallv(void* send_buff, int* send_counts, int* send_disps, MPI_Datatype send_type, void* recv_buff, int* recv_counts, int* recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); int pof2 = ((num_procs != 0) && ((num_procs & (~num_procs + 1)) == num_procs)); for (i = 0; i < num_procs; i++) { if (pof2 == 1) { /* use exclusive-or algorithm */ src = dst = rank ^ i; } else { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; } Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-pair-mpi-barrier.cpp0000644000175000017500000000430013217757317026024 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. MPI barriers are inserted between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_pair_mpi_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Colls::barrier(comm); Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-pair-one-barrier.cpp0000644000175000017500000000423713217757317026031 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_pair_one_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); Colls::barrier(comm); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-ring-mpi-barrier.cpp0000644000175000017500000000406113217757317026034 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. MPI barriers are added between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_ring_mpi_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Colls::barrier(comm); Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-ring-one-barrier.cpp0000644000175000017500000000374713217757317026042 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_ring_one_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); Colls::barrier(comm); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-bruck.cpp0000644000175000017500000000655613217757317024007 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /** * Alltoall Bruck * * Openmpi calls this routine when the message size sent to each rank < 2000 bytes and size < 12 * FIXME: uh, check smpi_pmpi again, but this routine is called for > 12, not * less... **/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_bruck::alltoallv(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLTOALLV; int i, rank, size, err, count; MPI_Aint lb; MPI_Aint sendext = 0; MPI_Aint recvext = 0; MPI_Request *requests; // FIXME: check implementation rank = comm->rank(); size = comm->size(); XBT_DEBUG("<%d> algorithm alltoall_bruck() called.", rank); sendtype->extent(&lb, &sendext); recvtype->extent(&lb, &recvext); /* Local copy from self */ err = Datatype::copy((char *)sendbuf + senddisps[rank] * sendext, sendcounts[rank], sendtype, (char *)recvbuf + recvdisps[rank] * recvext, recvcounts[rank], recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ int bblock = 4;//MPIR_PARAM_ALLTOALL_THROTTLE //if (bblock == 0) bblock = comm_size; // requests = xbt_new(MPI_Request, 2 * (bblock - 1)); int ii, ss, dst; /* post only bblock isends/irecvs at a time as suggested by Tony Ladd */ for (ii=0; ii skip request creation [src = %d, recvcount = %d]", rank, i, recvcounts[dst]); continue; } requests[count]=Request::irecv((char *)recvbuf + recvdisps[dst] * recvext, recvcounts[dst], recvtype, dst, system_tag, comm ); count++; } /* Now create all sends */ for ( i=0; i skip request creation [dst = %d, sendcount = %d]", rank, i, sendcounts[dst]); continue; } requests[count]=Request::isend((char *)sendbuf + senddisps[dst] * sendext, sendcounts[dst], sendtype, dst, system_tag, comm); count++; } /* Wait for them all. */ //Colls::startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); Request::waitall(count, requests, MPI_STATUSES_IGNORE); xbt_free(requests); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-ring-light-barrier.cpp0000644000175000017500000000526413217757317026364 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_ring_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_ring_light_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_dst, next_src; int tag = COLL_TAG_ALLTOALLV; char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); Request::sendrecv(send_ptr + send_disps[rank] * send_chunk, send_counts[rank], send_type, rank, tag, recv_ptr + recv_disps[rank] * recv_chunk, recv_counts[rank], recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_src = (rank - (i + 1) + num_procs) % num_procs; next_dst = (rank + (i + 1) + num_procs) % num_procs; Request::sendrecv(&send_sync, 1, MPI_CHAR, next_src, tag, &recv_sync, 1, MPI_CHAR, next_dst, tag, comm, &s); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-pair.cpp0000644000175000017500000000405613217757317023625 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_pair::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-ompi-basic-linear.cpp0000644000175000017500000000707513217757317026171 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* * Linear functions are copied from the basic coll module. For * some small number of nodes and/or small data sizes they are just as * fast as tuned/tree based segmenting operations and as such may be * selected by the decision functions. These are copied into this module * due to the way we select modules in V1. i.e. in V2 we will handle this * differently and so will not have to duplicate code. * GEF Oct05 after asking Jeff. */ namespace simgrid{ namespace smpi{ int Coll_alltoallv_ompi_basic_linear::alltoallv(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm) { int i, size, rank; char *psnd, *prcv; int nreqs; ptrdiff_t sext, rext; MPI_Request *preq; size = comm->size(); rank = comm->rank(); MPI_Request *ireqs= static_cast(xbt_malloc(sizeof(MPI_Request) * size * 2)); XBT_DEBUG( "coll:tuned:alltoallv_intra_basic_linear rank %d", rank); sext=sdtype->get_extent(); rext=rdtype->get_extent(); /* Simple optimization - handle send to self first */ psnd = ((char *) sbuf) + (sdisps[rank] * sext); prcv = ((char *) rbuf) + (rdisps[rank] * rext); if (0 != scounts[rank]) { Datatype::copy(psnd, scounts[rank], sdtype, prcv, rcounts[rank], rdtype); } /* If only one process, we're done. */ if (1 == size) { return MPI_SUCCESS; } /* Now, initiate all send/recv to/from others. */ nreqs = 0; preq = ireqs; /* Post all receives first */ for (i = 0; i < size; ++i) { if (i == rank || 0 == rcounts[i]) { continue; } prcv = ((char *) rbuf) + (rdisps[i] * rext); *preq = Request::irecv_init(prcv, rcounts[i], rdtype, i, COLL_TAG_ALLTOALLV, comm ); preq++; ++nreqs; } /* Now post all sends */ for (i = 0; i < size; ++i) { if (i == rank || 0 == scounts[i]) { continue; } psnd = ((char *) sbuf) + (sdisps[i] * sext); *preq=Request::isend_init(psnd, scounts[i], sdtype, i, COLL_TAG_ALLTOALLV, comm ); preq++; ++nreqs; } /* Start your engines. This will never return an error. */ Request::startall(nreqs, ireqs); /* Wait for them all. If there's an error, note that we don't care * what the error was -- just that there *was* an error. The PML * will finish all requests, even if one or more of them fail. * i.e., by the end of this call, all the requests are free-able. * So free them anyway -- even if there was an error, and return the * error after we free everything. */ Request::waitall(nreqs, ireqs, MPI_STATUSES_IGNORE); /* Free the requests. */ for (i = 0; i < nreqs; ++i) { if(ireqs[i]!=MPI_REQUEST_NULL) Request::unref(&ireqs[i]); } free(ireqs); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/alltoallv/alltoallv-pair-light-barrier.cpp0000644000175000017500000000534313217757317026356 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** * Function: alltoall_pair_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j exchanges data with node i ^ j. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_alltoallv_pair_light_barrier::alltoallv(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_partner; int tag = COLL_TAG_ALLTOALLV; /*, failure = 0; */ char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); Request::sendrecv(send_ptr + send_disps[rank] * send_chunk, send_counts[rank], send_type, rank, tag, recv_ptr + recv_disps[rank] * recv_chunk, recv_counts[rank], recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] *recv_chunk, recv_counts[dst], recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_partner = rank ^ (i + 1); Request::sendrecv(&send_sync, 1, MPI_CHAR, next_partner, tag, &recv_sync, 1, MPI_CHAR, next_partner, tag, comm, &s); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/coll_tuned_topo.hpp0000644000175000017500000000532013217757321022074 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2005 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #ifndef MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED #define MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED #include "colls_private.hpp" #define MAXTREEFANOUT 32 #define COLL_TUNED_COMPUTED_SEGCOUNT(SEGSIZE, TYPELNG, SEGCOUNT) \ if (((SEGSIZE) >= (TYPELNG)) && ((SEGSIZE) < ((TYPELNG) * (SEGCOUNT)))) { \ size_t residual; \ (SEGCOUNT) = (int)((SEGSIZE) / (TYPELNG)); \ residual = (SEGSIZE) - (SEGCOUNT) * (TYPELNG); \ if (residual > ((TYPELNG) >> 1)) \ (SEGCOUNT)++; \ } struct ompi_coll_tree_t { int32_t tree_root; int32_t tree_fanout; int32_t tree_bmtree; int32_t tree_prev; int32_t tree_next[MAXTREEFANOUT]; int32_t tree_nextsize; }; ompi_coll_tree_t* ompi_coll_tuned_topo_build_tree(int fanout, MPI_Comm com, int root); ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bintree(MPI_Comm comm); ompi_coll_tree_t* ompi_coll_tuned_topo_build_bmtree(MPI_Comm comm, int root); ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bmtree(MPI_Comm comm, int root); ompi_coll_tree_t* ompi_coll_tuned_topo_build_chain(int fanout, MPI_Comm com, int root); int ompi_coll_tuned_topo_destroy_tree(ompi_coll_tree_t** tree); /* debugging stuff, will be removed later */ int ompi_coll_tuned_topo_dump_tree(ompi_coll_tree_t* tree, int rank); #endif /* MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED */ SimGrid-3.18/src/smpi/colls/allgather/0000755000175000017500000000000013217757317020142 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/allgather/allgather-loosely-lr.cpp0000644000175000017500000001034213217757317024710 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_loosely_lr::allgather(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int comm_size, rank; int tag = COLL_TAG_ALLGATHER; int i, j, send_offset, recv_offset; int intra_rank, inter_rank, inter_comm_size, intra_comm_size; int inter_dst, inter_src; comm_size = comm->size(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } if(comm_size%num_core) THROWF(arg_error,0, "allgather loosely lr algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ",num_core); rank = comm->rank(); MPI_Aint rextent, sextent; rextent = rtype->get_extent(); sextent = stype->get_extent(); MPI_Request inter_rrequest; MPI_Request rrequest_array[128]; MPI_Request srequest_array[128]; MPI_Request inter_srequest_array[128]; int rrequest_count = 0; int srequest_count = 0; int inter_srequest_count = 0; MPI_Status status; intra_rank = rank % num_core; inter_rank = rank / num_core; inter_comm_size = (comm_size + num_core - 1) / num_core; intra_comm_size = num_core; int src_seg, dst_seg; //copy corresponding message from sbuf to rbuf recv_offset = rank * rextent * rcount; Request::sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + recv_offset, rcount, rtype, rank, tag, comm, &status); int dst, src; int inter_send_offset, inter_recv_offset; rrequest_count = 0; srequest_count = 0; inter_srequest_count = 0; for (i = 0; i < inter_comm_size; i++) { // inter_communication inter_dst = (rank + intra_comm_size) % comm_size; inter_src = (rank - intra_comm_size + comm_size) % comm_size; src_seg = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; dst_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; inter_send_offset = dst_seg * sextent * scount; inter_recv_offset = src_seg * rextent * rcount; for (j = 0; j < intra_comm_size; j++) { // inter communication if (intra_rank == j) { if (i != inter_comm_size - 1) { inter_rrequest = Request::irecv((char*)rbuf + inter_recv_offset, rcount, rtype, inter_src, tag, comm); inter_srequest_array[inter_srequest_count++] = Request::isend((char*)rbuf + inter_send_offset, scount, stype, inter_dst, tag, comm); } } //intra_communication src = inter_rank * intra_comm_size + j; dst = inter_rank * intra_comm_size + j; src_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + j; dst_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; send_offset = dst_seg * sextent * scount; recv_offset = src_seg * rextent * rcount; if (j != intra_rank) { rrequest_array[rrequest_count++] = Request::irecv((char *)rbuf + recv_offset, rcount, rtype, src, tag, comm); srequest_array[srequest_count++] = Request::isend((char *)rbuf + send_offset, scount, stype, dst, tag, comm); } } // intra loop // wait for inter communication to finish for these rounds (# of round equals num_core) if (i != inter_comm_size - 1) { Request::wait(&inter_rrequest, &status); } } //inter loop Request::waitall(rrequest_count, rrequest_array, MPI_STATUSES_IGNORE); Request::waitall(srequest_count, srequest_array, MPI_STATUSES_IGNORE); Request::waitall(inter_srequest_count, inter_srequest_array, MPI_STATUSES_IGNORE); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-mvapich-smp.cpp0000644000175000017500000001406113217757317025035 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_mvapich2_smp::allgather(void *sendbuf,int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt,MPI_Datatype recvtype, MPI_Comm comm) { int rank, size; int local_rank, local_size; int leader_comm_size = 0; int mpi_errno = MPI_SUCCESS; MPI_Aint recvtype_extent = 0; /* Datatype extent */ MPI_Comm shmem_comm, leader_comm; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } if (not comm->is_uniform() || not comm->is_blocked()) THROWF(arg_error,0, "allgather MVAPICH2 smp algorithm can't be used with irregular deployment. Please insure that processes deployed on the same node are contiguous and that each node has the same number of processes"); if (recvcnt == 0) { return MPI_SUCCESS; } rank = comm->rank(); size = comm->size(); /* extract the rank,size information for the intra-node communicator */ recvtype_extent=recvtype->get_extent(); shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader communicator */ leader_comm = comm->get_leaders_comm(); if(leader_comm==MPI_COMM_NULL){ leader_comm = MPI_COMM_WORLD; } leader_comm_size = leader_comm->size(); } /*If there is just one node, after gather itself, * root has all the data and it can do bcast*/ if(local_rank == 0) { mpi_errno = Colls::gather(sendbuf, sendcnt,sendtype, (void*)((char*)recvbuf + (rank * recvcnt * recvtype_extent)), recvcnt, recvtype, 0, shmem_comm); } else { /*Since in allgather all the processes could have * its own data in place*/ if(sendbuf == MPI_IN_PLACE) { mpi_errno = Colls::gather((void*)((char*)recvbuf + (rank * recvcnt * recvtype_extent)), recvcnt , recvtype, recvbuf, recvcnt, recvtype, 0, shmem_comm); } else { mpi_errno = Colls::gather(sendbuf, sendcnt,sendtype, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* Exchange the data between the node leaders*/ if (local_rank == 0 && (leader_comm_size > 1)) { /*When data in each socket is different*/ if (comm->is_uniform() != 1) { int *displs = NULL; int *recvcnts = NULL; int *node_sizes = NULL; int i = 0; node_sizes = comm->get_non_uniform_map(); displs = static_cast(xbt_malloc(sizeof (int) * leader_comm_size)); recvcnts = static_cast(xbt_malloc(sizeof (int) * leader_comm_size)); if (not displs || not recvcnts) { return MPI_ERR_OTHER; } recvcnts[0] = node_sizes[0] * recvcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * recvcnt; recvcnts[i] = node_sizes[i] * recvcnt; } void* sendbuf=((char*)recvbuf)+recvtype->get_extent()*displs[leader_comm->rank()]; mpi_errno = Colls::allgatherv(sendbuf, (recvcnt*local_size), recvtype, recvbuf, recvcnts, displs, recvtype, leader_comm); xbt_free(displs); xbt_free(recvcnts); } else { void* sendtmpbuf=((char*)recvbuf)+recvtype->get_extent()*(recvcnt*local_size)*leader_comm->rank(); mpi_errno = Coll_allgather_mpich::allgather(sendtmpbuf, (recvcnt*local_size), recvtype, recvbuf, (recvcnt*local_size), recvtype, leader_comm); } } /*Bcast the entire data from node leaders to all other cores*/ mpi_errno = Colls::bcast (recvbuf, recvcnt * size, recvtype, 0, shmem_comm); return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-rdb.cpp0000644000175000017500000001033313217757317023356 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include "smpi_status.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_rdb::allgather(void *sbuf, int send_count, MPI_Datatype send_type, void *rbuf, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { // MPI variables MPI_Status status; MPI_Aint send_chunk, recv_chunk; // local int variables unsigned int i, j, k, dst, send_offset, recv_offset, tree_root; int dst_tree_root, rank_tree_root, last_recv_count = 0, num_procs_completed; int offset, tmp_mask; int tag = COLL_TAG_ALLGATHER; unsigned int mask = 1; int success = 0; int curr_count = recv_count; // local string variables char *send_ptr = (char *) sbuf; char *recv_ptr = (char *) rbuf; // get size of the communicator, followed by rank unsigned int num_procs = comm->size(); unsigned int rank = comm->rank(); // get size of single element's type for send buffer and recv buffer send_chunk = send_type->get_extent(); recv_chunk = recv_type->get_extent(); // multiply size of each element by number of elements to send or recv send_chunk *= send_count; recv_chunk *= recv_count; // perform a local copy Request::sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &status); i = 0; while (mask < num_procs) { dst = rank ^ mask; dst_tree_root = dst >> i; dst_tree_root <<= i; rank_tree_root = rank >> i; rank_tree_root <<= i; send_offset = rank_tree_root * send_chunk; recv_offset = dst_tree_root * recv_chunk; if (dst < num_procs) { Request::sendrecv(recv_ptr + send_offset, curr_count, send_type, dst, tag, recv_ptr + recv_offset, mask * recv_count, recv_type, dst, tag, comm, &status); last_recv_count = Status::get_count(&status, recv_type); curr_count += last_recv_count; } if (dst_tree_root + mask > num_procs) { num_procs_completed = num_procs - rank_tree_root - mask; /* num_procs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; offset = recv_chunk * (rank_tree_root + mask); tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + num_procs_completed) && (dst >= tree_root + num_procs_completed)) { Request::send(recv_ptr + offset, last_recv_count, recv_type, dst, tag, comm); /* last_recv_cnt was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + num_procs_completed) && (rank >= tree_root + num_procs_completed)) { Request::recv(recv_ptr + offset, recv_count * num_procs_completed, recv_type, dst, tag, comm, &status); // num_procs_completed is also equal to the no. of processes // whose data we don't have last_recv_count = Status::get_count(&status, recv_type); curr_count += last_recv_count; } tmp_mask >>= 1; k--; } } mask <<= 1; i++; } return success; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-bruck.cpp0000644000175000017500000001316413217757317023722 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_bruck * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the bruck * algorithm. * Auther: MPICH * Comment: Original bruck algorithm from MPICH is slightly modified by * Ahmad Faraj. ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_bruck::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { // MPI variables MPI_Status status; MPI_Aint recv_extent; // local int variables int src, dst, rank, num_procs, count, remainder; int tag = COLL_TAG_ALLGATHER; int pof2 = 1; // local string variables char *tmp_buff; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; // get size of the communicator, followed by rank num_procs = comm->size(); rank = comm->rank(); // get size of single element's type for recv buffer recv_extent = recv_type->get_extent(); count = recv_count; tmp_buff = (char *) smpi_get_tmp_sendbuffer(num_procs * recv_count * recv_extent); // perform a local copy Datatype::copy(send_ptr, send_count, send_type, tmp_buff, recv_count, recv_type); while (pof2 <= (num_procs / 2)) { src = (rank + pof2) % num_procs; dst = (rank - pof2 + num_procs) % num_procs; Request::sendrecv(tmp_buff, count, recv_type, dst, tag, tmp_buff + count * recv_extent, count, recv_type, src, tag, comm, &status); count *= 2; pof2 *= 2; } remainder = num_procs - pof2; if (remainder) { src = (rank + pof2) % num_procs; dst = (rank - pof2 + num_procs) % num_procs; Request::sendrecv(tmp_buff, remainder * recv_count, recv_type, dst, tag, tmp_buff + count * recv_extent, remainder * recv_count, recv_type, src, tag, comm, &status); } Request::sendrecv(tmp_buff, (num_procs - rank) * recv_count, recv_type, rank, tag, recv_ptr + rank * recv_count * recv_extent, (num_procs - rank) * recv_count, recv_type, rank, tag, comm, &status); if (rank) Request::sendrecv(tmp_buff + (num_procs - rank) * recv_count * recv_extent, rank * recv_count, recv_type, rank, tag, recv_ptr, rank * recv_count, recv_type, rank, tag, comm, &status); smpi_free_tmp_buffer(tmp_buff); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-GB.cpp0000644000175000017500000000161113217757317023076 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ // Allgather - gather/bcast algorithm int Coll_allgather_GB::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { int num_procs; num_procs = comm->size(); Colls::gather(send_buff, send_count, send_type, recv_buff, recv_count, recv_type, 0, comm); Colls::bcast(recv_buff, (recv_count * num_procs), recv_type, 0, comm); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-NTSLR-NB.cpp0000644000175000017500000000450313217757317024010 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ // Allgather-Non-Topoloty-Scecific-Logical-Ring algorithm int Coll_allgather_NTSLR_NB::allgather(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { MPI_Aint rextent, sextent; MPI_Status status, status2; int i, to, from, rank, size; int send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; rank = comm->rank(); size = comm->size(); rextent = rtype->get_extent(); sextent = stype->get_extent(); MPI_Request *rrequest_array; MPI_Request *srequest_array; rrequest_array = (MPI_Request *) xbt_malloc(size * sizeof(MPI_Request)); srequest_array = (MPI_Request *) xbt_malloc(size * sizeof(MPI_Request)); // irregular case use default MPI fucntions if (scount * sextent != rcount * rextent) { XBT_WARN("MPI_allgather_NTSLR_NB use default MPI_allgather."); Coll_allgather_default::allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // topo non-specific to = (rank + 1) % size; from = (rank + size - 1) % size; //copy a single segment from sbuf to rbuf send_offset = rank * scount * sextent; Request::sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + send_offset, rcount, rtype, rank, tag, comm, &status); //start sending logical ring message int increment = scount * sextent; //post all irecv first for (i = 0; i < size - 1; i++) { recv_offset = ((rank - i - 1 + size) % size) * increment; rrequest_array[i] = Request::irecv((char *)rbuf + recv_offset, rcount, rtype, from, tag + i, comm); } for (i = 0; i < size - 1; i++) { send_offset = ((rank - i + size) % size) * increment; srequest_array[i] = Request::isend((char *)rbuf + send_offset, scount, stype, to, tag + i, comm); Request::wait(&rrequest_array[i], &status); Request::wait(&srequest_array[i], &status2); } free(rrequest_array); free(srequest_array); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-SMP-NTS.cpp0000644000175000017500000001376213217757317023721 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_SMP_NTS::allgather(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int src, dst, comm_size, rank; comm_size = comm->size(); rank = comm->rank(); MPI_Aint rextent, sextent; rextent = rtype->get_extent(); sextent = stype->get_extent(); int tag = COLL_TAG_ALLGATHER; int i, send_offset, recv_offset; int intra_rank, inter_rank; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } intra_rank = rank % num_core; inter_rank = rank / num_core; int inter_comm_size = (comm_size + num_core - 1) / num_core; int num_core_in_current_smp = num_core; if(comm_size%num_core) THROWF(arg_error,0, "allgather SMP NTS algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ", num_core); /* for too small number of processes, use default implementation */ if (comm_size <= num_core) { XBT_WARN("MPI_allgather_SMP_NTS use default MPI_allgather."); Coll_allgather_default::allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // the last SMP node may have fewer number of running processes than all others if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } //copy corresponding message from sbuf to rbuf recv_offset = rank * rextent * rcount; Request::sendrecv(sbuf, scount, stype, rank, tag, ((char *) rbuf + recv_offset), rcount, rtype, rank, tag, comm, MPI_STATUS_IGNORE); //gather to root of each SMP for (i = 1; i < num_core_in_current_smp; i++) { dst = (inter_rank * num_core) + (intra_rank + i) % (num_core_in_current_smp); src = (inter_rank * num_core) + (intra_rank - i + num_core_in_current_smp) % (num_core_in_current_smp); recv_offset = src * rextent * rcount; Request::sendrecv(sbuf, scount, stype, dst, tag, ((char *) rbuf + recv_offset), rcount, rtype, src, tag, comm, MPI_STATUS_IGNORE); } // INTER-SMP-ALLGATHER // Every root of each SMP node post INTER-Sendrecv, then do INTRA-Bcast for each receiving message // Use logical ring algorithm // root of each SMP if (intra_rank == 0) { MPI_Request *rrequest_array = xbt_new(MPI_Request, inter_comm_size - 1); MPI_Request *srequest_array = xbt_new(MPI_Request, inter_comm_size - 1); src = ((inter_rank - 1 + inter_comm_size) % inter_comm_size) * num_core; dst = ((inter_rank + 1) % inter_comm_size) * num_core; // post all inter Irecv for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; rrequest_array[i] = Request::irecv((char *)rbuf + recv_offset, rcount * num_core, rtype, src, tag + i, comm); } // send first message send_offset = ((inter_rank + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; srequest_array[0] = Request::isend((char *)rbuf + send_offset, scount * num_core, stype, dst, tag, comm); // loop : recv-inter , send-inter, send-intra (linear-bcast) for (i = 0; i < inter_comm_size - 2; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; Request::wait(&rrequest_array[i], MPI_STATUS_IGNORE); srequest_array[i + 1] = Request::isend((char *)rbuf + recv_offset, scount * num_core, stype, dst, tag + i + 1, comm); if (num_core_in_current_smp > 1) { Request::send((char *)rbuf + recv_offset, scount * num_core, stype, (rank + 1), tag + i + 1, comm); } } // recv last message and send_intra recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; //recv_offset = ((inter_rank + 1) % inter_comm_size) * num_core * sextent * scount; //i=inter_comm_size-2; Request::wait(&rrequest_array[i], MPI_STATUS_IGNORE); if (num_core_in_current_smp > 1) { Request::send((char *)rbuf + recv_offset, scount * num_core, stype, (rank + 1), tag + i + 1, comm); } Request::waitall(inter_comm_size - 1, srequest_array, MPI_STATUSES_IGNORE); xbt_free(rrequest_array); xbt_free(srequest_array); } // last rank of each SMP else if (intra_rank == (num_core_in_current_smp - 1)) { for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; Request::recv((char *) rbuf + recv_offset, (rcount * num_core), rtype, rank - 1, tag + i + 1, comm, MPI_STATUS_IGNORE); } } // intermediate rank of each SMP else { for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; Request::recv((char *) rbuf + recv_offset, (rcount * num_core), rtype, rank - 1, tag + i + 1, comm, MPI_STATUS_IGNORE); Request::send((char *) rbuf + recv_offset, (scount * num_core), stype, (rank + 1), tag + i + 1, comm); } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-rhv.cpp0000644000175000017500000000613313217757317023411 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ // now only work with power of two processes int Coll_allgather_rhv::allgather(void *sbuf, int send_count, MPI_Datatype send_type, void *rbuf, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status status; MPI_Aint s_extent, r_extent; // local int variables int i, dst, send_base_offset, recv_base_offset, send_chunk, recv_chunk, send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; unsigned int mask; int curr_count; // get size of the communicator, followed by rank unsigned int num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgather rhv algorithm can't be used with non power of two number of processes ! "); unsigned int rank = comm->rank(); // get size of single element's type for send buffer and recv buffer s_extent = send_type->get_extent(); r_extent = recv_type->get_extent(); // multiply size of each element by number of elements to send or recv send_chunk = s_extent * send_count; recv_chunk = r_extent * recv_count; if (send_chunk != recv_chunk) { XBT_WARN("MPI_allgather_rhv use default MPI_allgather."); Coll_allgather_default::allgather(sbuf, send_count, send_type, rbuf, recv_count, recv_type, comm); return MPI_SUCCESS; } // compute starting offset location to perform local copy int size = num_procs / 2; int base_offset = 0; mask = 1; while (mask < num_procs) { if (rank & mask) { base_offset += size; } mask <<= 1; size /= 2; } // printf("node %d base_offset %d\n",rank,base_offset); //perform a remote copy dst = base_offset; Request::sendrecv(sbuf, send_count, send_type, dst, tag, (char *)rbuf + base_offset * recv_chunk, recv_count, recv_type, dst, tag, comm, &status); mask >>= 1; i = 1; int phase = 0; curr_count = recv_count; while (mask >= 1) { // destination pair for both send and recv dst = rank ^ mask; // compute offsets send_base_offset = base_offset; if (rank & mask) { recv_base_offset = base_offset - i; base_offset -= i; } else { recv_base_offset = base_offset + i; } send_offset = send_base_offset * recv_chunk; recv_offset = recv_base_offset * recv_chunk; // printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_base_offset, recv_base_offset, curr_count); Request::sendrecv((char*)rbuf + send_offset, curr_count, recv_type, dst, tag, (char*)rbuf + recv_offset, curr_count, recv_type, dst, tag, comm, &status); curr_count *= 2; i *= 2; mask >>= 1; phase++; } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-spreading-simple.cpp0000644000175000017500000001201413217757317026050 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_spreading_simple * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Let i -> j denote the communication from node i to node j. The * order of communications for node i is i -> i + 1, i -> i + 2, ..., * i -> (i + p -1) % P. * * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_spreading_simple::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *reqs, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs, num_reqs; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *recv_ptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); num_reqs = (2 * num_procs) - 2; reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); if (not reqs) { printf("allgather-spreading-simple.c:40: cannot allocate memory\n"); MPI_Finalize(); exit(0); } req_ptr = reqs; Request::sendrecv(send_buff, send_count, send_type, rank, tag, (char *) recv_buff + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 0; i < num_procs; i++) { src = (rank + i) % num_procs; if (src == rank) continue; *(req_ptr++) = Request::irecv(recv_ptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm); } for (i = 0; i < num_procs; i++) { dst = (rank + i) % num_procs; if (dst == rank) continue; *(req_ptr++) = Request::isend(send_buff, send_count, send_type, dst, tag, comm); } Request::waitall(num_reqs, reqs, MPI_STATUSES_IGNORE); free(reqs); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-ring.cpp0000644000175000017500000001057213217757317023553 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j+ i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_ring::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *sendptr = (char *) send_buff; char *recvptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); // local send/recv Request::sendrecv(sendptr, send_count, send_type, rank, tag, recvptr + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(sendptr, send_count, send_type, dst, tag, recvptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-3dmesh.cpp0000644000175000017500000001641213217757317023776 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: is_2dmesh * return: int * num: the number of processors in a communicator * i: x dimension * j: y dimension * k: z dimension * descp: takes a number and tries to find a factoring of x*y*z mesh out of it ****************************************************************************/ #ifndef THREED #define THREED static int is_3dmesh(int num, int *i, int *j, int *k) { int x, max = num / 3; x = cbrt(num); *i = *j = *k = 0; while (x <= max) { if ((num % (x * x)) == 0) { *i = *j = x; *k = num / (x * x); return 1; } x++; } return 0; } #endif /***************************************************************************** * Function: allgather_3dmesh_shoot * return: int * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the 2dmesh * algorithm. Allgather ommunication occurs first in the x dimension, y * dimension, and then in the z dimension. Communication in each dimension * follows "simple" * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_3dmesh::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *req, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs, block_size, my_z_base; int my_z, X, Y, Z, send_offset, recv_offset; int two_dsize, my_row_base, my_col_base, src_row_base, src_z_base, num_reqs; int tag = COLL_TAG_ALLGATHER; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); if (not is_3dmesh(num_procs, &X, &Y, &Z)) THROWF(arg_error,0, "allgather_3dmesh algorithm can't be used with this number of processes! "); num_reqs = X; if (Y > X) num_reqs = Y; if (Z > Y) num_reqs = Z; two_dsize = X * Y; my_z = rank / two_dsize; my_row_base = (rank / X) * X; my_col_base = (rank % Y) + (my_z * two_dsize); my_z_base = my_z * two_dsize; block_size = extent * send_count; req = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = req; // do local allgather/local copy recv_offset = rank * block_size; Datatype::copy(send_buff, send_count, send_type, (char *)recv_buff + recv_offset, recv_count, recv_type); // do rowwise comm for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = src * block_size; *(req_ptr++) = Request::irecv((char *)recv_buff + recv_offset, send_count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; Request::send(send_buff, send_count, send_type, dst, tag, comm); } Request::waitall(Y - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; // do colwise comm, it does not matter here if i*X or i *Y since X == Y for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / X) * X; recv_offset = src_row_base * block_size; *(req_ptr++) = Request::irecv((char *)recv_buff + recv_offset, recv_count * Y, recv_type, src, tag, comm); } send_offset = my_row_base * block_size; for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; Request::send((char *)recv_buff + send_offset, send_count * Y, send_type, dst, tag, comm); } Request::waitall(X - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; for (i = 1; i < Z; i++) { src = (rank + i * two_dsize) % num_procs; src_z_base = (src / two_dsize) * two_dsize; recv_offset = (src_z_base * block_size); *(req_ptr++) = Request::irecv((char *)recv_buff + recv_offset, recv_count * two_dsize, recv_type, src, tag, comm); } for (i = 1; i < Z; i++) { dst = (rank + i * two_dsize) % num_procs; send_offset = my_z_base * block_size; Request::send((char *)recv_buff + send_offset, send_count * two_dsize, send_type, dst, tag, comm); } Request::waitall(Z - 1, req, MPI_STATUSES_IGNORE); free(req); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-NTSLR.cpp0000644000175000017500000000360413217757317023514 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ // Allgather-Non-Topoloty-Scecific-Logical-Ring algorithm int Coll_allgather_NTSLR::allgather(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { MPI_Aint rextent, sextent; MPI_Status status; int i, to, from, rank, size; int send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; rank = comm->rank(); size = comm->size(); rextent = rtype->get_extent(); sextent = stype->get_extent(); // irregular case use default MPI fucntions if (scount * sextent != rcount * rextent) { XBT_WARN("MPI_allgather_NTSLR use default MPI_allgather."); Coll_allgather_default::allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // topo non-specific to = (rank + 1) % size; from = (rank + size - 1) % size; //copy a single segment from sbuf to rbuf send_offset = rank * scount * sextent; Request::sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + send_offset, rcount, rtype, rank, tag, comm, &status); //start sending logical ring message int increment = scount * sextent; for (i = 0; i < size - 1; i++) { send_offset = ((rank - i + size) % size) * increment; recv_offset = ((rank - i - 1 + size) % size) * increment; Request::sendrecv((char *) rbuf + send_offset, scount, stype, to, tag + i, (char *) rbuf + recv_offset, rcount, rtype, from, tag + i, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-pair.cpp0000644000175000017500000001104313217757317023541 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_pair * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works when P is power of two. In each phase of P - 1 * phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_pair::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; unsigned int i, src, dst; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; unsigned int rank = comm->rank(); unsigned int num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgather pair algorithm can't be used with non power of two number of processes ! "); extent = send_type->get_extent(); // local send/recv Request::sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr, send_count, send_type, dst, tag, recv_ptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-ompi-neighborexchange.cpp0000644000175000017500000001527413217757317027062 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * ompi_coll_tuned_allgather_intra_neighborexchange * * Function: allgather using N/2 steps (O(N)) * Accepts: Same arguments as MPI_Allgather * Returns: MPI_SUCCESS or error code * * Description: Neighbor Exchange algorithm for allgather. * Described by Chen et.al. in * "Performance Evaluation of Allgather Algorithms on * Terascale Linux Cluster with Fast Ethernet", * Proceedings of the Eighth International Conference on * High-Performance Computing inn Asia-Pacific Region * (HPCASIA'05), 2005 * * Rank r exchanges message with one of its neighbors and * forwards the data further in the next step. * * No additional memory requirements. * * Limitations: Algorithm works only on even number of processes. * For odd number of processes we switch to ring algorithm. * * Example on 6 nodes: * Initial state * # 0 1 2 3 4 5 * [0] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] * [ ] [ ] [ ] [ ] [ ] [5] * Step 0: * # 0 1 2 3 4 5 * [0] [0] [ ] [ ] [ ] [ ] * [1] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [2] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [4] * [ ] [ ] [ ] [ ] [5] [5] * Step 1: * # 0 1 2 3 4 5 * [0] [0] [0] [ ] [ ] [0] * [1] [1] [1] [ ] [ ] [1] * [ ] [2] [2] [2] [2] [ ] * [ ] [3] [3] [3] [3] [ ] * [4] [ ] [ ] [4] [4] [4] * [5] [ ] [ ] [5] [5] [5] * Step 2: * # 0 1 2 3 4 5 * [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_ompi_neighborexchange::allgather(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int line = -1; int rank, size; int neighbor[2], offset_at_step[2], recv_data_from[2], send_data_from; int i, even_rank; int err = 0; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; size = comm->size(); rank = comm->rank(); if (size % 2) { XBT_DEBUG( "coll:tuned:allgather_intra_neighborexchange WARNING: odd size %d, switching to ring algorithm", size); return Coll_allgather_ring::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } XBT_DEBUG( "coll:tuned:allgather_intra_neighborexchange rank %d", rank); err = sdtype->extent(&slb, &sext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = rdtype->extent(&rlb, &rext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to appropriate block of receive buffer */ tmprecv = (char*) rbuf + rank * rcount * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; Datatype::copy (tmpsend, scount, sdtype, tmprecv, rcount, rdtype); } /* Determine neighbors, order in which blocks will arrive, etc. */ even_rank = not(rank % 2); if (even_rank) { neighbor[0] = (rank + 1) % size; neighbor[1] = (rank - 1 + size) % size; recv_data_from[0] = rank; recv_data_from[1] = rank; offset_at_step[0] = (+2); offset_at_step[1] = (-2); } else { neighbor[0] = (rank - 1 + size) % size; neighbor[1] = (rank + 1) % size; recv_data_from[0] = neighbor[0]; recv_data_from[1] = neighbor[0]; offset_at_step[0] = (-2); offset_at_step[1] = (+2); } /* Communication loop: - First step is special: exchange a single block with neighbor[0]. - Rest of the steps: update recv_data_from according to offset, and exchange two blocks with appropriate neighbor. the send location becomes previous receve location. */ tmprecv = (char*)rbuf + neighbor[0] * rcount * rext; tmpsend = (char*)rbuf + rank * rcount * rext; /* Sendreceive */ Request::sendrecv(tmpsend, rcount, rdtype, neighbor[0], COLL_TAG_ALLGATHER, tmprecv, rcount, rdtype, neighbor[0], COLL_TAG_ALLGATHER, comm, MPI_STATUS_IGNORE); /* Determine initial sending location */ if (even_rank) { send_data_from = rank; } else { send_data_from = recv_data_from[0]; } for (i = 1; i < (size / 2); i++) { const int i_parity = i % 2; recv_data_from[i_parity] = (recv_data_from[i_parity] + offset_at_step[i_parity] + size) % size; tmprecv = (char*)rbuf + recv_data_from[i_parity] * rcount * rext; tmpsend = (char*)rbuf + send_data_from * rcount * rext; /* Sendreceive */ Request::sendrecv(tmpsend, 2 * rcount, rdtype, neighbor[i_parity], COLL_TAG_ALLGATHER, tmprecv, 2 * rcount, rdtype, neighbor[i_parity], COLL_TAG_ALLGATHER, comm, MPI_STATUS_IGNORE); send_data_from = recv_data_from[i_parity]; } return MPI_SUCCESS; err_hndl: XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-smp-simple.cpp0000644000175000017500000001131113217757317024672 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgather_smp_simple::allgather(void *send_buf, int scount, MPI_Datatype stype, void *recv_buf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int src, dst, comm_size, rank; comm_size = comm->size(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } if(comm_size%num_core) THROWF(arg_error,0, "allgather SMP simple algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ", num_core); rank = comm->rank(); MPI_Aint rextent, sextent; rextent = rtype->get_extent(); sextent = stype->get_extent(); int tag = COLL_TAG_ALLGATHER; MPI_Status status; int i, send_offset, recv_offset; int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; int inter_comm_size = (comm_size + num_core - 1) / num_core; int num_core_in_current_smp = num_core; // the last SMP node may have fewer number of running processes than all others if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } //INTRA-SMP-ALLGATHER recv_offset = rank * rextent * rcount; Request::sendrecv(send_buf, scount, stype, rank, tag, ((char *) recv_buf + recv_offset), rcount, rtype, rank, tag, comm, &status); for (i = 1; i < num_core_in_current_smp; i++) { dst = (inter_rank * num_core) + (intra_rank + i) % (num_core_in_current_smp); src = (inter_rank * num_core) + (intra_rank - i + num_core_in_current_smp) % (num_core_in_current_smp); recv_offset = src * rextent * rcount; Request::sendrecv(send_buf, scount, stype, dst, tag, ((char *) recv_buf + recv_offset), rcount, rtype, src, tag, comm, &status); } // INTER-SMP-ALLGATHER // Every root of each SMP node post INTER-Sendrecv, then do INTRA-Bcast for each receiving message if (intra_rank == 0) { MPI_Request *reqs, *req_ptr; int num_req = (inter_comm_size - 1) * 2; reqs = (MPI_Request *) xbt_malloc(num_req * sizeof(MPI_Request)); req_ptr = reqs; MPI_Status *stat; stat = (MPI_Status *) xbt_malloc(num_req * sizeof(MPI_Status)); for (i = 1; i < inter_comm_size; i++) { //dst = ((inter_rank+i)%inter_comm_size) * num_core; src = ((inter_rank - i + inter_comm_size) % inter_comm_size) * num_core; //send_offset = (rank * sextent * scount); recv_offset = (src * sextent * scount); // Request::sendrecv((recv_buf+send_offset), (scount * num_core), stype, dst, tag, // (recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, &status); //MPIC_Isend((recv_buf+send_offset), (scount * num_core), stype, dst, tag, comm, req_ptr++); *(req_ptr++) = Request::irecv(((char *) recv_buf + recv_offset), (rcount * num_core), rtype, src, tag, comm); } for (i = 1; i < inter_comm_size; i++) { dst = ((inter_rank + i) % inter_comm_size) * num_core; //src = ((inter_rank-i+inter_comm_size)%inter_comm_size) * num_core; send_offset = (rank * sextent * scount); //recv_offset = (src * sextent * scount); // Request::sendrecv((recv_buf+send_offset), (scount * num_core), stype, dst, tag, // (recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, &status); *(req_ptr++) = Request::isend(((char *) recv_buf + send_offset), (scount * num_core), stype, dst, tag, comm); //MPIC_Irecv((recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, req_ptr++); } Request::waitall(num_req, reqs, stat); free(reqs); free(stat); } //INTRA-BCAST (use flat tree) if (intra_rank == 0) { for (i = 1; i < num_core_in_current_smp; i++) { //printf("rank = %d, num = %d send to %d\n",rank, num_core_in_current_smp, (rank + i)); Request::send(recv_buf, (scount * comm_size), stype, (rank + i), tag, comm); } } else { //printf("rank = %d recv from %d\n",rank, (inter_rank * num_core)); Request::recv(recv_buf, (rcount * comm_size), rtype, (inter_rank * num_core), tag, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgather/allgather-2dmesh.cpp0000644000175000017500000001467013217757317024001 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: is_2dmesh * Return: int * Inputs: num: the number of processors in a communicator i: x dimension j: y dimension * Descp: takes a number and tries to find a factoring of x, y mesh out of it * Auther: Ahmad Faraj ****************************************************************************/ #ifndef TWOD #define TWOD static int is_2dmesh(int num, int *i, int *j) { int x, max = num / 2; x = sqrt(num); while (x <= max) { if ((num % x) == 0) { *i = x; *j = num / x; if (*i > *j) { x = *i; *i = *j; *j = x; } return 1; } x++; } return 0; } #endif /***************************************************************************** * Function: allgather_2dmesh_shoot * return: int * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the 2dmesh * algorithm. Allgather ommunication occurs first in the x dimension then in * the y dimension. The communication in each dimension follows * "simple" * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgather_2dmesh::allgather(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *req, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs; int X, Y, send_offset, recv_offset; int my_row_base, my_col_base, src_row_base, block_size, num_reqs; int tag = COLL_TAG_ALLGATHER; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); block_size = extent * send_count; if (not is_2dmesh(num_procs, &X, &Y)) THROWF(arg_error,0, "allgather_2dmesh algorithm can't be used with this number of processes! "); my_row_base = (rank / Y) * Y; my_col_base = rank % Y; num_reqs = X; if (Y > X) num_reqs = Y; req = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = req; // do local allgather/local copy recv_offset = rank * block_size; Datatype::copy(send_buff, send_count, send_type, (char *)recv_buff + recv_offset, recv_count, recv_type); // do row-wise comm for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = src * block_size; *(req_ptr++) = Request::irecv((char *)recv_buff + recv_offset, recv_count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; Request::send(send_buff, send_count, send_type, dst, tag, comm); } Request::waitall(Y - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; // do colwise comm for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / Y) * Y; recv_offset = src_row_base * block_size; *(req_ptr++) = Request::irecv((char *)recv_buff + recv_offset, recv_count * Y, recv_type, src, tag, comm); } for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; send_offset = my_row_base * block_size; Request::send((char *)recv_buff + send_offset, send_count * Y, send_type, dst, tag, comm); } Request::waitall(X - 1, req, MPI_STATUSES_IGNORE); free(req); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/coll_tuned_topo.cpp0000644000175000017500000004276613217757320022105 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2005 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "coll_tuned_topo.hpp" #include "colls_private.hpp" /* * Some static helpers. */ static int pown( int fanout, int num ) { int j, p = 1; if( num < 0 ) return 0; if (1==num) return fanout; if (2==fanout) { return p< 1) return ((pown(fanout,level) - 1)/(fanout - 1)); else return 0; // is this right ? } /* * And now the building functions. * * An example for fanout = 2, comm_size = 7 * * 0 <-- delta = 1 (fanout^0) * / \ * 1 2 <-- delta = 2 (fanout^1) * / \ / \ * 3 5 4 6 <-- delta = 4 (fanout^2) */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_tree( int fanout, MPI_Comm comm, int root ) { int rank, size; int schild, sparent; int level; /* location of my rank in the tree structure of size */ int delta; /* number of nodes on my level */ int slimit; /* total number of nodes on levels above me */ int shiftedrank; int i; ompi_coll_tree_t* tree; XBT_DEBUG("coll:tuned:topo_build_tree Building fo %d rt %d", fanout, root); if (fanout<1) { XBT_DEBUG("coll:tuned:topo_build_tree invalid fanout %d", fanout); return NULL; } if (fanout>MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo_build_tree invalid fanout %d bigger than max %d", fanout, MAXTREEFANOUT); return NULL; } /* * Get size and rank of the process in this communicator */ size = comm->size(); rank = comm->rank(); tree = new ompi_coll_tree_t; if (not tree) { XBT_DEBUG("coll:tuned:topo_build_tree PANIC::out of memory"); return NULL; } tree->tree_root = MPI_UNDEFINED; tree->tree_nextsize = MPI_UNDEFINED; /* * Set root */ tree->tree_root = root; /* * Initialize tree */ tree->tree_fanout = fanout; tree->tree_bmtree = 0; tree->tree_root = root; tree->tree_prev = -1; tree->tree_nextsize = 0; for( i = 0; i < fanout; i++ ) { tree->tree_next[i] = -1; } /* return if we have less than 2 processes */ if( size < 2 ) { return tree; } /* * Shift all ranks by root, so that the algorithm can be * designed as if root would be always 0 * shiftedrank should be used in calculating distances * and position in tree */ shiftedrank = rank - root; if( shiftedrank < 0 ) { shiftedrank += size; } /* calculate my level */ level = calculate_level( fanout, shiftedrank ); delta = pown( fanout, level ); /* find my children */ for( i = 0; i < fanout; i++ ) { schild = shiftedrank + delta * (i+1); if( schild < size ) { tree->tree_next[i] = (schild+root)%size; tree->tree_nextsize = tree->tree_nextsize + 1; } else { break; } } /* find my parent */ slimit = calculate_num_nodes_up_to_level( fanout, level ); sparent = shiftedrank; if( sparent < fanout ) { sparent = 0; } else { while( sparent >= slimit ) { sparent -= delta/fanout; } } tree->tree_prev = (sparent+root)%size; return tree; } /* * Constructs in-order binary tree which can be used for non-commutative reduce * operations. * Root of this tree is always rank (size-1) and fanout is 2. * Here are some of the examples of this tree: * size == 2 size == 3 size == 4 size == 9 * 1 2 3 8 * / / \ / \ / \ * 0 1 0 2 1 7 3 * / / \ / \ * 0 6 5 2 1 * / / * 4 0 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bintree( MPI_Comm comm ) { int rank, size; int myrank, rightsize, delta; int parent, lchild, rchild; ompi_coll_tree_t* tree; /* * Get size and rank of the process in this communicator */ size = comm->size(); rank = comm->rank(); tree = new ompi_coll_tree_t; if (not tree) { XBT_DEBUG("coll:tuned:topo_build_tree PANIC::out of memory"); return NULL; } tree->tree_root = MPI_UNDEFINED; tree->tree_nextsize = MPI_UNDEFINED; /* * Initialize tree */ tree->tree_fanout = 2; tree->tree_bmtree = 0; tree->tree_root = size - 1; tree->tree_prev = -1; tree->tree_nextsize = 0; tree->tree_next[0] = -1; tree->tree_next[1] = -1; XBT_DEBUG( "coll:tuned:topo_build_in_order_tree Building fo %d rt %d", tree->tree_fanout, tree->tree_root); /* * Build the tree */ myrank = rank; parent = size - 1; delta = 0; while ( 1 ) { /* Compute the size of the right subtree */ rightsize = size >> 1; /* Determine the left and right child of this parent */ lchild = -1; rchild = -1; if (size - 1 > 0) { lchild = parent - 1; if (lchild > 0) { rchild = rightsize - 1; } } /* The following cases are possible: myrank can be - a parent, - belong to the left subtree, or - belong to the right subtee Each of the cases need to be handled differently. */ if (myrank == parent) { /* I am the parent: - compute real ranks of my children, and exit the loop. */ if (lchild >= 0) tree->tree_next[0] = lchild + delta; if (rchild >= 0) tree->tree_next[1] = rchild + delta; break; } if (myrank > rchild) { /* I belong to the left subtree: - If I am the left child, compute real rank of my parent - Iterate down through tree: compute new size, shift ranks down, and update delta. */ if (myrank == lchild) { tree->tree_prev = parent + delta; } size = size - rightsize - 1; delta = delta + rightsize; myrank = myrank - rightsize; parent = size - 1; } else { /* I belong to the right subtree: - If I am the right child, compute real rank of my parent - Iterate down through tree: compute new size and parent, but the delta and rank do not need to change. */ if (myrank == rchild) { tree->tree_prev = parent + delta; } size = rightsize; parent = rchild; } } if (tree->tree_next[0] >= 0) { tree->tree_nextsize = 1; } if (tree->tree_next[1] >= 0) { tree->tree_nextsize += 1; } return tree; } int ompi_coll_tuned_topo_destroy_tree( ompi_coll_tree_t** tree ) { ompi_coll_tree_t *ptr; if ((tree == nullptr) || (*tree == nullptr)) { return MPI_SUCCESS; } ptr = *tree; delete ptr; *tree = NULL; /* mark tree as gone */ return MPI_SUCCESS; } /* * * Here are some of the examples of this tree: * size == 2 size = 4 size = 8 * 0 0 0 * / | \ / | \ * 1 2 1 4 2 1 * | | |\ * 3 6 5 3 * | * 7 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_bmtree( MPI_Comm comm, int root ) { int childs = 0; int rank; int size; int mask = 1; int index; int remote; ompi_coll_tree_t *bmtree; int i; XBT_DEBUG("coll:tuned:topo:build_bmtree rt %d", root); /* * Get size and rank of the process in this communicator */ size = comm->size(); rank = comm->rank(); index = rank -root; bmtree = new ompi_coll_tree_t; if (not bmtree) { XBT_DEBUG("coll:tuned:topo:build_bmtree PANIC out of memory"); return NULL; } bmtree->tree_bmtree = 1; bmtree->tree_root = MPI_UNDEFINED; bmtree->tree_nextsize = MPI_UNDEFINED; for(i=0;itree_next[i] = -1; } if( index < 0 ) index += size; while( mask <= index ) mask <<= 1; /* Now I can compute my father rank */ if( root == rank ) { bmtree->tree_prev = root; } else { remote = (index ^ (mask >> 1)) + root; if( remote >= size ) remote -= size; bmtree->tree_prev = remote; } /* And now let's fill my childs */ while( mask < size ) { remote = (index ^ mask); if( remote >= size ) break; remote += root; if( remote >= size ) remote -= size; if (childs==MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo:build_bmtree max fanout incorrect %d needed %d", MAXTREEFANOUT, childs); return NULL; } bmtree->tree_next[childs] = remote; mask <<= 1; childs++; } bmtree->tree_nextsize = childs; bmtree->tree_root = root; return bmtree; } /* * Constructs in-order binomial tree which can be used for gather/scatter * operations. * * Here are some of the examples of this tree: * size == 2 size = 4 size = 8 * 0 0 0 * / / | / | \ * 1 1 2 1 2 4 * | | | \ * 3 3 5 6 * | * 7 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bmtree(MPI_Comm comm, int root) { int childs = 0; int rank, vrank; int size; int mask = 1; int remote; ompi_coll_tree_t *bmtree; int i; XBT_DEBUG("coll:tuned:topo:build_in_order_bmtree rt %d", root); /* * Get size and rank of the process in this communicator */ size = comm->size(); rank = comm->rank(); vrank = (rank - root + size) % size; bmtree = new ompi_coll_tree_t; if (not bmtree) { XBT_DEBUG("coll:tuned:topo:build_bmtree PANIC out of memory"); return NULL; } bmtree->tree_bmtree = 1; bmtree->tree_root = MPI_UNDEFINED; bmtree->tree_nextsize = MPI_UNDEFINED; for(i=0;itree_next[i] = -1; } if (root == rank) { bmtree->tree_prev = root; } while (mask < size) { remote = vrank ^ mask; if (remote < vrank) { bmtree->tree_prev = (remote + root) % size; break; } else if (remote < size) { bmtree->tree_next[childs] = (remote + root) % size; childs++; if (childs == MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo:build_bmtree max fanout incorrect %d needed %d", MAXTREEFANOUT, childs); return NULL; } } mask <<= 1; } bmtree->tree_nextsize = childs; bmtree->tree_root = root; return bmtree; } ompi_coll_tree_t* ompi_coll_tuned_topo_build_chain( int fanout, MPI_Comm comm, int root ) { int rank, size; int srank; /* shifted rank */ int i,maxchainlen; int mark,head,len; ompi_coll_tree_t *chain; XBT_DEBUG("coll:tuned:topo:build_chain fo %d rt %d", fanout, root); /* * Get size and rank of the process in this communicator */ size = comm->size(); rank = comm->rank(); if( fanout < 1 ) { XBT_DEBUG("coll:tuned:topo:build_chain WARNING invalid fanout of ZERO, forcing to 1 (pipeline)!"); fanout = 1; } if (fanout>MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo:build_chain WARNING invalid fanout %d bigger than max %d, forcing to max!", fanout, MAXTREEFANOUT); fanout = MAXTREEFANOUT; } /* * Allocate space for topology arrays if needed */ chain = new ompi_coll_tree_t; if (not chain) { XBT_DEBUG("coll:tuned:topo:build_chain PANIC out of memory"); fflush(stdout); return NULL; } chain->tree_root = MPI_UNDEFINED; chain->tree_nextsize = -1; for(i=0;itree_next[i] = -1; /* * Set root & numchain */ chain->tree_root = root; if( (size - 1) < fanout ) { chain->tree_nextsize = size-1; fanout = size-1; } else { chain->tree_nextsize = fanout; } /* * Shift ranks */ srank = rank - root; if (srank < 0) srank += size; /* * Special case - fanout == 1 */ if( fanout == 1 ) { if( srank == 0 ) chain->tree_prev = -1; else chain->tree_prev = (srank-1+root)%size; if( (srank + 1) >= size) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } else { chain->tree_next[0] = (srank+1+root)%size; chain->tree_nextsize = 1; } return chain; } /* Let's handle the case where there is just one node in the communicator */ if( size == 1 ) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; chain->tree_prev = -1; return chain; } /* * Calculate maximum chain length */ maxchainlen = (size-1) / fanout; if( (size-1) % fanout != 0 ) { maxchainlen++; mark = (size-1)%fanout; } else { mark = fanout+1; } /* * Find your own place in the list of shifted ranks */ if( srank != 0 ) { int column; if( srank-1 < (mark * maxchainlen) ) { column = (srank-1)/maxchainlen; head = 1+column*maxchainlen; len = maxchainlen; } else { column = mark + (srank-1-mark*maxchainlen)/(maxchainlen-1); head = mark*maxchainlen+1+(column-mark)*(maxchainlen-1); len = maxchainlen-1; } if( srank == head ) { chain->tree_prev = 0; /*root*/ } else { chain->tree_prev = srank-1; /* rank -1 */ } if( srank == (head + len - 1) ) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } else { if( (srank + 1) < size ) { chain->tree_next[0] = srank+1; chain->tree_nextsize = 1; } else { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } } } /* * Unshift values */ if( rank == root ) { chain->tree_prev = -1; chain->tree_next[0] = (root+1)%size; for( i = 1; i < fanout; i++ ) { chain->tree_next[i] = chain->tree_next[i-1] + maxchainlen; if( i > mark ) { chain->tree_next[i]--; } chain->tree_next[i] %= size; } chain->tree_nextsize = fanout; } else { chain->tree_prev = (chain->tree_prev+root)%size; if( chain->tree_next[0] != -1 ) { chain->tree_next[0] = (chain->tree_next[0]+root)%size; } } return chain; } int ompi_coll_tuned_topo_dump_tree (ompi_coll_tree_t* tree, int rank) { int i; XBT_DEBUG("coll:tuned:topo:topo_dump_tree %1d tree root %d" " fanout %d BM %1d nextsize %d prev %d", rank, tree->tree_root, tree->tree_bmtree, tree->tree_fanout, tree->tree_nextsize, tree->tree_prev); if( tree->tree_nextsize ) { for( i = 0; i < tree->tree_nextsize; i++ ) XBT_DEBUG("[%1d] %d", i, tree->tree_next[i]); } return (0); } SimGrid-3.18/src/smpi/colls/bcast/0000755000175000017500000000000013217757320017265 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/bcast/bcast-scatter-rdb-allgather.cpp0000644000175000017500000003014013217757320025234 0ustar mquinsonmquinson/* Copyright (c) 2011-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include "smpi_status.hpp" namespace simgrid{ namespace smpi{ static int scatter_for_bcast( int root, MPI_Comm comm, int nbytes, void *tmp_buf) { MPI_Status status; int rank, comm_size, src, dst; int relative_rank, mask; int mpi_errno = MPI_SUCCESS; int scatter_size, curr_size, recv_size = 0, send_size; comm_size = comm->size(); rank = comm->rank(); relative_rank = (rank >= root) ? rank - root : rank - root + comm_size; /* use long message algorithm: binomial tree scatter followed by an allgather */ /* The scatter algorithm divides the buffer into nprocs pieces and scatters them among the processes. Root gets the first piece, root+1 gets the second piece, and so forth. Uses the same binomial tree algorithm as above. Ceiling division is used to compute the size of each piece. This means some processes may not get any data. For example if bufsize = 97 and nprocs = 16, ranks 15 and 16 will get 0 data. On each process, the scattered data is stored at the same offset in the buffer as it is on the root process. */ scatter_size = (nbytes + comm_size - 1)/comm_size; /* ceiling division */ curr_size = (rank == root) ? nbytes : 0; /* root starts with all the data */ mask = 0x1; while (mask < comm_size) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += comm_size; recv_size = nbytes - relative_rank*scatter_size; /* recv_size is larger than what might actually be sent by the sender. We don't need compute the exact value because MPI allows you to post a larger recv.*/ if (recv_size <= 0) { curr_size = 0; /* this process doesn't receive any data because of uneven division */ } else { Request::recv(((char *)tmp_buf + relative_rank*scatter_size), recv_size, MPI_BYTE, src, COLL_TAG_BCAST, comm, &status); /* query actual size of data received */ curr_size=Status::get_count(&status, MPI_BYTE); } break; } mask <<= 1; } /* This process is responsible for all processes that have bits set from the LSB upto (but not including) mask. Because of the "not including", we start by shifting mask back down one. */ mask >>= 1; while (mask > 0) { if (relative_rank + mask < comm_size) { send_size = curr_size - scatter_size * mask; /* mask is also the size of this process's subtree */ if (send_size > 0) { dst = rank + mask; if (dst >= comm_size) dst -= comm_size; Request::send(((char *)tmp_buf + scatter_size*(relative_rank+mask)), send_size, MPI_BYTE, dst, COLL_TAG_BCAST, comm); curr_size -= send_size; } } mask >>= 1; } return mpi_errno; } int Coll_bcast_scatter_rdb_allgather::bcast ( void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { MPI_Status status; int rank, comm_size, dst; int relative_rank, mask; int mpi_errno = MPI_SUCCESS; int scatter_size, curr_size, recv_size = 0; int j, k, i, tmp_mask, is_contig, is_homogeneous; MPI_Aint type_size = 0, nbytes = 0; int relative_dst, dst_tree_root, my_tree_root, send_offset; int recv_offset, tree_root, nprocs_completed, offset; int position; MPI_Aint true_extent, true_lb; void *tmp_buf; comm_size = comm->size(); rank = comm->rank(); relative_rank = (rank >= root) ? rank - root : rank - root + comm_size; /* If there is only one process, return */ if (comm_size == 1) goto fn_exit; //if (HANDLE_GET_KIND(datatype) == HANDLE_KIND_BUILTIN) if(datatype->flags() & DT_FLAG_CONTIGUOUS) is_contig = 1; else { is_contig = 0; } is_homogeneous = 1; /* MPI_Type_size() might not give the accurate size of the packed * datatype for heterogeneous systems (because of padding, encoding, * etc). On the other hand, MPI_Pack_size() can become very * expensive, depending on the implementation, especially for * heterogeneous systems. We want to use MPI_Type_size() wherever * possible, and MPI_Pack_size() in other places. */ if (is_homogeneous) type_size=datatype->size(); nbytes = type_size * count; if (nbytes == 0) goto fn_exit; /* nothing to do */ if (is_contig && is_homogeneous) { /* contiguous and homogeneous. no need to pack. */ datatype->extent(&true_lb, &true_extent); tmp_buf = (char *) buffer + true_lb; } else { tmp_buf=(void*)xbt_malloc(nbytes); /* TODO: Pipeline the packing and communication */ position = 0; if (rank == root) { mpi_errno = datatype->pack(buffer, count, tmp_buf, nbytes, &position, comm); if (mpi_errno) xbt_die("crash while packing %d", mpi_errno); } } scatter_size = (nbytes + comm_size - 1)/comm_size; /* ceiling division */ mpi_errno = scatter_for_bcast(root, comm, nbytes, tmp_buf); if (mpi_errno) { xbt_die("crash while scattering %d", mpi_errno); } /* curr_size is the amount of data that this process now has stored in * buffer at byte offset (relative_rank*scatter_size) */ curr_size = scatter_size < (nbytes - (relative_rank * scatter_size)) ? scatter_size : (nbytes - (relative_rank * scatter_size)) ; if (curr_size < 0) curr_size = 0; /* medium size allgather and pof2 comm_size. use recurive doubling. */ mask = 0x1; i = 0; while (mask < comm_size) { relative_dst = relative_rank ^ mask; dst = (relative_dst + root) % comm_size; /* find offset into send and recv buffers. zero out the least significant "i" bits of relative_rank and relative_dst to find root of src and dst subtrees. Use ranks of roots as index to send from and recv into buffer */ dst_tree_root = relative_dst >> i; dst_tree_root <<= i; my_tree_root = relative_rank >> i; my_tree_root <<= i; send_offset = my_tree_root * scatter_size; recv_offset = dst_tree_root * scatter_size; if (relative_dst < comm_size) { Request::sendrecv(((char *)tmp_buf + send_offset), curr_size, MPI_BYTE, dst, COLL_TAG_BCAST, ((char *)tmp_buf + recv_offset), (nbytes-recv_offset < 0 ? 0 : nbytes-recv_offset), MPI_BYTE, dst, COLL_TAG_BCAST, comm, &status); recv_size=Status::get_count(&status, MPI_BYTE); curr_size += recv_size; } /* if some processes in this process's subtree in this step did not have any destination process to communicate with because of non-power-of-two, we need to send them the data that they would normally have received from those processes. That is, the haves in this subtree must send to the havenots. We use a logarithmic recursive-halfing algorithm for this. */ /* This part of the code will not currently be executed because we are not using recursive doubling for non power of two. Mark it as experimental so that it doesn't show up as red in the coverage tests. */ /* --BEGIN EXPERIMENTAL-- */ if (dst_tree_root + mask > comm_size) { nprocs_completed = comm_size - my_tree_root - mask; /* nprocs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; offset = scatter_size * (my_tree_root + mask); tmp_mask = mask >> 1; while (tmp_mask) { relative_dst = relative_rank ^ tmp_mask; dst = (relative_dst + root) % comm_size; tree_root = relative_rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. */ /* if (rank == 3) { printf("rank %d, dst %d, root %d, nprocs_completed %d\n", relative_rank, relative_dst, tree_root, nprocs_completed); fflush(stdout); }*/ if ((relative_dst > relative_rank) && (relative_rank < tree_root + nprocs_completed) && (relative_dst >= tree_root + nprocs_completed)) { /* printf("Rank %d, send to %d, offset %d, size %d\n", rank, dst, offset, recv_size); fflush(stdout); */ Request::send(((char *)tmp_buf + offset), recv_size, MPI_BYTE, dst, COLL_TAG_BCAST, comm); /* recv_size was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((relative_dst < relative_rank) && (relative_dst < tree_root + nprocs_completed) && (relative_rank >= tree_root + nprocs_completed)) { /* printf("Rank %d waiting to recv from rank %d\n", relative_rank, dst); */ Request::recv(((char *)tmp_buf + offset), nbytes - offset, MPI_BYTE, dst, COLL_TAG_BCAST, comm, &status); /* nprocs_completed is also equal to the no. of processes whose data we don't have */ recv_size=Status::get_count(&status, MPI_BYTE); curr_size += recv_size; /* printf("Rank %d, recv from %d, offset %d, size %d\n", rank, dst, offset, recv_size); fflush(stdout);*/ } tmp_mask >>= 1; k--; } } /* --END EXPERIMENTAL-- */ mask <<= 1; i++; } /* check that we received as much as we expected */ /* recvd_size may not be accurate for packed heterogeneous data */ if (is_homogeneous && curr_size != nbytes) { xbt_die("we didn't receive enough !"); } if (not is_contig || not is_homogeneous) { if (rank != root) { position = 0; mpi_errno = MPI_Unpack(tmp_buf, nbytes, &position, buffer, count, datatype, comm); if (mpi_errno) xbt_die("error when unpacking %d", mpi_errno); } } fn_exit: /* xbt_free(tmp_buf);*/ return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-binomial-tree.cpp0000644000175000017500000001015013217757320023607 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: bcast_binomial_tree * Return: int * Inputs: buff: send input buffer count: number of elements to send data_type: data type of elements being sent root: source of data comm: communicator * Descrp: broadcasts using a bionomial tree. * Auther: MPIH / modified by Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_bcast_binomial_tree::bcast(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { int src, dst, rank, num_procs, mask, relative_rank; int tag = COLL_TAG_BCAST; rank = comm->rank(); num_procs = comm->size(); relative_rank = (rank >= root) ? rank - root : rank - root + num_procs; mask = 0x1; while (mask < num_procs) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += num_procs; Request::recv(buff, count, data_type, src, tag, comm, MPI_STATUS_IGNORE); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (relative_rank + mask < num_procs) { dst = rank + mask; if (dst >= num_procs) dst -= num_procs; Request::send(buff, count, data_type, dst, tag, comm); } mask >>= 1; } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-ompi-split-bintree.cpp0000644000175000017500000003107713217757320024616 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * - Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * - Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer listed * in this license in the documentation and/or other materials * provided with the distribution. * - Neither the name of the copyright holders nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * The copyright holders provide no reassurances that the source code * provided does not infringe any patent, copyright, or any other * intellectual property rights of third parties. The copyright holders * disclaim any liability to any recipient for claims brought against * recipient by any third party for infringement of that parties * intellectual property rights. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" #define MAXTREEFANOUT 32 namespace simgrid{ namespace smpi{ int Coll_bcast_ompi_split_bintree::bcast ( void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { unsigned int segsize ; int rank, size; int segindex, i, lr, pair; int segcount[2]; /* Number ompi_request_wait_allof elements sent with each segment */ uint32_t counts[2]; int num_segments[2]; /* Number of segmenets */ int sendcount[2]; /* the same like segcount, except for the last segment */ size_t realsegsize[2]; char *tmpbuf[2]; size_t type_size; ptrdiff_t type_extent; MPI_Request base_req, new_req; ompi_coll_tree_t *tree; // mca_coll_tuned_module_t *tuned_module = (mca_coll_tuned_module_t*) module; // mca_coll_tuned_comm_t *data = tuned_module->tuned_data; size = comm->size(); rank = comm->rank(); //compute again segsize const size_t intermediate_message_size = 370728; size_t message_size = datatype->size() * (unsigned long)count; if(message_size < intermediate_message_size) segsize = 1024 ; else segsize = 1024 << 3; XBT_DEBUG("ompi_coll_tuned_bcast_intra_split_bintree rank %d root %d ss %5u", rank, root, segsize); if (size == 1) { return MPI_SUCCESS; } /* setup the binary tree topology. */ tree = ompi_coll_tuned_topo_build_tree(2,comm,root); type_size = datatype->size(); /* Determine number of segments and number of elements per segment */ counts[0] = count/2; if (count % 2 != 0) counts[0]++; counts[1] = count - counts[0]; if ( segsize > 0 ) { /* Note that ompi_datatype_type_size() will never return a negative value in typelng; it returns an int [vs. an unsigned type] because of the MPI spec. */ if (segsize < ((uint32_t)type_size)) { segsize = type_size; /* push segsize up to hold one type */ } segcount[0] = segcount[1] = segsize / type_size; num_segments[0] = counts[0]/segcount[0]; if ((counts[0] % segcount[0]) != 0) num_segments[0]++; num_segments[1] = counts[1]/segcount[1]; if ((counts[1] % segcount[1]) != 0) num_segments[1]++; } else { segcount[0] = counts[0]; segcount[1] = counts[1]; num_segments[0] = num_segments[1] = 1; } /* if the message is too small to be split into segments */ if( (counts[0] == 0 || counts[1] == 0) || (segsize > counts[0] * type_size) || (segsize > counts[1] * type_size) ) { /* call linear version here ! */ return (Coll_bcast_SMP_linear::bcast ( buffer, count, datatype, root, comm)); } type_extent = datatype->get_extent(); /* Determine real segment size */ realsegsize[0] = segcount[0] * type_extent; realsegsize[1] = segcount[1] * type_extent; /* set the buffer pointers */ tmpbuf[0] = (char *) buffer; tmpbuf[1] = (char *) buffer+counts[0] * type_extent; /* Step 1: Root splits the buffer in 2 and sends segmented message down the branches. Left subtree of the tree receives first half of the buffer, while right subtree receives the remaining message. */ /* determine if I am left (0) or right (1), (root is right) */ lr = ((rank + size - root)%size + 1)%2; /* root code */ if( rank == root ) { /* determine segment count */ sendcount[0] = segcount[0]; sendcount[1] = segcount[1]; /* for each segment */ for (segindex = 0; segindex < num_segments[0]; segindex++) { /* for each child */ for( i = 0; i < tree->tree_nextsize && i < 2; i++ ) { if (segindex >= num_segments[i]) { /* no more segments */ continue; } /* determine how many elements are being sent in this round */ if(segindex == (num_segments[i] - 1)) sendcount[i] = counts[i] - segindex*segcount[i]; /* send data */ Request::send(tmpbuf[i], sendcount[i], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); /* update tmp buffer */ tmpbuf[i] += realsegsize[i]; } } } /* intermediate nodes code */ else if( tree->tree_nextsize > 0 ) { /* Intermediate nodes: * It will receive segments only from one half of the data. * Which one is determined by whether the node belongs to the "left" or "right" * subtree. Topoloby building function builds binary tree such that * odd "shifted ranks" ((rank + size - root)%size) are on the left subtree, * and even on the right subtree. * * Create the pipeline. We first post the first receive, then in the loop we * post the next receive and after that wait for the previous receive to complete * and we disseminating the data to all children. */ sendcount[lr] = segcount[lr]; base_req=Request::irecv(tmpbuf[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments[lr]; segindex++ ) { /* determine how many elements to expect in this round */ if( segindex == (num_segments[lr] - 1)) sendcount[lr] = counts[lr] - segindex*segcount[lr]; /* post new irecv */ new_req = Request::irecv( tmpbuf[lr] + realsegsize[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait for and forward current segment */ Request::waitall( 1, &base_req, MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { /* send data to children (segcount[lr]) */ Request::send( tmpbuf[lr], segcount[lr], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* end of for each child */ /* upate the base request */ base_req = new_req; /* go to the next buffer (ie. the one corresponding to the next recv) */ tmpbuf[lr] += realsegsize[lr]; } /* end of for segindex */ /* wait for the last segment and forward current segment */ Request::waitall( 1, &base_req, MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { /* send data to children */ Request::send(tmpbuf[lr], sendcount[lr], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* end of for each child */ } /* leaf nodes */ else { /* Just consume segments as fast as possible */ sendcount[lr] = segcount[lr]; for (segindex = 0; segindex < num_segments[lr]; segindex++) { /* determine how many elements to expect in this round */ if (segindex == (num_segments[lr] - 1)) sendcount[lr] = counts[lr] - segindex*segcount[lr]; /* receive segments */ Request::recv(tmpbuf[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); /* update the initial pointer to the buffer */ tmpbuf[lr] += realsegsize[lr]; } } /* reset the buffer pointers */ tmpbuf[0] = (char *) buffer; tmpbuf[1] = (char *) buffer+counts[0] * type_extent; /* Step 2: Find your immediate pair (identical node in opposite subtree) and SendRecv data buffer with them. The tree building function ensures that if (we are not root) if we are in the left subtree (lr == 0) our pair is (rank+1)%size. if we are in the right subtree (lr == 1) our pair is (rank-1)%size If we have even number of nodes the rank (size-1) will pair up with root. */ if (lr == 0) { pair = (rank+1)%size; } else { pair = (rank+size-1)%size; } if ( (size%2) != 0 && rank != root) { Request::sendrecv( tmpbuf[lr], counts[lr], datatype, pair, COLL_TAG_BCAST, tmpbuf[(lr+1)%2], counts[(lr+1)%2], datatype, pair, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } else if ( (size%2) == 0 ) { /* root sends right buffer to the last node */ if( rank == root ) { Request::send(tmpbuf[1], counts[1], datatype, (root+size-1)%size, COLL_TAG_BCAST, comm); } /* last node receives right buffer from the root */ else if (rank == (root+size-1)%size) { Request::recv(tmpbuf[1], counts[1], datatype, root, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } /* everyone else exchanges buffers */ else { Request::sendrecv( tmpbuf[lr], counts[lr], datatype, pair, COLL_TAG_BCAST, tmpbuf[(lr+1)%2], counts[(lr+1)%2], datatype, pair, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } } ompi_coll_tuned_topo_destroy_tree(&tree); return (MPI_SUCCESS); } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-flattree.cpp0000644000175000017500000000224513217757320022674 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_bcast_flattree::bcast(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { MPI_Request *req_ptr; MPI_Request *reqs; int i, rank, num_procs; int tag = COLL_TAG_BCAST; rank = comm->rank(); num_procs = comm->size(); if (rank != root) { Request::recv(buff, count, data_type, root, tag, comm, MPI_STATUS_IGNORE); } else { reqs = (MPI_Request *) xbt_malloc((num_procs - 1) * sizeof(MPI_Request)); req_ptr = reqs; // Root sends data to all others for (i = 0; i < num_procs; i++) { if (i == rank) continue; *(req_ptr++) = Request::isend(buff, count, data_type, i, tag, comm); } // wait on all requests Request::waitall(num_procs - 1, reqs, MPI_STATUSES_IGNORE); free(reqs); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-flattree-pipeline.cpp0000644000175000017500000000361313217757320024477 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" int flattree_segment_in_byte = 8192; namespace simgrid{ namespace smpi{ int Coll_bcast_flattree_pipeline::bcast(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { int i, j, rank, num_procs; int tag = COLL_TAG_BCAST; MPI_Aint extent; extent = data_type->get_extent(); int segment = flattree_segment_in_byte / extent; segment = segment == 0 ? 1 :segment; int pipe_length = count / segment; int increment = segment * extent; if (pipe_length==0) { XBT_WARN("MPI_bcast_flattree_pipeline use default MPI_bcast_flattree."); return Coll_bcast_flattree::bcast(buff, count, data_type, root, comm); } rank = comm->rank(); num_procs = comm->size(); MPI_Request *request_array; MPI_Status *status_array; request_array = (MPI_Request *) xbt_malloc(pipe_length * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc(pipe_length * sizeof(MPI_Status)); if (rank != root) { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *)buff + (i * increment), segment, data_type, root, tag, comm); } Request::waitall(pipe_length, request_array, status_array); } else { // Root sends data to all others for (j = 0; j < num_procs; j++) { if (j == rank) continue; else { for (i = 0; i < pipe_length; i++) { Request::send((char *)buff + (i * increment), segment, data_type, j, tag, comm); } } } } free(request_array); free(status_array); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-arrival-scatter.cpp0000644000175000017500000001550213217757320024171 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #ifndef BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE #define BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE 128 #endif #ifndef BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE #define BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE 128 #endif namespace simgrid{ namespace smpi{ /* Non-topology-specific pipelined linear-bcast function */ int Coll_bcast_arrival_scatter::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = -COLL_TAG_BCAST;//in order to use ANY_TAG, make this one positive int header_tag = 10; MPI_Status status; int curr_remainder; int curr_size; int curr_increment; int send_offset; int recv_offset; int send_count; int recv_count; MPI_Status temp_status_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int rank, size; int i, k; int sent_count; int header_index; int flag_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int already_sent[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int header_buf[BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE]; char temp_buf[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int will_send[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int max_node = BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; int header_size = BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE; MPI_Aint extent; extent = datatype->get_extent(); /* source and destination */ int to, from; rank = comm->rank(); size = comm->size(); /* message too small */ if (count < size) { XBT_WARN("MPI_bcast_arrival_scatter use default MPI_bcast."); Colls::bcast(buf, count, datatype, root, comm); return MPI_SUCCESS; } /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag - 1, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag - 1, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < max_node; i++) { already_sent[i] = 0; } /* start bcast */ /* root */ if (rank == 0) { for (i = 0; i < max_node; i++) will_send[i] = 0; sent_count = 0; while (sent_count < (size - 1)) { for (k = 0; k < 3; k++) { for (i = 1; i < size; i++) { if ((already_sent[i] == 0) && (will_send[i] == 0)) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; Request::recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); i = 0; } } } } header_index = 0; /* recv 1-byte message in this round */ for (i = 1; i < size; i++) { /* message arrive */ if ((will_send[i] == 1) && (already_sent[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* if (header_index != 0) { printf("header index = %d node = ",header_index); for (i=0;iget_extent(); rank = comm->rank(); size = comm->size(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); }else{ //implementation buggy in this case return Coll_bcast_mpich::bcast( buf , count, datatype, root, comm); } int segment = bcast_SMP_linear_segment_byte / extent; segment = segment == 0 ? 1 :segment; int pipe_length = count / segment; int remainder = count % segment; int increment = segment * extent; /* leader of each SMP do inter-communication and act as a root for intra-communication */ int to_inter = (rank + num_core) % size; int to_intra = (rank + 1) % size; int from_inter = (rank - num_core + size) % size; int from_intra = (rank + size - 1) % size; // call native when MPI communication size is too small if (size <= num_core) { XBT_WARN("MPI_bcast_SMP_linear use default MPI_bcast."); Coll_bcast_default::bcast(buf, count, datatype, root, comm); return MPI_SUCCESS; } // if root is not zero send to rank zero first if (root != 0) { if (rank == root) Request::send(buf, count, datatype, 0, tag, comm); else if (rank == 0) Request::recv(buf, count, datatype, root, tag, comm, &status); } // when a message is smaller than a block size => no pipeline if (count <= segment) { // case ROOT if (rank == 0) { Request::send(buf, count, datatype, to_inter, tag, comm); Request::send(buf, count, datatype, to_intra, tag, comm); } // case last ROOT of each SMP else if (rank == (((size - 1) / num_core) * num_core)) { request = Request::irecv(buf, count, datatype, from_inter, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to_intra, tag, comm); } // case intermediate ROOT of each SMP else if (rank % num_core == 0) { request = Request::irecv(buf, count, datatype, from_inter, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to_inter, tag, comm); Request::send(buf, count, datatype, to_intra, tag, comm); } // case last non-ROOT of each SMP else if (((rank + 1) % num_core == 0) || (rank == (size - 1))) { request = Request::irecv(buf, count, datatype, from_intra, tag, comm); Request::wait(&request, &status); } // case intermediate non-ROOT of each SMP else { request = Request::irecv(buf, count, datatype, from_intra, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to_intra, tag, comm); } return MPI_SUCCESS; } // pipeline bcast else { request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); // case ROOT of each SMP if (rank % num_core == 0) { // case real root if (rank == 0) { for (i = 0; i < pipe_length; i++) { Request::send((char *) buf + (i * increment), segment, datatype, to_inter, (tag + i), comm); Request::send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } // case last ROOT of each SMP else if (rank == (((size - 1) / num_core) * num_core)) { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); Request::send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } // case intermediate ROOT of each SMP else { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); Request::send((char *) buf + (i * increment), segment, datatype, to_inter, (tag + i), comm); Request::send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } } else { // case last non-ROOT of each SMP if (((rank + 1) % num_core == 0) || (rank == (size - 1))) { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); } } // case intermediate non-ROOT of each SMP else { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); Request::send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } } free(request_array); free(status_array); } // when count is not divisible by block size, use default BCAST for the remainder if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_SMP_linear use default MPI_bcast."); Colls::bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-NTSL-Isend.cpp0000644000175000017500000001120213217757320022677 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" static int bcast_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ namespace simgrid{ namespace smpi{ int Coll_bcast_NTSL_Isend::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = datatype->get_extent(); rank = comm->rank(); size = comm->size(); /* source node and destination nodes (same through out the functions) */ int to = (rank + 1) % size; int from = (rank + size - 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSL_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { Request::send(buf, count, datatype, to, tag, comm); } else if (rank == (size - 1)) { request = Request::irecv(buf, count, datatype, from, tag, comm); Request::wait(&request, &status); } else { request = Request::irecv(buf, count, datatype, from, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to, tag, comm); } return MPI_SUCCESS; } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root send data */ if (rank == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* last node only receive data */ else if (rank == (size - 1)) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } Request::waitall((pipe_length), recv_request_array, recv_status_array); } /* intermediate nodes relay (receive, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_NTSL_Isend_nb use default MPI_bcast."); Colls::bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-SMP-binomial.cpp0000644000175000017500000000650013217757320023313 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_bcast_SMP_binomial::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mask = 1; int size; int rank; MPI_Status status; int tag = COLL_TAG_BCAST; size = comm->size(); rank = comm->rank(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); }else{ //implementation buggy in this case return Coll_bcast_mpich::bcast( buf , count, datatype, root, comm); } int to_intra, to_inter; int from_intra, from_inter; int inter_rank = rank / num_core; int inter_size = (size - 1) / num_core + 1; int intra_rank = rank % num_core; int intra_size = num_core; if (((rank / num_core) * num_core) == ((size / num_core) * num_core)) intra_size = size - (rank / num_core) * num_core; // if root is not zero send to rank zero first if (root != 0) { if (rank == root) Request::send(buf, count, datatype, 0, tag, comm); else if (rank == 0) Request::recv(buf, count, datatype, root, tag, comm, &status); } //FIRST STEP node 0 send to every root-of-each-SMP with binomial tree //printf("node %d inter_rank = %d, inter_size = %d\n",rank,inter_rank, inter_size); if (intra_rank == 0) { mask = 1; while (mask < inter_size) { if (inter_rank & mask) { from_inter = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, from_inter, mask); Request::recv(buf, count, datatype, from_inter, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { if (inter_rank < inter_size) { to_inter = (inter_rank + mask) * num_core; if (to_inter < size) { //printf("Node %d send to node %d when mask is %d\n", rank, to_inter, mask); Request::send(buf, count, datatype, to_inter, tag, comm); } } mask >>= 1; } } // SECOND STEP every root-of-each-SMP send to all children with binomial tree // base is a rank of root-of-each-SMP int base = (rank / num_core) * num_core; mask = 1; while (mask < intra_size) { if (intra_rank & mask) { from_intra = base + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, from_inter, mask); Request::recv(buf, count, datatype, from_intra, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { if (intra_rank < intra_size) { to_intra = base + (intra_rank + mask); if (to_intra < size) { //printf("Node %d send to node %d when mask is %d\n", rank, to_inter, mask); Request::send(buf, count, datatype, to_intra, tag, comm); } } mask >>= 1; } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-ompi-pipeline.cpp0000644000175000017500000001726113217757320023641 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" #define MAXTREEFANOUT 32 namespace simgrid{ namespace smpi{ int Coll_bcast_ompi_pipeline::bcast( void* buffer, int original_count, MPI_Datatype datatype, int root, MPI_Comm comm) { int count_by_segment = original_count; size_t type_size; size_t segsize =1024 << 7; //mca_coll_tuned_module_t *tuned_module = (mca_coll_tuned_module_t*) module; //mca_coll_tuned_comm_t *data = tuned_module->tuned_data; // return ompi_coll_tuned_bcast_intra_generic( buffer, count, datatype, root, comm, module, // count_by_segment, data->cached_pipeline ); ompi_coll_tree_t * tree = ompi_coll_tuned_topo_build_chain( 1, comm, root ); int i; int rank, size; int segindex; int num_segments; /* Number of segments */ int sendcount; /* number of elements sent in this segment */ size_t realsegsize; char *tmpbuf; ptrdiff_t extent; MPI_Request recv_reqs[2] = {MPI_REQUEST_NULL, MPI_REQUEST_NULL}; MPI_Request *send_reqs = NULL; int req_index; /** * Determine number of elements sent per operation. */ type_size = datatype->size(); size = comm->size(); rank = comm->rank(); if(size==1)return MPI_SUCCESS; const double a_p16 = 3.2118e-6; /* [1 / byte] */ const double b_p16 = 8.7936; const double a_p64 = 2.3679e-6; /* [1 / byte] */ const double b_p64 = 1.1787; const double a_p128 = 1.6134e-6; /* [1 / byte] */ const double b_p128 = 2.1102; size_t message_size; /* else we need data size for decision function */ message_size = type_size * (unsigned long)original_count; /* needed for decision */ if (size < (a_p128 * message_size + b_p128)) { //Pipeline with 128KB segments segsize = 1024 << 7; }else if (size < (a_p64 * message_size + b_p64)) { // Pipeline with 64KB segments segsize = 1024 << 6; }else if (size < (a_p16 * message_size + b_p16)) { //Pipeline with 16KB segments segsize = 1024 << 4; } COLL_TUNED_COMPUTED_SEGCOUNT( segsize, type_size, count_by_segment ); XBT_DEBUG("coll:tuned:bcast_intra_pipeline rank %d ss %5zu type_size %lu count_by_segment %d", comm->rank(), segsize, (unsigned long)type_size, count_by_segment); extent = datatype->get_extent(); num_segments = (original_count + count_by_segment - 1) / count_by_segment; realsegsize = count_by_segment * extent; /* Set the buffer pointers */ tmpbuf = (char *) buffer; if( tree->tree_nextsize != 0 ) { send_reqs = new MPI_Request[tree->tree_nextsize]; } /* Root code */ if( rank == root ) { /* For each segment: - send segment to all children. The last segment may have less elements than other segments. */ sendcount = count_by_segment; for( segindex = 0; segindex < num_segments; segindex++ ) { if( segindex == (num_segments - 1) ) { sendcount = original_count - segindex * count_by_segment; } for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i] = Request::isend(tmpbuf, sendcount, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* complete the sends before starting the next sends */ Request::waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); /* update tmp buffer */ tmpbuf += realsegsize; } } /* Intermediate nodes code */ else if( tree->tree_nextsize > 0 ) { /* Create the pipeline. 1) Post the first receive 2) For segments 1 .. num_segments - post new receive - wait on the previous receive to complete - send this data to children 3) Wait on the last segment 4) Compute number of elements in last segment. 5) Send the last segment to children */ req_index = 0; recv_reqs[req_index]=Request::irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments; segindex++ ) { req_index = req_index ^ 0x1; /* post new irecv */ recv_reqs[req_index]= Request::irecv( tmpbuf + realsegsize, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait for and forward the previous segment to children */ Request::wait( &recv_reqs[req_index ^ 0x1], MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i]=Request::isend(tmpbuf, count_by_segment, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm ); } /* complete the sends before starting the next iteration */ Request::waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); /* Update the receive buffer */ tmpbuf += realsegsize; } /* Process the last segment */ Request::wait( &recv_reqs[req_index], MPI_STATUSES_IGNORE ); sendcount = original_count - (num_segments - 1) * count_by_segment; for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i] = Request::isend(tmpbuf, sendcount, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } Request::waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); } /* Leaf nodes */ else { /* Receive all segments from parent in a loop: 1) post irecv for the first segment 2) for segments 1 .. num_segments - post irecv for the next segment - wait on the previous segment to arrive 3) wait for the last segment */ req_index = 0; recv_reqs[req_index] = Request::irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments; segindex++ ) { req_index = req_index ^ 0x1; tmpbuf += realsegsize; /* post receive for the next segment */ recv_reqs[req_index] = Request::irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait on the previous segment */ Request::wait( &recv_reqs[req_index ^ 0x1], MPI_STATUS_IGNORE ); } Request::wait( &recv_reqs[req_index], MPI_STATUS_IGNORE ); } delete[] send_reqs; ompi_coll_tuned_topo_destroy_tree(&tree); return (MPI_SUCCESS); } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-arrival-pattern-aware.cpp0000644000175000017500000002626213217757320025303 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" static int bcast_NTSL_segment_size_in_byte = 8192; #define HEADER_SIZE 1024 #define MAX_NODE 1024 namespace simgrid{ namespace smpi{ /* Non-topology-specific pipelined linear-bcast function */ int Coll_bcast_arrival_pattern_aware::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = -COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[MAX_NODE]; int rank, size; int i, j; int sent_count; int header_index; int flag_array[MAX_NODE]; int already_sent[MAX_NODE]; int to_clean[MAX_NODE]; int header_buf[HEADER_SIZE]; char temp_buf[MAX_NODE]; MPI_Aint extent; extent = datatype->get_extent(); /* destination */ int to; rank = comm->rank(); size = comm->size(); /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSL_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < MAX_NODE; i++) { already_sent[i] = 0; to_clean[i] = 0; } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { sent_count = 0; while (sent_count < (size - 1)) { for (i = 1; i < size; i++) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], MPI_STATUSES_IGNORE); } header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((flag_array[i] == 1) && (already_sent[i] == 0)) { Request::recv(temp_buf, 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; Request::send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); Request::send(buf, count, datatype, to, tag, comm); } /* randomly MPI_Send to one */ else { /* search for the first node that never received data before */ for (i = 1; i < size; i++) { if (already_sent[i] == 0) { header_buf[0] = i; header_buf[1] = -1; Request::send(header_buf, HEADER_SIZE, MPI_INT, i, tag, comm); Request::send(buf, count, datatype, i, tag, comm); already_sent[i] = 1; sent_count++; break; } } } } /* while loop */ } /* non-root */ else { /* send 1-byte message to root */ Request::send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header and data, forward when required */ Request::recv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm, &status); Request::recv(buf, count, datatype, MPI_ANY_SOURCE, tag, comm, &status); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* send header followed by data */ if (header_buf[myordering + 1] != -1) { Request::send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); Request::send(buf, count, datatype, header_buf[myordering + 1], tag, comm); } } } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); if (rank == 0) { //double start2 = MPI_Wtime(); sent_count = 0; //int iteration = 0; while (sent_count < (size - 1)) { //iteration++; //start = MPI_Wtime(); for (i = 1; i < size; i++) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); } //total = MPI_Wtime() - start; //total *= 1000; //printf("Iprobe time = %.2f\n",total); header_index = 0; MPI_Wtime(); /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((flag_array[i] == 1) && (already_sent[i] == 0)) { Request::recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } //total = MPI_Wtime() - start; //total *= 1000; //printf("Recv 1-byte time = %.2f\n",total); /* if (header_index != 0) { printf("header index = %d node = ",header_index); for (i=0;i segment)) { XBT_WARN("MPI_bcast_arrival_pattern_aware use default MPI_bcast."); Colls::bcast((char *)buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-NTSL.cpp0000644000175000017500000001121713217757320021645 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" static int bcast_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ namespace simgrid{ namespace smpi{ int Coll_bcast_NTSL::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = datatype->get_extent(); rank = comm->rank(); size = comm->size(); /* source node and destination nodes (same through out the functions) */ int to = (rank + 1) % size; int from = (rank + size - 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = extent == 0 ? 1 : (bcast_NTSL_segment_size_in_byte / extent); segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { Request::send(buf, count, datatype, to, tag, comm); } else if (rank == (size - 1)) { request = Request::irecv(buf, count, datatype, from, tag, comm); Request::wait(&request, &status); } else { request = Request::irecv(buf, count, datatype, from, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to, tag, comm); } return MPI_SUCCESS; } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root send data */ if (rank == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* last node only receive data */ else if (rank == (size - 1)) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } Request::waitall((pipe_length), recv_request_array, recv_status_array); } /* intermediate nodes relay (receive, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_arrival_NTSL use default MPI_bcast."); Colls::bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-mvapich-smp.cpp0000644000175000017500000003341713217757320023317 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" extern int (*MV2_Bcast_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); extern int (*MV2_Bcast_intra_node_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); extern int zcpy_knomial_factor; extern int mv2_pipelined_zcpy_knomial_factor; extern int bcast_segment_size; extern int mv2_inter_node_knomial_factor; extern int mv2_intra_node_knomial_factor; extern int mv2_bcast_two_level_system_size; #define INTRA_NODE_ROOT 0 #define MPIR_Pipelined_Bcast_Zcpy_MV2 Coll_bcast_mpich::bcast #define MPIR_Pipelined_Bcast_MV2 Coll_bcast_mpich::bcast #define MPIR_Bcast_binomial_MV2 Coll_bcast_binomial_tree::bcast #define MPIR_Bcast_scatter_ring_allgather_shm_MV2 Coll_bcast_scatter_LR_allgather::bcast #define MPIR_Bcast_scatter_doubling_allgather_MV2 Coll_bcast_scatter_rdb_allgather::bcast #define MPIR_Bcast_scatter_ring_allgather_MV2 Coll_bcast_scatter_LR_allgather::bcast #define MPIR_Shmem_Bcast_MV2 Coll_bcast_mpich::bcast #define MPIR_Bcast_tune_inter_node_helper_MV2 Coll_bcast_mvapich2_inter_node::bcast #define MPIR_Bcast_inter_node_helper_MV2 Coll_bcast_mvapich2_inter_node::bcast #define MPIR_Knomial_Bcast_intra_node_MV2 Coll_bcast_mvapich2_knomial_intra_node::bcast #define MPIR_Bcast_intra_MV2 Coll_bcast_mvapich2_intra_node::bcast extern int zcpy_knomial_factor; extern int mv2_pipelined_zcpy_knomial_factor; extern int bcast_segment_size; extern int mv2_inter_node_knomial_factor; extern int mv2_intra_node_knomial_factor; #define mv2_bcast_two_level_system_size 64 #define mv2_bcast_short_msg 16384 #define mv2_bcast_large_msg 512*1024 #define mv2_knomial_intra_node_threshold 131072 #define mv2_scatter_rd_inter_leader_bcast 1 namespace simgrid{ namespace smpi{ int Coll_bcast_mvapich2_inter_node::bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int rank; int mpi_errno = MPI_SUCCESS; MPI_Comm shmem_comm, leader_comm; int local_rank, local_size, global_rank = -1; int leader_root, leader_of_root; rank = comm->rank(); //comm_size = comm->size(); if (MV2_Bcast_function==NULL){ MV2_Bcast_function=Coll_bcast_mpich::bcast; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= Coll_bcast_mpich::bcast; } if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); leader_comm = comm->get_leaders_comm(); if ((local_rank == 0) && (local_size > 1)) { global_rank = leader_comm->rank(); } int* leaders_map = comm->get_leaders_map(); leader_of_root = comm->group()->rank(leaders_map[root]); leader_root = leader_comm->group()->rank(leaders_map[root]); if (local_size > 1) { if ((local_rank == 0) && (root != rank) && (leader_root == global_rank)) { Request::recv(buffer, count, datatype, root, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } if ((local_rank != 0) && (root == rank)) { Request::send(buffer, count, datatype, leader_of_root, COLL_TAG_BCAST, comm); } } #if defined(_MCST_SUPPORT_) if (comm_ptr->ch.is_mcast_ok) { mpi_errno = MPIR_Mcast_inter_node_MV2(buffer, count, datatype, root, comm_ptr, errflag); if (mpi_errno == MPI_SUCCESS) { goto fn_exit; } else { goto fn_fail; } } #endif /* if (local_rank == 0) { leader_comm = comm->get_leaders_comm(); root = leader_root; } if (MV2_Bcast_function == &MPIR_Pipelined_Bcast_MV2) { mpi_errno = MPIR_Pipelined_Bcast_MV2(buffer, count, datatype, root, comm); } else if (MV2_Bcast_function == &MPIR_Bcast_scatter_ring_allgather_shm_MV2) { mpi_errno = MPIR_Bcast_scatter_ring_allgather_shm_MV2(buffer, count, datatype, root, comm); } else */{ if (local_rank == 0) { /* if (MV2_Bcast_function == &MPIR_Knomial_Bcast_inter_node_wrapper_MV2) { mpi_errno = MPIR_Knomial_Bcast_inter_node_wrapper_MV2(buffer, count, datatype, root, comm); } else {*/ mpi_errno = MV2_Bcast_function(buffer, count, datatype, leader_root, leader_comm); // } } } return mpi_errno; } int Coll_bcast_mvapich2_knomial_intra_node::bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int local_size = 0, rank; int mpi_errno = MPI_SUCCESS; MPI_Request *reqarray = NULL; MPI_Status *starray = NULL; int src, dst, mask, relative_rank; int k; if (MV2_Bcast_function==NULL){ MV2_Bcast_function=Coll_bcast_mpich::bcast; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= Coll_bcast_mpich::bcast; } if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } local_size = comm->size(); rank = comm->rank(); reqarray=(MPI_Request *)xbt_malloc(2 * mv2_intra_node_knomial_factor * sizeof (MPI_Request)); starray=(MPI_Status *)xbt_malloc(2 * mv2_intra_node_knomial_factor * sizeof (MPI_Status)); /* intra-node k-nomial bcast */ if (local_size > 1) { relative_rank = (rank >= root) ? rank - root : rank - root + local_size; mask = 0x1; while (mask < local_size) { if (relative_rank % (mv2_intra_node_knomial_factor * mask)) { src = relative_rank / (mv2_intra_node_knomial_factor * mask) * (mv2_intra_node_knomial_factor * mask) + root; if (src >= local_size) { src -= local_size; } Request::recv(buffer, count, datatype, src, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); break; } mask *= mv2_intra_node_knomial_factor; } mask /= mv2_intra_node_knomial_factor; while (mask > 0) { int reqs = 0; for (k = 1; k < mv2_intra_node_knomial_factor; k++) { if (relative_rank + mask * k < local_size) { dst = rank + mask * k; if (dst >= local_size) { dst -= local_size; } reqarray[reqs++]=Request::isend(buffer, count, datatype, dst, COLL_TAG_BCAST, comm); } } Request::waitall(reqs, reqarray, starray); mask /= mv2_intra_node_knomial_factor; } } xbt_free(reqarray); xbt_free(starray); return mpi_errno; } int Coll_bcast_mvapich2_intra_node::bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int comm_size; int two_level_bcast = 1; size_t nbytes = 0; int is_homogeneous, is_contig; MPI_Aint type_size; void *tmp_buf = NULL; MPI_Comm shmem_comm; if (count == 0) return MPI_SUCCESS; if (MV2_Bcast_function==NULL){ MV2_Bcast_function=Coll_bcast_mpich::bcast; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= Coll_bcast_mpich::bcast; } if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } comm_size = comm->size(); // rank = comm->rank(); /* if (HANDLE_GET_KIND(datatype) == HANDLE_KIND_BUILTIN)*/ is_contig = 1; /* else { MPID_Datatype_get_ptr(datatype, dtp); is_contig = dtp->is_contig; } */ is_homogeneous = 1; #ifdef MPID_HAS_HETERO if (comm_ptr->is_hetero) is_homogeneous = 0; #endif /* MPI_Type_size() might not give the accurate size of the packed * datatype for heterogeneous systems (because of padding, encoding, * etc). On the other hand, MPI_Pack_size() can become very * expensive, depending on the implementation, especially for * heterogeneous systems. We want to use MPI_Type_size() wherever * possible, and MPI_Pack_size() in other places. */ //if (is_homogeneous) { type_size=datatype->size(); //} /* else {*/ /* MPIR_Pack_size_impl(1, datatype, &type_size);*/ /* }*/ nbytes = (size_t) (count) * (type_size); if (comm_size <= mv2_bcast_two_level_system_size) { if (nbytes > mv2_bcast_short_msg && nbytes < mv2_bcast_large_msg) { two_level_bcast = 1; } else { two_level_bcast = 0; } } if (two_level_bcast == 1 #if defined(_MCST_SUPPORT_) || comm_ptr->ch.is_mcast_ok #endif ) { if (not is_contig || not is_homogeneous) { tmp_buf = (void*)smpi_get_tmp_sendbuffer(nbytes); /* TODO: Pipeline the packing and communication */ // position = 0; /* if (rank == root) {*/ /* mpi_errno =*/ /* MPIR_Pack_impl(buffer, count, datatype, tmp_buf, nbytes, &position);*/ /* if (mpi_errno)*/ /* MPIU_ERR_POP(mpi_errno);*/ /* }*/ } shmem_comm = comm->get_intra_comm(); if (not is_contig || not is_homogeneous) { mpi_errno = MPIR_Bcast_inter_node_helper_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { mpi_errno = MPIR_Bcast_inter_node_helper_MV2(buffer, count, datatype, root, comm); } /* We are now done with the inter-node phase */ if (nbytes <= mv2_knomial_intra_node_threshold) { if (not is_contig || not is_homogeneous) { mpi_errno = MPIR_Shmem_Bcast_MV2(tmp_buf, nbytes, MPI_BYTE, root, shmem_comm); } else { mpi_errno = MPIR_Shmem_Bcast_MV2(buffer, count, datatype, root, shmem_comm); } } else { if (not is_contig || not is_homogeneous) { mpi_errno = MPIR_Knomial_Bcast_intra_node_MV2(tmp_buf, nbytes, MPI_BYTE, INTRA_NODE_ROOT, shmem_comm); } else { mpi_errno = MPIR_Knomial_Bcast_intra_node_MV2(buffer, count, datatype, INTRA_NODE_ROOT, shmem_comm); } } } else { if (nbytes <= mv2_bcast_short_msg) { mpi_errno = MPIR_Bcast_binomial_MV2(buffer, count, datatype, root, comm); } else { if (mv2_scatter_rd_inter_leader_bcast) { mpi_errno = MPIR_Bcast_scatter_ring_allgather_MV2(buffer, count, datatype, root, comm); } else { mpi_errno = MPIR_Bcast_scatter_doubling_allgather_MV2(buffer, count, datatype, root, comm); } } } return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-scatter-LR-allgather.cpp0000644000175000017500000001510313217757320025004 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include "smpi_status.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: bcast_scatter_LR_allgather * Return: int * Inputs: buff: send input buffer count: number of elements to send data_type: data type of elements being sent root: source of data comm: communicator * Descrp: broadcasts using a scatter followed by LR allgather. * Auther: MPIH / modified by Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_bcast_scatter_LR_allgather::bcast(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { MPI_Aint extent; MPI_Status status; int i, src, dst, rank, num_procs; int mask, relative_rank, curr_size, recv_size, send_size, nbytes; int scatter_size, left, right, next_src, *recv_counts, *disps; int tag = COLL_TAG_BCAST; rank = comm->rank(); num_procs = comm->size(); extent = data_type->get_extent(); nbytes = extent * count; scatter_size = (nbytes + num_procs - 1) / num_procs; // ceiling division curr_size = (rank == root) ? nbytes : 0; // root starts with all the data relative_rank = (rank >= root) ? rank - root : rank - root + num_procs; mask = 0x1; while (mask < num_procs) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += num_procs; recv_size = nbytes - relative_rank * scatter_size; // recv_size is larger than what might actually be sent by the // sender. We don't need compute the exact value because MPI // allows you to post a larger recv. if (recv_size <= 0) curr_size = 0; // this process doesn't receive any data // because of uneven division else { Request::recv((char *) buff + relative_rank * scatter_size, recv_size, MPI_BYTE, src, tag, comm, &status); curr_size = Status::get_count(&status, MPI_BYTE); } break; } mask <<= 1; } // This process is responsible for all processes that have bits // set from the LSB upto (but not including) mask. Because of // the "not including", we start by shifting mask back down // one. mask >>= 1; while (mask > 0) { if (relative_rank + mask < num_procs) { send_size = curr_size - scatter_size * mask; // mask is also the size of this process's subtree if (send_size > 0) { dst = rank + mask; if (dst >= num_procs) dst -= num_procs; Request::send((char *) buff + scatter_size * (relative_rank + mask), send_size, MPI_BYTE, dst, tag, comm); curr_size -= send_size; } } mask >>= 1; } // done scatter now do allgather recv_counts = (int *) xbt_malloc(sizeof(int) * num_procs); disps = (int *) xbt_malloc(sizeof(int) * num_procs); for (i = 0; i < num_procs; i++) { recv_counts[i] = nbytes - i * scatter_size; if (recv_counts[i] > scatter_size) recv_counts[i] = scatter_size; if (recv_counts[i] < 0) recv_counts[i] = 0; } disps[0] = 0; for (i = 1; i < num_procs; i++) disps[i] = disps[i - 1] + recv_counts[i - 1]; left = (num_procs + rank - 1) % num_procs; right = (rank + 1) % num_procs; src = rank; next_src = left; for (i = 1; i < num_procs; i++) { Request::sendrecv((char *) buff + disps[(src - root + num_procs) % num_procs], recv_counts[(src - root + num_procs) % num_procs], MPI_BYTE, right, tag, (char *) buff + disps[(next_src - root + num_procs) % num_procs], recv_counts[(next_src - root + num_procs) % num_procs], MPI_BYTE, left, tag, comm, &status); src = next_src; next_src = (num_procs + next_src - 1) % num_procs; } free(recv_counts); free(disps); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-arrival-pattern-aware-wait.cpp0000644000175000017500000001722613217757320026245 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" int bcast_arrival_pattern_aware_wait_segment_size_in_byte = 8192; #ifndef BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE #define BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE 1024 #endif #ifndef BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE #define BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE 128 #endif namespace simgrid{ namespace smpi{ /* Non-topology-specific pipelined linear-bcast function */ int Coll_bcast_arrival_pattern_aware_wait::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int rank, size; int i, j, k; int tag = -COLL_TAG_BCAST; int will_send[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int sent_count; int header_index; int flag_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int already_sent[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int header_buf[BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE]; char temp_buf[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int max_node = BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; int header_size = BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE; MPI_Aint extent; extent = datatype->get_extent(); /* source and destination */ int to, from; rank = comm->rank(); size = comm->size(); /* segment is segment size in number of elements (not bytes) */ int segment = bcast_arrival_pattern_aware_wait_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < max_node; i++) { already_sent[i] = 0; } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { segment = count; pipe_length = 1; } /* start pipeline bcast */ send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root */ if (rank == 0) { sent_count = 0; int iteration = 0; for (i = 0; i < BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; i++) will_send[i] = 0; while (sent_count < (size - 1)) { iteration++; /* loop k times to let more processes arrive before start sending data */ for (k = 0; k < 3; k++) { for (i = 1; i < size; i++) { if ((already_sent[i] == 0) && (will_send[i] == 0)) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; Request::recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); i = 0; } } } } header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((will_send[i] == 1) && (already_sent[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; /* send header */ Request::send(header_buf, header_size, MPI_INT, to, tag, comm); /* send data - pipeline */ for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *)buf + (i * increment), segment, datatype, to, tag, comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* end - send header followed by data */ /* randomly MPI_Send to one node */ /* this part has been commented out - performance-wise */ else if (2 == 3) { /* search for the first node that never received data before */ for (i = 0; i < size; i++) { if (i == root) continue; if (already_sent[i] == 0) { header_buf[0] = i; header_buf[1] = -1; to = i; Request::send(header_buf, header_size, MPI_INT, to, tag, comm); /* still need to chop data so that we can use the same non-root code */ for (j = 0; j < pipe_length; j++) { Request::send((char *)buf + (j * increment), segment, datatype, to, tag, comm); } } } } } /* end - while (send_count < size-1) loop */ } /* end - root */ /* none root */ else { /* send 1-byte message to root */ Request::send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header forward when required */ request = Request::irecv(header_buf, header_size, MPI_INT, MPI_ANY_SOURCE, tag, comm); Request::wait(&request, MPI_STATUS_IGNORE); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } to = header_buf[myordering + 1]; if (myordering == 0) { from = 0; } else { from = header_buf[myordering - 1]; } /* send header when required */ if (to != -1) { Request::send(header_buf, header_size, MPI_INT, to, tag, comm); } /* receive data */ for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *)buf + (i * increment), segment, datatype, from, tag, comm); } /* forward data */ if (to != -1) { for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], MPI_STATUS_IGNORE); send_request_array[i] = Request::isend((char *)buf + (i * increment), segment, datatype, to, tag, comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* recv only */ else { Request::waitall((pipe_length), recv_request_array, recv_status_array); } } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_arrival_pattern_aware_wait use default MPI_bcast."); Colls::bcast((char *)buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-NTSB.cpp0000644000175000017500000001503613217757320021636 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" int bcast_NTSB_segment_size_in_byte = 8192; namespace simgrid{ namespace smpi{ int Coll_bcast_NTSB::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; int rank, size; int i; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Aint extent; extent = datatype->get_extent(); rank = comm->rank(); size = comm->size(); /* source node and destination nodes (same through out the functions) */ int from = (rank - 1) / 2; int to_left = rank * 2 + 1; int to_right = rank * 2 + 2; if (to_left >= size) to_left = -1; if (to_right >= size) to_right = -1; /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSB_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first */ if (root != 0) { if (rank == root) { Request::send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { Request::recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { /* case: root */ if (rank == 0) { /* case root has only a left child */ if (to_right == -1) { Request::send(buf, count, datatype, to_left, tag, comm); } /* case root has both left and right children */ else { Request::send(buf, count, datatype, to_left, tag, comm); Request::send(buf, count, datatype, to_right, tag, comm); } } /* case: leaf ==> receive only */ else if (to_left == -1) { Request::recv(buf, count, datatype, from, tag, comm, &status); } /* case: intermidiate node with only left child ==> relay message */ else if (to_right == -1) { Request::recv(buf, count, datatype, from, tag, comm, &status); Request::send(buf, count, datatype, to_left, tag, comm); } /* case: intermidiate node with both left and right children ==> relay message */ else { Request::recv(buf, count, datatype, from, tag, comm, &status); Request::send(buf, count, datatype, to_left, tag, comm); Request::send(buf, count, datatype, to_right, tag, comm); } return MPI_SUCCESS; } // pipelining else { send_request_array = (MPI_Request *) xbt_malloc(2 * (size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc(2 * (size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* case: root */ if (rank == 0) { /* case root has only a left child */ if (to_right == -1) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* case root has both left and right children */ else { for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); send_request_array[i + pipe_length] = Request::isend((char *) buf + (i * increment), segment, datatype, to_right, tag + i, comm); } Request::waitall((2 * pipe_length), send_request_array, send_status_array); } } /* case: leaf ==> receive only */ else if (to_left == -1) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } Request::waitall((pipe_length), recv_request_array, recv_status_array); } /* case: intermidiate node with only left child ==> relay message */ else if (to_right == -1) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); } Request::waitall(pipe_length, send_request_array, send_status_array); } /* case: intermidiate node with both left and right children ==> relay message */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); send_request_array[i] = Request::isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); send_request_array[i + pipe_length] = Request::isend((char *) buf + (i * increment), segment, datatype, to_right, tag + i, comm); } Request::waitall((2 * pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_NTSB use default MPI_bcast."); Colls::bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/bcast/bcast-SMP-binary.cpp0000644000175000017500000002236213217757320023011 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" int bcast_SMP_binary_segment_byte = 8192; namespace simgrid{ namespace smpi{ int Coll_bcast_SMP_binary::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *request_array; MPI_Status *status_array; int rank, size; int i; MPI_Aint extent; extent = datatype->get_extent(); rank = comm->rank(); size = comm->size(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int host_num_core=1; if (comm->is_uniform()){ host_num_core = comm->get_intra_comm()->size(); }else{ //implementation buggy in this case return Coll_bcast_mpich::bcast( buf , count, datatype, root, comm); } int segment = bcast_SMP_binary_segment_byte / extent; int pipe_length = count / segment; int remainder = count % segment; int to_intra_left = (rank / host_num_core) * host_num_core + (rank % host_num_core) * 2 + 1; int to_intra_right = (rank / host_num_core) * host_num_core + (rank % host_num_core) * 2 + 2; int to_inter_left = ((rank / host_num_core) * 2 + 1) * host_num_core; int to_inter_right = ((rank / host_num_core) * 2 + 2) * host_num_core; int from_inter = (((rank / host_num_core) - 1) / 2) * host_num_core; int from_intra = (rank / host_num_core) * host_num_core + ((rank % host_num_core) - 1) / 2; int increment = segment * extent; int base = (rank / host_num_core) * host_num_core; int num_core = host_num_core; if (((rank / host_num_core) * host_num_core) == ((size / host_num_core) * host_num_core)) num_core = size - (rank / host_num_core) * host_num_core; // if root is not zero send to rank zero first if (root != 0) { if (rank == root) Request::send(buf, count, datatype, 0, tag, comm); else if (rank == 0) Request::recv(buf, count, datatype, root, tag, comm, &status); } // when a message is smaller than a block size => no pipeline if (count <= segment) { // case ROOT-of-each-SMP if (rank % host_num_core == 0) { // case ROOT if (rank == 0) { //printf("node %d left %d right %d\n",rank,to_inter_left,to_inter_right); if (to_inter_left < size) Request::send(buf, count, datatype, to_inter_left, tag, comm); if (to_inter_right < size) Request::send(buf, count, datatype, to_inter_right, tag, comm); if ((to_intra_left - base) < num_core) Request::send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) Request::send(buf, count, datatype, to_intra_right, tag, comm); } // case LEAVES ROOT-of-eash-SMP else if (to_inter_left >= size) { //printf("node %d from %d\n",rank,from_inter); request = Request::irecv(buf, count, datatype, from_inter, tag, comm); Request::wait(&request, &status); if ((to_intra_left - base) < num_core) Request::send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) Request::send(buf, count, datatype, to_intra_right, tag, comm); } // case INTERMEDIAT ROOT-of-each-SMP else { //printf("node %d left %d right %d from %d\n",rank,to_inter_left,to_inter_right,from_inter); request = Request::irecv(buf, count, datatype, from_inter, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to_inter_left, tag, comm); if (to_inter_right < size) Request::send(buf, count, datatype, to_inter_right, tag, comm); if ((to_intra_left - base) < num_core) Request::send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) Request::send(buf, count, datatype, to_intra_right, tag, comm); } } // case non ROOT-of-each-SMP else { // case leaves if ((to_intra_left - base) >= num_core) { request = Request::irecv(buf, count, datatype, from_intra, tag, comm); Request::wait(&request, &status); } // case intermediate else { request = Request::irecv(buf, count, datatype, from_intra, tag, comm); Request::wait(&request, &status); Request::send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) Request::send(buf, count, datatype, to_intra_right, tag, comm); } } return MPI_SUCCESS; } // pipeline bcast else { request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); // case ROOT-of-each-SMP if (rank % host_num_core == 0) { // case ROOT if (rank == 0) { for (i = 0; i < pipe_length; i++) { //printf("node %d left %d right %d\n",rank,to_inter_left,to_inter_right); if (to_inter_left < size) Request::send((char *) buf + (i * increment), segment, datatype, to_inter_left, (tag + i), comm); if (to_inter_right < size) Request::send((char *) buf + (i * increment), segment, datatype, to_inter_right, (tag + i), comm); if ((to_intra_left - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } // case LEAVES ROOT-of-eash-SMP else if (to_inter_left >= size) { //printf("node %d from %d\n",rank,from_inter); for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); if ((to_intra_left - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } // case INTERMEDIAT ROOT-of-each-SMP else { //printf("node %d left %d right %d from %d\n",rank,to_inter_left,to_inter_right,from_inter); for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); Request::send((char *) buf + (i * increment), segment, datatype, to_inter_left, (tag + i), comm); if (to_inter_right < size) Request::send((char *) buf + (i * increment), segment, datatype, to_inter_right, (tag + i), comm); if ((to_intra_left - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } } // case non-ROOT-of-each-SMP else { // case leaves if ((to_intra_left - base) >= num_core) { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } Request::waitall((pipe_length), request_array, status_array); } // case intermediate else { for (i = 0; i < pipe_length; i++) { request_array[i] = Request::irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&request_array[i], &status); Request::send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) Request::send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } } free(request_array); free(status_array); } // when count is not divisible by block size, use default BCAST for the remainder if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_SMP_binary use default MPI_bcast."); Colls::bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return 1; } } } SimGrid-3.18/src/smpi/colls/scatter/0000755000175000017500000000000013217757320017636 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/scatter/scatter-ompi.cpp0000644000175000017500000001734513217757320022763 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_scatter_ompi_binomial::scatter(void* sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int line = -1; int i; int rank; int vrank; int size; int total_send = 0; char *ptmp = NULL; char *tempbuf = NULL; int err; ompi_coll_tree_t* bmtree; MPI_Status status; MPI_Aint sextent, slb, strue_lb, strue_extent; MPI_Aint rextent, rlb, rtrue_lb, rtrue_extent; size = comm->size(); rank = comm->rank(); XBT_DEBUG( "Coll_scatter_ompi_binomial::scatter rank %d", rank); /* create the binomial tree */ // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); bmtree = ompi_coll_tuned_topo_build_in_order_bmtree( comm, root);//ompi_ data->cached_in_order_bmtree; sdtype->extent(&slb, &sextent); sdtype->extent(&strue_lb, &strue_extent); rdtype->extent(&rlb, &rextent); rdtype->extent(&rtrue_lb, &rtrue_extent); vrank = (rank - root + size) % size; if (rank == root) { if (0 == root) { /* root on 0, just use the send buffer */ ptmp = (char*)sbuf; if (rbuf != MPI_IN_PLACE) { /* local copy to rbuf */ err = Datatype::copy(sbuf, scount, sdtype, rbuf, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } else { /* root is not on 0, allocate temp buffer for send */ tempbuf = (char*)smpi_get_tmp_sendbuffer(strue_extent + (scount * size - 1) * sextent); if (NULL == tempbuf) { err = MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - slb; /* and rotate data so they will eventually in the right place */ err = Datatype::copy((char*)sbuf + sextent * root * scount, scount * (size - root), sdtype, ptmp, scount * (size - root), sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = Datatype::copy((char*)sbuf, scount * root, sdtype, ptmp + sextent * scount * (size - root), scount * root, sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } if (rbuf != MPI_IN_PLACE) { /* local copy to rbuf */ err = Datatype::copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } total_send = scount; } else if (not(vrank % 2)) { /* non-root, non-leaf nodes, allocate temp buffer for recv * the most we need is rcount*size/2 */ tempbuf = (char*)smpi_get_tmp_recvbuffer(rtrue_extent + (rcount * size - 1) * rextent); if (NULL == tempbuf) { err = MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - rlb; sdtype = rdtype; scount = rcount; sextent = rextent; total_send = scount; } else { /* leaf nodes, just use rbuf */ ptmp = (char*)rbuf; } if (not(vrank % 2)) { if (rank != root) { /* recv from parent on non-root */ Request::recv(ptmp, rcount * size, rdtype, bmtree->tree_prev, COLL_TAG_SCATTER, comm, &status); /* local copy to rbuf */ Datatype::copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); } /* send to children on all non-leaf */ for (i = 0; i < bmtree->tree_nextsize; i++) { int mycount = 0, vkid; /* figure out how much data I have to send to this child */ vkid = (bmtree->tree_next[i] - root + size) % size; mycount = vkid - vrank; if (mycount > (size - vkid)) mycount = size - vkid; mycount *= scount; Request::send(ptmp + total_send * sextent, mycount, sdtype, bmtree->tree_next[i], COLL_TAG_SCATTER, comm); total_send += mycount; } } else { /* recv from parent on leaf nodes */ Request::recv(ptmp, rcount, rdtype, bmtree->tree_prev, COLL_TAG_SCATTER, comm, &status); } if (NULL != tempbuf) smpi_free_tmp_buffer(tempbuf); // not FIXME : store the tree, as done in ompi, instead of calculating it each time ? ompi_coll_tuned_topo_destroy_tree(&bmtree); return MPI_SUCCESS; err_hndl: if (NULL != tempbuf) free(tempbuf); XBT_DEBUG("%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * JPG following the examples from other coll_tuned implementations. Dec06. */ /* copied function (with appropriate renaming) starts here */ /* * scatter_intra * * Function: - basic scatter operation * Accepts: - same arguments as MPI_Scatter() * Returns: - MPI_SUCCESS or error code */ int Coll_scatter_ompi_basic_linear::scatter(void* sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int i, rank, size, err; char *ptmp; ptrdiff_t lb, incr; /* Initialize */ rank = comm->rank(); size = comm->size(); /* If not root, receive data. */ if (rank != root) { Request::recv(rbuf, rcount, rdtype, root, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); return MPI_SUCCESS; } /* I am the root, loop sending data. */ err = sdtype->extent(&lb, &incr); if (MPI_SUCCESS != err) { return MPI_ERR_OTHER; } incr *= scount; for (i = 0, ptmp = (char *) sbuf; i < size; ++i, ptmp += incr) { /* simple optimization */ if (i == rank) { if (MPI_IN_PLACE != rbuf) { err = Datatype::copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); } } else { Request::send(ptmp, scount, sdtype, i, COLL_TAG_SCATTER, comm); } if (MPI_SUCCESS != err) { return err; } } /* All done */ return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/scatter/scatter-mvapich-two-level.cpp0000644000175000017500000004043313217757320025354 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" #define MPIR_Scatter_MV2_Binomial Coll_scatter_ompi_binomial::scatter #define MPIR_Scatter_MV2_Direct Coll_scatter_ompi_basic_linear::scatter extern int (*MV2_Scatter_intra_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm); namespace simgrid{ namespace smpi{ int Coll_scatter_mvapich2_two_level_direct::scatter(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int comm_size, rank; int local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = -1; int mpi_errno = MPI_SUCCESS; int recvtype_size, sendtype_size, nbytes; void *tmp_buf = NULL; void *leader_scatter_buf = NULL; MPI_Status status; int leader_root, leader_of_root = -1; MPI_Comm shmem_comm, leader_comm; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Scatter_intra_function==NULL) MV2_Scatter_intra_function=Coll_scatter_mpich::scatter; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } comm_size = comm->size(); rank = comm->rank(); if (((rank == root) && (recvcnt == 0)) || ((rank != root) && (sendcnt == 0))) { return MPI_SUCCESS; } /* extract the rank,size information for the intra-node * communicator */ shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader * communicator */ leader_comm = comm->get_leaders_comm(); leader_comm_size = leader_comm->size(); leader_comm_rank = leader_comm->rank(); } if (local_size == comm_size) { /* purely intra-node scatter. Just use the direct algorithm and we are done */ mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { recvtype_size=recvtype->size(); sendtype_size=sendtype->size(); if (rank == root) { nbytes = sendcnt * sendtype_size; } else { nbytes = recvcnt * recvtype_size; } if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ tmp_buf = smpi_get_tmp_sendbuffer(nbytes * local_size); } leader_comm = comm->get_leaders_comm(); int* leaders_map = comm->get_leaders_map(); leader_of_root = comm->group()->rank(leaders_map[root]); leader_root = leader_comm->group()->rank(leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { /* The root of the scatter operation is not the node leader. Recv * data from the node leader */ leader_scatter_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); Request::recv(leader_scatter_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_SCATTER, comm, &status); } if (rank == root && local_rank != 0) { /* The root of the scatter operation is not the node leader. Send * data to the node leader */ Request::send(sendbuf, sendcnt * comm_size, sendtype, leader_of_root, COLL_TAG_SCATTER, comm ); } if (leader_comm_size > 1 && local_rank == 0) { if (not comm->is_uniform()) { int* displs = NULL; int* sendcnts = NULL; int* node_sizes; int i = 0; node_sizes = comm->get_non_uniform_map(); if (root != leader_of_root) { if (leader_comm_rank == leader_root) { displs = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; sendcnts[i] = node_sizes[i] * nbytes; } } Colls::scatterv(leader_scatter_buf, sendcnts, displs, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { if (leader_comm_rank == leader_root) { displs = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts[0] = node_sizes[0] * sendcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * sendcnt; sendcnts[i] = node_sizes[i] * sendcnt; } } Colls::scatterv(sendbuf, sendcnts, displs, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(sendcnts); } } else { if (leader_of_root != root) { mpi_errno = MPIR_Scatter_MV2_Direct(leader_scatter_buf, nbytes * local_size, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt * local_size, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } /* The leaders are now done with the inter-leader part. Scatter the data within the nodes */ if (rank == root && recvbuf == MPI_IN_PLACE) { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, (void *)sendbuf, sendcnt, sendtype, 0, shmem_comm); } else { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* check if multiple threads are calling this collective function */ if (comm_size != local_size && local_rank == 0) { smpi_free_tmp_buffer(tmp_buf); if (leader_of_root == rank && root != rank) { smpi_free_tmp_buffer(leader_scatter_buf); } } return (mpi_errno); } int Coll_scatter_mvapich2_two_level_binomial::scatter(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int comm_size, rank; int local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = -1; int mpi_errno = MPI_SUCCESS; int recvtype_size, sendtype_size, nbytes; void *tmp_buf = NULL; void *leader_scatter_buf = NULL; MPI_Status status; int leader_root = -1, leader_of_root = -1; MPI_Comm shmem_comm, leader_comm; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Scatter_intra_function==NULL) MV2_Scatter_intra_function=Coll_scatter_mpich::scatter; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } comm_size = comm->size(); rank = comm->rank(); if (((rank == root) && (recvcnt == 0)) || ((rank != root) && (sendcnt == 0))) { return MPI_SUCCESS; } /* extract the rank,size information for the intra-node * communicator */ shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader * communicator */ leader_comm = comm->get_leaders_comm(); leader_comm_size = leader_comm->size(); leader_comm_rank = leader_comm->rank(); } if (local_size == comm_size) { /* purely intra-node scatter. Just use the direct algorithm and we are done */ mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { recvtype_size=recvtype->size(); sendtype_size=sendtype->size(); if (rank == root) { nbytes = sendcnt * sendtype_size; } else { nbytes = recvcnt * recvtype_size; } if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ tmp_buf = smpi_get_tmp_sendbuffer(nbytes * local_size); } leader_comm = comm->get_leaders_comm(); int* leaders_map = comm->get_leaders_map(); leader_of_root = comm->group()->rank(leaders_map[root]); leader_root = leader_comm->group()->rank(leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { /* The root of the scatter operation is not the node leader. Recv * data from the node leader */ leader_scatter_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); Request::recv(leader_scatter_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_SCATTER, comm, &status); } if (rank == root && local_rank != 0) { /* The root of the scatter operation is not the node leader. Send * data to the node leader */ Request::send(sendbuf, sendcnt * comm_size, sendtype, leader_of_root, COLL_TAG_SCATTER, comm); } if (leader_comm_size > 1 && local_rank == 0) { if (not comm->is_uniform()) { int* displs = NULL; int* sendcnts = NULL; int* node_sizes; int i = 0; node_sizes = comm->get_non_uniform_map(); if (root != leader_of_root) { if (leader_comm_rank == leader_root) { displs = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; sendcnts[i] = node_sizes[i] * nbytes; } } Colls::scatterv(leader_scatter_buf, sendcnts, displs, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { if (leader_comm_rank == leader_root) { displs = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); sendcnts[0] = node_sizes[0] * sendcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * sendcnt; sendcnts[i] = node_sizes[i] * sendcnt; } } Colls::scatterv(sendbuf, sendcnts, displs, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(sendcnts); } } else { if (leader_of_root != root) { mpi_errno = MPIR_Scatter_MV2_Binomial(leader_scatter_buf, nbytes * local_size, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { mpi_errno = MPIR_Scatter_MV2_Binomial(sendbuf, sendcnt * local_size, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } /* The leaders are now done with the inter-leader part. Scatter the data within the nodes */ if (rank == root && recvbuf == MPI_IN_PLACE) { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, (void *)sendbuf, sendcnt, sendtype, 0, shmem_comm); } else { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* check if multiple threads are calling this collective function */ if (comm_size != local_size && local_rank == 0) { smpi_free_tmp_buffer(tmp_buf); if (leader_of_root == rank && root != rank) { smpi_free_tmp_buffer(leader_scatter_buf); } } return (mpi_errno); } } } SimGrid-3.18/src/smpi/colls/smpi_mvapich2_selector.cpp0000644000175000017500000010371013217757320023340 0ustar mquinsonmquinson/* selector for collective algorithms based on mvapich decision logic */ /* Copyright (c) 2009-2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.hpp" #include "smpi_mvapich2_selector_stampede.hpp" namespace simgrid{ namespace smpi{ int Coll_alltoall_mvapich2::alltoall( void *sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { if(mv2_alltoall_table_ppn_conf==NULL) init_mv2_alltoall_tables_stampede(); int sendtype_size, recvtype_size, comm_size; char * tmp_buf = NULL; int mpi_errno=MPI_SUCCESS; int range = 0; int range_threshold = 0; int conf_index = 0; comm_size = comm->size(); sendtype_size=sendtype->size(); recvtype_size=recvtype->size(); long nbytes = sendtype_size * sendcount; /* check if safe to use partial subscription mode */ /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_alltoall_tuning_table[conf_index] - 1)) && (comm_size > mv2_alltoall_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_alltoall_thresholds_table[conf_index][range].size_table - 1)) && (nbytes > mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold].max) && (mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold].max != -1)) { range_threshold++; } MV2_Alltoall_function = mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold] .MV2_pt_Alltoall_function; if(sendbuf != MPI_IN_PLACE) { mpi_errno = MV2_Alltoall_function(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } else { range_threshold = 0; if(nbytes < mv2_alltoall_thresholds_table[conf_index][range].in_place_algo_table[range_threshold].min ||nbytes > mv2_alltoall_thresholds_table[conf_index][range].in_place_algo_table[range_threshold].max ) { tmp_buf = (char *)smpi_get_tmp_sendbuffer( comm_size * recvcount * recvtype_size ); Datatype::copy((char *)recvbuf, comm_size*recvcount, recvtype, (char *)tmp_buf, comm_size*recvcount, recvtype); mpi_errno = MV2_Alltoall_function(tmp_buf, recvcount, recvtype, recvbuf, recvcount, recvtype, comm ); smpi_free_tmp_buffer(tmp_buf); } else { mpi_errno = MPIR_Alltoall_inplace_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm ); } } return (mpi_errno); } int Coll_allgather_mvapich2::allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; long nbytes = 0, comm_size, recvtype_size; int range = 0; int partial_sub_ok = 0; int conf_index = 0; int range_threshold = 0; int is_two_level = 0; int local_size = -1; MPI_Comm shmem_comm; //MPI_Comm *shmem_commptr=NULL; /* Get the size of the communicator */ comm_size = comm->size(); recvtype_size=recvtype->size(); nbytes = recvtype_size * recvcount; if(mv2_allgather_table_ppn_conf==NULL) init_mv2_allgather_tables_stampede(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int i; if (comm->is_uniform()){ shmem_comm = comm->get_intra_comm(); local_size = shmem_comm->size(); i = 0; if (mv2_allgather_table_ppn_conf[0] == -1) { // Indicating user defined tuning conf_index = 0; goto conf_check_end; } do { if (local_size == mv2_allgather_table_ppn_conf[i]) { conf_index = i; partial_sub_ok = 1; break; } i++; } while(i < mv2_allgather_num_ppn_conf); } conf_check_end: if (partial_sub_ok != 1) { conf_index = 0; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allgather_tuning_table[conf_index] - 1)) && (comm_size > mv2_allgather_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_allgather_thresholds_table[conf_index][range].size_inter_table - 1)) && (nbytes > mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold].max) && (mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Allgatherction = mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold]. MV2_pt_Allgatherction; is_two_level = mv2_allgather_thresholds_table[conf_index][range].two_level[range_threshold]; /* intracommunicator */ if(is_two_level ==1){ if(partial_sub_ok ==1){ if (comm->is_blocked()){ mpi_errno = MPIR_2lvl_Allgather_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }else{ mpi_errno = Coll_allgather_mpich::allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } } else { mpi_errno = MPIR_Allgather_RD_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } } else if(MV2_Allgatherction == &MPIR_Allgather_Bruck_MV2 || MV2_Allgatherction == &MPIR_Allgather_RD_MV2 || MV2_Allgatherction == &MPIR_Allgather_Ring_MV2) { mpi_errno = MV2_Allgatherction(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }else{ return MPI_ERR_OTHER; } return mpi_errno; } int Coll_gather_mvapich2::gather(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { if(mv2_gather_thresholds_table==NULL) init_mv2_gather_tables_stampede(); int mpi_errno = MPI_SUCCESS; int range = 0; int range_threshold = 0; int range_intra_threshold = 0; long nbytes = 0; int comm_size = 0; int recvtype_size, sendtype_size; int rank = -1; comm_size = comm->size(); rank = comm->rank(); if (rank == root) { recvtype_size=recvtype->size(); nbytes = recvcnt * recvtype_size; } else { sendtype_size=sendtype->size(); nbytes = sendcnt * sendtype_size; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_gather_tuning_table - 1)) && (comm_size > mv2_gather_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_gather_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_gather_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_gather_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra node function */ while ((range_intra_threshold < (mv2_gather_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_gather_thresholds_table[range].intra_node[range_intra_threshold].max) && (mv2_gather_thresholds_table[range].intra_node[range_intra_threshold].max != -1)) { range_intra_threshold++; } if (comm->is_blocked() ) { // Set intra-node function pt for gather_two_level MV2_Gather_intra_node_function = mv2_gather_thresholds_table[range].intra_node[range_intra_threshold]. MV2_pt_Gather_function; //Set inter-leader pt MV2_Gather_inter_leader_function = mv2_gather_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Gather_function; // We call Gather function mpi_errno = MV2_Gather_inter_leader_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { // Indeed, direct (non SMP-aware)gather is MPICH one mpi_errno = Coll_gather_mpich::gather(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } return mpi_errno; } int Coll_allgatherv_mvapich2::allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm ) { int mpi_errno = MPI_SUCCESS; int range = 0, comm_size, total_count, recvtype_size, i; int range_threshold = 0; long nbytes = 0; if(mv2_allgatherv_thresholds_table==NULL) init_mv2_allgatherv_tables_stampede(); comm_size = comm->size(); total_count = 0; for (i = 0; i < comm_size; i++) total_count += recvcounts[i]; recvtype_size=recvtype->size(); nbytes = total_count * recvtype_size; /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allgatherv_tuning_table - 1)) && (comm_size > mv2_allgatherv_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_allgatherv_thresholds_table[range].size_inter_table - 1)) && (nbytes > comm_size * mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Allgatherv_function = mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Allgatherv_function; if (MV2_Allgatherv_function == &MPIR_Allgatherv_Rec_Doubling_MV2) { if (not(comm_size & (comm_size - 1))) { mpi_errno = MPIR_Allgatherv_Rec_Doubling_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } else { mpi_errno = MPIR_Allgatherv_Bruck_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } } else { mpi_errno = MV2_Allgatherv_function(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } return mpi_errno; } int Coll_allreduce_mvapich2::allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; //int rank = 0, int comm_size = 0; comm_size = comm->size(); //rank = comm->rank(); if (count == 0) { return MPI_SUCCESS; } if (mv2_allreduce_thresholds_table == NULL) init_mv2_allreduce_tables_stampede(); /* check if multiple threads are calling this collective function */ MPI_Aint sendtype_size = 0; long nbytes = 0; int range = 0, range_threshold = 0, range_threshold_intra = 0; int is_two_level = 0; int is_commutative = 0; MPI_Aint true_lb, true_extent; sendtype_size=datatype->size(); nbytes = count * sendtype_size; datatype->extent(&true_lb, &true_extent); //MPI_Op *op_ptr; //is_commutative = op->is_commutative(); { /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allreduce_tuning_table - 1)) && (comm_size > mv2_allreduce_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ /* skip mcast poiters if mcast is not available */ if(mv2_allreduce_thresholds_table[range].mcast_enabled != 1){ while ((range_threshold < (mv2_allreduce_thresholds_table[range].size_inter_table - 1)) && ((mv2_allreduce_thresholds_table[range]. inter_leader[range_threshold].MV2_pt_Allreducection == &MPIR_Allreduce_mcst_reduce_redscat_gather_MV2) || (mv2_allreduce_thresholds_table[range]. inter_leader[range_threshold].MV2_pt_Allreducection == &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2) )) { range_threshold++; } } while ((range_threshold < (mv2_allreduce_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_allreduce_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_allreduce_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } if(mv2_allreduce_thresholds_table[range].is_two_level_allreduce[range_threshold] == 1){ is_two_level = 1; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_allreduce_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra].max) && (mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Allreducection = mv2_allreduce_thresholds_table[range].inter_leader[range_threshold] .MV2_pt_Allreducection; MV2_Allreduce_intra_function = mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra] .MV2_pt_Allreducection; /* check if mcast is ready, otherwise replace mcast with other algorithm */ if((MV2_Allreducection == &MPIR_Allreduce_mcst_reduce_redscat_gather_MV2)|| (MV2_Allreducection == &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2)){ { MV2_Allreducection = &MPIR_Allreduce_pt2pt_rd_MV2; } if(is_two_level != 1) { MV2_Allreducection = &MPIR_Allreduce_pt2pt_rd_MV2; } } if(is_two_level == 1){ // check if shm is ready, if not use other algorithm first if (is_commutative) { if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } mpi_errno = MPIR_Allreduce_two_level_MV2(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MPIR_Allreduce_pt2pt_rd_MV2(sendbuf, recvbuf, count, datatype, op, comm); } } else { mpi_errno = MV2_Allreducection(sendbuf, recvbuf, count, datatype, op, comm); } } //comm->ch.intra_node_done=0; return (mpi_errno); } int Coll_alltoallv_mvapich2::alltoallv(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { if (sbuf == MPI_IN_PLACE) { return Coll_alltoallv_ompi_basic_linear::alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } else /* For starters, just keep the original algorithm. */ return Coll_alltoallv_ring::alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int Coll_barrier_mvapich2::barrier(MPI_Comm comm) { return Coll_barrier_mvapich2_pair::barrier(comm); } int Coll_bcast_mvapich2::bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int comm_size/*, rank*/; int two_level_bcast = 1; long nbytes = 0; int range = 0; int range_threshold = 0; int range_threshold_intra = 0; int is_homogeneous, is_contig; MPI_Aint type_size; //, position; void *tmp_buf = NULL; MPI_Comm shmem_comm; //MPID_Datatype *dtp; if (count == 0) return MPI_SUCCESS; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } if (not mv2_bcast_thresholds_table) init_mv2_bcast_tables_stampede(); comm_size = comm->size(); //rank = comm->rank(); is_contig=1; /* if (HANDLE_GET_KIND(datatype) == HANDLE_KIND_BUILTIN)*/ /* is_contig = 1;*/ /* else {*/ /* MPID_Datatype_get_ptr(datatype, dtp);*/ /* is_contig = dtp->is_contig;*/ /* }*/ is_homogeneous = 1; /* MPI_Type_size() might not give the accurate size of the packed * datatype for heterogeneous systems (because of padding, encoding, * etc). On the other hand, MPI_Pack_size() can become very * expensive, depending on the implementation, especially for * heterogeneous systems. We want to use MPI_Type_size() wherever * possible, and MPI_Pack_size() in other places. */ //if (is_homogeneous) { type_size=datatype->size(); /* } else { MPIR_Pack_size_impl(1, datatype, &type_size); }*/ nbytes = (count) * (type_size); /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_bcast_tuning_table - 1)) && (comm_size > mv2_bcast_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_bcast_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_bcast_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_bcast_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_bcast_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_bcast_thresholds_table[range].intra_node[range_threshold_intra].max) && (mv2_bcast_thresholds_table[range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Bcast_function = mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Bcast_function; MV2_Bcast_intra_node_function = mv2_bcast_thresholds_table[range]. intra_node[range_threshold_intra].MV2_pt_Bcast_function; /* if (mv2_user_bcast_intra == NULL && */ /* MV2_Bcast_intra_node_function == &MPIR_Knomial_Bcast_intra_node_MV2) {*/ /* MV2_Bcast_intra_node_function = &MPIR_Shmem_Bcast_MV2;*/ /* }*/ if (mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. zcpy_pipelined_knomial_factor != -1) { zcpy_knomial_factor = mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. zcpy_pipelined_knomial_factor; } if (mv2_pipelined_zcpy_knomial_factor != -1) { zcpy_knomial_factor = mv2_pipelined_zcpy_knomial_factor; } if(MV2_Bcast_intra_node_function == NULL) { /* if tuning table do not have any intra selection, set func pointer to ** default one for mcast intra node */ MV2_Bcast_intra_node_function = &MPIR_Shmem_Bcast_MV2; } /* Set value of pipeline segment size */ bcast_segment_size = mv2_bcast_thresholds_table[range].bcast_segment_size; /* Set value of inter node knomial factor */ mv2_inter_node_knomial_factor = mv2_bcast_thresholds_table[range].inter_node_knomial_factor; /* Set value of intra node knomial factor */ mv2_intra_node_knomial_factor = mv2_bcast_thresholds_table[range].intra_node_knomial_factor; /* Check if we will use a two level algorithm or not */ two_level_bcast = #if defined(_MCST_SUPPORT_) mv2_bcast_thresholds_table[range].is_two_level_bcast[range_threshold] || comm->ch.is_mcast_ok; #else mv2_bcast_thresholds_table[range].is_two_level_bcast[range_threshold]; #endif if (two_level_bcast == 1) { if (not is_contig || not is_homogeneous) { tmp_buf = (void*)smpi_get_tmp_sendbuffer(nbytes); /* position = 0;*/ /* if (rank == root) {*/ /* mpi_errno =*/ /* MPIR_Pack_impl(buffer, count, datatype, tmp_buf, nbytes, &position);*/ /* if (mpi_errno)*/ /* MPIU_ERR_POP(mpi_errno);*/ /* }*/ } #ifdef CHANNEL_MRAIL_GEN2 if ((mv2_enable_zcpy_bcast == 1) && (&MPIR_Pipelined_Bcast_Zcpy_MV2 == MV2_Bcast_function)) { if (not is_contig || not is_homogeneous) { mpi_errno = MPIR_Pipelined_Bcast_Zcpy_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { mpi_errno = MPIR_Pipelined_Bcast_Zcpy_MV2(buffer, count, datatype, root, comm); } } else #endif /* defined(CHANNEL_MRAIL_GEN2) */ { shmem_comm = comm->get_intra_comm(); if (not is_contig || not is_homogeneous) { MPIR_Bcast_tune_inter_node_helper_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { MPIR_Bcast_tune_inter_node_helper_MV2(buffer, count, datatype, root, comm); } /* We are now done with the inter-node phase */ root = INTRA_NODE_ROOT; if (not is_contig || not is_homogeneous) { mpi_errno = MV2_Bcast_intra_node_function(tmp_buf, nbytes, MPI_BYTE, root, shmem_comm); } else { mpi_errno = MV2_Bcast_intra_node_function(buffer, count, datatype, root, shmem_comm); } } /* if (not is_contig || not is_homogeneous) {*/ /* if (rank != root) {*/ /* position = 0;*/ /* mpi_errno = MPIR_Unpack_impl(tmp_buf, nbytes, &position, buffer,*/ /* count, datatype);*/ /* }*/ /* }*/ } else { /* We use Knomial for intra node */ MV2_Bcast_intra_node_function = &MPIR_Knomial_Bcast_intra_node_MV2; /* if (mv2_enable_shmem_bcast == 0) {*/ /* Fall back to non-tuned version */ /* MPIR_Bcast_intra_MV2(buffer, count, datatype, root, comm);*/ /* } else {*/ mpi_errno = MV2_Bcast_function(buffer, count, datatype, root, comm); /* }*/ } return mpi_errno; } int Coll_reduce_mvapich2::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { if(mv2_reduce_thresholds_table == NULL) init_mv2_reduce_tables_stampede(); int mpi_errno = MPI_SUCCESS; int range = 0; int range_threshold = 0; int range_intra_threshold = 0; int is_commutative, pof2; int comm_size = 0; long nbytes = 0; int sendtype_size; int is_two_level = 0; comm_size = comm->size(); sendtype_size=datatype->size(); nbytes = count * sendtype_size; if (count == 0) return MPI_SUCCESS; is_commutative = (op==MPI_OP_NULL || op->is_commutative()); /* find nearest power-of-two less than or equal to comm_size */ for( pof2 = 1; pof2 <= comm_size; pof2 <<= 1 ); pof2 >>=1; /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_reduce_tuning_table - 1)) && (comm_size > mv2_reduce_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_reduce_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_reduce_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_reduce_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra node function */ while ((range_intra_threshold < (mv2_reduce_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold].max) && (mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold].max != -1)) { range_intra_threshold++; } /* Set intra-node function pt for reduce_two_level */ MV2_Reduce_intra_function = mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold]. MV2_pt_Reduce_function; /* Set inter-leader pt */ MV2_Reduce_function = mv2_reduce_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Reduce_function; if(mv2_reduce_intra_knomial_factor<0) { mv2_reduce_intra_knomial_factor = mv2_reduce_thresholds_table[range].intra_k_degree; } if(mv2_reduce_inter_knomial_factor<0) { mv2_reduce_inter_knomial_factor = mv2_reduce_thresholds_table[range].inter_k_degree; } if(mv2_reduce_thresholds_table[range].is_two_level_reduce[range_threshold] == 1){ is_two_level = 1; } /* We call Reduce function */ if(is_two_level == 1) { if (is_commutative == 1) { if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } mpi_errno = MPIR_Reduce_two_level_helper_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else if(MV2_Reduce_function == &MPIR_Reduce_inter_knomial_wrapper_MV2 ){ if(is_commutative ==1) { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else if(MV2_Reduce_function == &MPIR_Reduce_redscat_gather_MV2){ if (/*(HANDLE_GET_KIND(op) == HANDLE_KIND_BUILTIN) &&*/ (count >= pof2)) { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } return mpi_errno; } int Coll_reduce_scatter_mvapich2::reduce_scatter(void *sendbuf, void *recvbuf, int *recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int i = 0, comm_size = comm->size(), total_count = 0, type_size = 0, nbytes = 0; int range = 0; int range_threshold = 0; int is_commutative = 0; int* disps = new int[comm_size]; if(mv2_red_scat_thresholds_table==NULL) init_mv2_reduce_scatter_tables_stampede(); is_commutative=(op==MPI_OP_NULL || op->is_commutative()); for (i = 0; i < comm_size; i++) { disps[i] = total_count; total_count += recvcnts[i]; } type_size=datatype->size(); nbytes = total_count * type_size; if (is_commutative) { /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_red_scat_tuning_table - 1)) && (comm_size > mv2_red_scat_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_red_scat_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_red_scat_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_red_scat_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Red_scat_function = mv2_red_scat_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Red_scat_function; mpi_errno = MV2_Red_scat_function(sendbuf, recvbuf, recvcnts, datatype, op, comm); } else { int is_block_regular = 1; for (i = 0; i < (comm_size - 1); ++i) { if (recvcnts[i] != recvcnts[i+1]) { is_block_regular = 0; break; } } int pof2 = 1; while (pof2 < comm_size) pof2 <<= 1; if (pof2 == comm_size && is_block_regular) { /* noncommutative, pof2 size, and block regular */ MPIR_Reduce_scatter_non_comm_MV2(sendbuf, recvbuf, recvcnts, datatype, op, comm); } mpi_errno = Coll_reduce_scatter_mpich_rdb::reduce_scatter(sendbuf, recvbuf, recvcnts, datatype, op, comm); } delete[] disps; return mpi_errno; } int Coll_scatter_mvapich2::scatter(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int range = 0, range_threshold = 0, range_threshold_intra = 0; int mpi_errno = MPI_SUCCESS; // int mpi_errno_ret = MPI_SUCCESS; int rank, nbytes, comm_size; int recvtype_size, sendtype_size; int partial_sub_ok = 0; int conf_index = 0; int local_size = -1; int i; MPI_Comm shmem_comm; // MPID_Comm *shmem_commptr=NULL; if(mv2_scatter_thresholds_table==NULL) init_mv2_scatter_tables_stampede(); if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } comm_size = comm->size(); rank = comm->rank(); if (rank == root) { sendtype_size=sendtype->size(); nbytes = sendcnt * sendtype_size; } else { recvtype_size=recvtype->size(); nbytes = recvcnt * recvtype_size; } // check if safe to use partial subscription mode if (comm->is_uniform()) { shmem_comm = comm->get_intra_comm(); local_size = shmem_comm->size(); i = 0; if (mv2_scatter_table_ppn_conf[0] == -1) { // Indicating user defined tuning conf_index = 0; }else{ do { if (local_size == mv2_scatter_table_ppn_conf[i]) { conf_index = i; partial_sub_ok = 1; break; } i++; } while(i < mv2_scatter_num_ppn_conf); } } if (partial_sub_ok != 1) { conf_index = 0; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_scatter_tuning_table[conf_index] - 1)) && (comm_size > mv2_scatter_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_scatter_thresholds_table[conf_index][range].size_inter_table - 1)) && (nbytes > mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold].max) && (mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_scatter_thresholds_table[conf_index][range].size_intra_table - 1)) && (nbytes > mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra].max) && (mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Scatter_function = mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold] .MV2_pt_Scatter_function; if(MV2_Scatter_function == &MPIR_Scatter_mcst_wrap_MV2) { #if defined(_MCST_SUPPORT_) if(comm->ch.is_mcast_ok == 1 && mv2_use_mcast_scatter == 1 && comm->ch.shmem_coll_ok == 1) { MV2_Scatter_function = &MPIR_Scatter_mcst_MV2; } else #endif /*#if defined(_MCST_SUPPORT_) */ { if(mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold + 1]. MV2_pt_Scatter_function != NULL) { MV2_Scatter_function = mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold + 1] .MV2_pt_Scatter_function; } else { /* Fallback! */ MV2_Scatter_function = &MPIR_Scatter_MV2_Binomial; } } } if( (MV2_Scatter_function == &MPIR_Scatter_MV2_two_level_Direct) || (MV2_Scatter_function == &MPIR_Scatter_MV2_two_level_Binomial)) { if( comm->is_blocked()) { MV2_Scatter_intra_function = mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra] .MV2_pt_Scatter_function; mpi_errno = MV2_Scatter_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { mpi_errno = MPIR_Scatter_MV2_Binomial(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } } else { mpi_errno = MV2_Scatter_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } return (mpi_errno); } } } void smpi_coll_cleanup_mvapich2() { if (mv2_alltoall_thresholds_table) delete[] mv2_alltoall_thresholds_table[0]; delete[] mv2_alltoall_thresholds_table; delete[] mv2_size_alltoall_tuning_table; delete[] mv2_alltoall_table_ppn_conf; delete[] mv2_gather_thresholds_table; if (mv2_allgather_thresholds_table) delete[] mv2_allgather_thresholds_table[0]; delete[] mv2_size_allgather_tuning_table; delete[] mv2_allgather_table_ppn_conf; delete[] mv2_allgather_thresholds_table; delete[] mv2_allgatherv_thresholds_table; delete[] mv2_reduce_thresholds_table; delete[] mv2_red_scat_thresholds_table; delete[] mv2_allreduce_thresholds_table; delete[] mv2_bcast_thresholds_table; if (mv2_scatter_thresholds_table) delete[] mv2_scatter_thresholds_table[0]; delete[] mv2_scatter_thresholds_table; delete[] mv2_size_scatter_tuning_table; delete[] mv2_scatter_table_ppn_conf; } SimGrid-3.18/src/smpi/colls/gather/0000755000175000017500000000000013217757320017443 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/gather/gather-mvapich.cpp0000644000175000017500000004037413217757320023056 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" #include #define MPIR_Gather_MV2_Direct Coll_gather_ompi_basic_linear::gather #define MPIR_Gather_MV2_two_level_Direct Coll_gather_ompi_basic_linear::gather #define MPIR_Gather_intra Coll_gather_mpich::gather typedef int (*MV2_Gather_function_ptr) (void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm); extern MV2_Gather_function_ptr MV2_Gather_inter_leader_function; extern MV2_Gather_function_ptr MV2_Gather_intra_node_function; #define TEMP_BUF_HAS_NO_DATA (0) #define TEMP_BUF_HAS_DATA (1) namespace simgrid{ namespace smpi{ /* sendbuf - (in) sender's buffer * sendcnt - (in) sender's element count * sendtype - (in) sender's data type * recvbuf - (in) receiver's buffer * recvcnt - (in) receiver's element count * recvtype - (in) receiver's data type * root - (in)root for the gather operation * rank - (in) global rank(rank in the global comm) * tmp_buf - (out/in) tmp_buf into which intra node * data is gathered * is_data_avail - (in) based on this, tmp_buf acts * as in/out parameter. * 1 - tmp_buf acts as in parameter * 0 - tmp_buf acts as out parameter * comm_ptr - (in) pointer to the communicator * (shmem_comm or intra_sock_comm or * inter-sock_leader_comm) * intra_node_fn_ptr - (in) Function ptr to choose the * intra node gather function * errflag - (out) to record errors */ static int MPIR_pt_pt_intra_gather( void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, int rank, void *tmp_buf, int nbytes, int is_data_avail, MPI_Comm comm, MV2_Gather_function_ptr intra_node_fn_ptr) { int mpi_errno = MPI_SUCCESS; MPI_Aint recvtype_extent = 0; /* Datatype extent */ MPI_Aint true_lb, sendtype_true_extent, recvtype_true_extent; if (sendtype != MPI_DATATYPE_NULL) { sendtype->extent(&true_lb, &sendtype_true_extent); } if (recvtype != MPI_DATATYPE_NULL) { recvtype_extent=recvtype->get_extent(); recvtype->extent(&true_lb, &recvtype_true_extent); } /* Special case, when tmp_buf itself has data */ if (rank == root && sendbuf == MPI_IN_PLACE && is_data_avail) { mpi_errno = intra_node_fn_ptr(MPI_IN_PLACE, sendcnt, sendtype, tmp_buf, nbytes, MPI_BYTE, 0, comm); } else if (rank == root && sendbuf == MPI_IN_PLACE) { mpi_errno = intra_node_fn_ptr((char*)recvbuf + rank * recvcnt * recvtype_extent, recvcnt, recvtype, tmp_buf, nbytes, MPI_BYTE, 0, comm); } else { mpi_errno = intra_node_fn_ptr(sendbuf, sendcnt, sendtype, tmp_buf, nbytes, MPI_BYTE, 0, comm); } return mpi_errno; } int Coll_gather_mvapich2_two_level::gather(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { void *leader_gather_buf = NULL; int comm_size, rank; int local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = 0; int mpi_errno = MPI_SUCCESS; int recvtype_size = 0, sendtype_size = 0, nbytes=0; int leader_root, leader_of_root; MPI_Status status; MPI_Aint sendtype_extent = 0, recvtype_extent = 0; /* Datatype extent */ MPI_Aint true_lb = 0, sendtype_true_extent = 0, recvtype_true_extent = 0; MPI_Comm shmem_comm, leader_comm; void* tmp_buf = NULL; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Gather_intra_node_function==NULL) MV2_Gather_intra_node_function= Coll_gather_mpich::gather; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } comm_size = comm->size(); rank = comm->rank(); if (((rank == root) && (recvcnt == 0)) || ((rank != root) && (sendcnt == 0))) { return MPI_SUCCESS; } if (sendtype != MPI_DATATYPE_NULL) { sendtype_extent=sendtype->get_extent(); sendtype_size=sendtype->size(); sendtype->extent(&true_lb, &sendtype_true_extent); } if (recvtype != MPI_DATATYPE_NULL) { recvtype_extent=recvtype->get_extent(); recvtype_size=recvtype->size(); recvtype->extent(&true_lb, &recvtype_true_extent); } /* extract the rank,size information for the intra-node * communicator */ shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader * communicator */ leader_comm = comm->get_leaders_comm(); if(leader_comm==MPI_COMM_NULL){ leader_comm = MPI_COMM_WORLD; } leader_comm_size = leader_comm->size(); leader_comm_rank = leader_comm->size(); } if (rank == root) { nbytes = recvcnt * recvtype_size; } else { nbytes = sendcnt * sendtype_size; } #if defined(_SMP_LIMIC_) if((g_use_limic2_coll) && (shmem_commptr->ch.use_intra_sock_comm == 1) && (use_limic_gather) &&((num_scheme == USE_GATHER_PT_PT_BINOMIAL) || (num_scheme == USE_GATHER_PT_PT_DIRECT) ||(num_scheme == USE_GATHER_PT_LINEAR_BINOMIAL) || (num_scheme == USE_GATHER_PT_LINEAR_DIRECT) || (num_scheme == USE_GATHER_LINEAR_PT_BINOMIAL) || (num_scheme == USE_GATHER_LINEAR_PT_DIRECT) || (num_scheme == USE_GATHER_LINEAR_LINEAR) || (num_scheme == USE_GATHER_SINGLE_LEADER))) { mpi_errno = MV2_Gather_intra_node_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt,recvtype, root, comm); } else #endif/*#if defined(_SMP_LIMIC_)*/ { if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ if (rank == root) { tmp_buf = smpi_get_tmp_recvbuffer(recvcnt * std::max(recvtype_extent, recvtype_true_extent) * local_size); } else { tmp_buf = smpi_get_tmp_sendbuffer(sendcnt * std::max(sendtype_extent, sendtype_true_extent) * local_size); } if (tmp_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } /*while testing mpich2 gather test, we see that * which basically splits the comm, and we come to * a point, where use_intra_sock_comm == 0, but if the * intra node function is MPIR_Intra_node_LIMIC_Gather_MV2, * it would use the intra sock comm. In such cases, we * fallback to binomial as a default case.*/ #if defined(_SMP_LIMIC_) if(*MV2_Gather_intra_node_function == MPIR_Intra_node_LIMIC_Gather_MV2) { mpi_errno = MPIR_pt_pt_intra_gather(sendbuf,sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, rank, tmp_buf, nbytes, TEMP_BUF_HAS_NO_DATA, shmem_commptr, MPIR_Gather_intra); } else #endif { /*We are gathering the data into tmp_buf and the output * will be of MPI_BYTE datatype. Since the tmp_buf has no * local data, we pass is_data_avail = TEMP_BUF_HAS_NO_DATA*/ mpi_errno = MPIR_pt_pt_intra_gather(sendbuf,sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, rank, tmp_buf, nbytes, TEMP_BUF_HAS_NO_DATA, shmem_comm, MV2_Gather_intra_node_function ); } } leader_comm = comm->get_leaders_comm(); int* leaders_map = comm->get_leaders_map(); leader_of_root = comm->group()->rank(leaders_map[root]); leader_root = leader_comm->group()->rank(leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if (not comm->is_uniform()) { if (local_rank == 0) { int* displs = NULL; int* recvcnts = NULL; int* node_sizes; int i = 0; /* Node leaders have all the data. But, different nodes can have * different number of processes. Do a Gather first to get the * buffer lengths at each leader, followed by a Gatherv to move * the actual data */ if (leader_comm_rank == leader_root && root != leader_of_root) { /* The root of the Gather operation is not a node-level * leader and this process's rank in the leader_comm * is the same as leader_root */ if (rank == root) { leader_gather_buf = smpi_get_tmp_recvbuffer(recvcnt * std::max(recvtype_extent, recvtype_true_extent) * comm_size); } else { leader_gather_buf = smpi_get_tmp_sendbuffer(sendcnt * std::max(sendtype_extent, sendtype_true_extent) * comm_size); } if (leader_gather_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } node_sizes = comm->get_non_uniform_map(); if (leader_comm_rank == leader_root) { displs = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); recvcnts = static_cast(xbt_malloc(sizeof(int) * leader_comm_size)); if (not displs || not recvcnts) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } if (root == leader_of_root) { /* The root of the gather operation is also the node * leader. Receive into recvbuf and we are done */ if (leader_comm_rank == leader_root) { recvcnts[0] = node_sizes[0] * recvcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * recvcnt; recvcnts[i] = node_sizes[i] * recvcnt; } } Colls::gatherv(tmp_buf, local_size * nbytes, MPI_BYTE, recvbuf, recvcnts, displs, recvtype, leader_root, leader_comm); } else { /* The root of the gather operation is not the node leader. * Receive into leader_gather_buf and then send * to the root */ if (leader_comm_rank == leader_root) { recvcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; recvcnts[i] = node_sizes[i] * nbytes; } } Colls::gatherv(tmp_buf, local_size * nbytes, MPI_BYTE, leader_gather_buf, recvcnts, displs, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(recvcnts); } } } else { /* All nodes have the same number of processes. * Just do one Gather to get all * the data at the leader of the root process */ if (local_rank == 0) { if (leader_comm_rank == leader_root && root != leader_of_root) { /* The root of the Gather operation is not a node-level leader */ leader_gather_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); if (leader_gather_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } if (root == leader_of_root) { mpi_errno = MPIR_Gather_MV2_Direct(tmp_buf, nbytes * local_size, MPI_BYTE, recvbuf, recvcnt * local_size, recvtype, leader_root, leader_comm); } else { mpi_errno = MPIR_Gather_MV2_Direct(tmp_buf, nbytes * local_size, MPI_BYTE, leader_gather_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { Request::send(leader_gather_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_GATHER, comm); } if (rank == root && local_rank != 0) { /* The root of the gather operation is not the node leader. Receive y* data from the node leader */ Request::recv(recvbuf, recvcnt * comm_size, recvtype, leader_of_root, COLL_TAG_GATHER, comm, &status); } /* check if multiple threads are calling this collective function */ if (local_rank == 0 ) { if (tmp_buf != NULL) { smpi_free_tmp_buffer(tmp_buf); } if (leader_gather_buf != NULL) { smpi_free_tmp_buffer(leader_gather_buf); } } return (mpi_errno); } } } SimGrid-3.18/src/smpi/colls/gather/gather-ompi.cpp0000644000175000017500000003247713217757320022400 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_gather_ompi_binomial::gather(void* sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int line = -1; int i; int rank; int vrank; int size; int total_recv = 0; char *ptmp = NULL; char *tempbuf = NULL; int err; ompi_coll_tree_t* bmtree; MPI_Status status; MPI_Aint sextent, slb, strue_lb, strue_extent; MPI_Aint rextent, rlb, rtrue_lb, rtrue_extent; size = comm->size(); rank = comm->rank(); XBT_DEBUG("smpi_coll_tuned_gather_ompi_binomial rank %d", rank); /* create the binomial tree */ // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); bmtree = ompi_coll_tuned_topo_build_in_order_bmtree(comm, root); // data->cached_in_order_bmtree; sdtype->extent(&slb, &sextent); sdtype->extent(&strue_lb, &strue_extent); vrank = (rank - root + size) % size; if (rank == root) { rdtype->extent(&rlb, &rextent); rdtype->extent(&rtrue_lb, &rtrue_extent); if (0 == root) { /* root on 0, just use the recv buffer */ ptmp = (char*)rbuf; if (sbuf != MPI_IN_PLACE) { err = Datatype::copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } else { /* root is not on 0, allocate temp buffer for recv, * rotate data at the end */ tempbuf = (char*)smpi_get_tmp_recvbuffer(rtrue_extent + (rcount * size - 1) * rextent); if (NULL == tempbuf) { err = MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - rlb; if (sbuf != MPI_IN_PLACE) { /* copy from sbuf to temp buffer */ err = Datatype::copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } else { /* copy from rbuf to temp buffer */ err = Datatype::copy((char*)rbuf + rank * rextent * rcount, rcount, rdtype, ptmp, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } total_recv = rcount; } else if (!(vrank % 2)) { /* other non-leaf nodes, allocate temp buffer for data received from * children, the most we need is half of the total data elements due * to the property of binimoal tree */ tempbuf = (char*)smpi_get_tmp_sendbuffer(strue_extent + (scount * size - 1) * sextent); if (NULL == tempbuf) { err = MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - slb; /* local copy to tempbuf */ err = Datatype::copy(sbuf, scount, sdtype, ptmp, scount, sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* use sdtype,scount as rdtype,rdcount since they are ignored on * non-root procs */ rdtype = sdtype; rcount = scount; rextent = sextent; total_recv = rcount; } else { /* leaf nodes, no temp buffer needed, use sdtype,scount as * rdtype,rdcount since they are ignored on non-root procs */ ptmp = (char*)sbuf; total_recv = scount; } if (!(vrank % 2)) { /* all non-leaf nodes recv from children */ for (i = 0; i < bmtree->tree_nextsize; i++) { int mycount = 0, vkid; /* figure out how much data I have to send to this child */ vkid = (bmtree->tree_next[i] - root + size) % size; mycount = vkid - vrank; if (mycount > (size - vkid)) mycount = size - vkid; mycount *= rcount; XBT_DEBUG("smpi_coll_tuned_gather_ompi_binomial rank %d recv %d mycount = %d", rank, bmtree->tree_next[i], mycount); Request::recv(ptmp + total_recv * rextent, mycount, rdtype, bmtree->tree_next[i], COLL_TAG_GATHER, comm, &status); total_recv += mycount; } } if (rank != root) { /* all nodes except root send to parents */ XBT_DEBUG("smpi_coll_tuned_gather_ompi_binomial rank %d send %d count %d\n", rank, bmtree->tree_prev, total_recv); Request::send(ptmp, total_recv, sdtype, bmtree->tree_prev, COLL_TAG_GATHER, comm); } if (rank == root) { if (root != 0) { /* rotate received data on root if root != 0 */ err = Datatype::copy(ptmp, rcount * (size - root), rdtype, (char*)rbuf + rextent * root * rcount, rcount * (size - root), rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = Datatype::copy(ptmp + rextent * rcount * (size - root), rcount * root, rdtype, (char*)rbuf, rcount * root, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } smpi_free_tmp_buffer(tempbuf); } } else if (!(vrank % 2)) { /* other non-leaf nodes */ smpi_free_tmp_buffer(tempbuf); } ompi_coll_tuned_topo_destroy_tree(&bmtree); return MPI_SUCCESS; err_hndl: if (NULL != tempbuf) smpi_free_tmp_buffer(tempbuf); XBT_DEBUG("%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } /* * gather_intra_linear_sync * * Function: - synchronized gather operation with * Accepts: - same arguments as MPI_Gather(), first segment size * Returns: - MPI_SUCCESS or error code */ int Coll_gather_ompi_linear_sync::gather(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int i; int ret, line; int rank, size; int first_segment_count; size_t typelng; MPI_Aint extent; MPI_Aint lb; int first_segment_size=0; size = comm->size(); rank = comm->rank(); size_t dsize, block_size; if (rank == root) { dsize= rdtype->size(); block_size = dsize * rcount; } else { dsize=sdtype->size(); block_size = dsize * scount; } if (block_size > 92160){ first_segment_size = 32768; }else{ first_segment_size = 1024; } XBT_DEBUG("smpi_coll_tuned_gather_ompi_linear_sync rank %d, segment %d", rank, first_segment_size); if (rank != root) { /* Non-root processes: - receive zero byte message from the root, - send the first segment of the data synchronously, - send the second segment of the data. */ typelng = sdtype->size(); sdtype->extent(&lb, &extent); first_segment_count = scount; COLL_TUNED_COMPUTED_SEGCOUNT((size_t)first_segment_size, typelng, first_segment_count); Request::recv(sbuf, 0, MPI_BYTE, root, COLL_TAG_GATHER, comm, MPI_STATUS_IGNORE); Request::send(sbuf, first_segment_count, sdtype, root, COLL_TAG_GATHER, comm); Request::send((char*)sbuf + extent * first_segment_count, (scount - first_segment_count), sdtype, root, COLL_TAG_GATHER, comm); } else { /* Root process, - For every non-root node: - post irecv for the first segment of the message - send zero byte message to signal node to send the message - post irecv for the second segment of the message - wait for the first segment to complete - Copy local data if necessary - Waitall for all the second segments to complete. */ char* ptmp; MPI_Request first_segment_req; MPI_Request* reqs = new (std::nothrow) MPI_Request[size]; if (NULL == reqs) { ret = -1; line = __LINE__; goto error_hndl; } typelng=rdtype->size(); rdtype->extent(&lb, &extent); first_segment_count = rcount; COLL_TUNED_COMPUTED_SEGCOUNT( (size_t)first_segment_size, typelng, first_segment_count ); for (i = 0; i < size; ++i) { if (i == rank) { /* skip myself */ reqs[i] = MPI_REQUEST_NULL; continue; } /* irecv for the first segment from i */ ptmp = (char*)rbuf + i * rcount * extent; first_segment_req = Request::irecv(ptmp, first_segment_count, rdtype, i, COLL_TAG_GATHER, comm ); /* send sync message */ Request::send(rbuf, 0, MPI_BYTE, i, COLL_TAG_GATHER, comm); /* irecv for the second segment */ ptmp = (char*)rbuf + (i * rcount + first_segment_count) * extent; reqs[i]=Request::irecv(ptmp, (rcount - first_segment_count), rdtype, i, COLL_TAG_GATHER, comm ); /* wait on the first segment to complete */ Request::wait(&first_segment_req, MPI_STATUS_IGNORE); } /* copy local data if necessary */ if (MPI_IN_PLACE != sbuf) { ret = Datatype::copy(sbuf, scount, sdtype, (char*)rbuf + rank * rcount * extent, rcount, rdtype); if (ret != MPI_SUCCESS) { line = __LINE__; goto error_hndl; } } /* wait all second segments to complete */ ret = Request::waitall(size, reqs, MPI_STATUSES_IGNORE); if (ret != MPI_SUCCESS) { line = __LINE__; goto error_hndl; } delete[] reqs; } /* All done */ return MPI_SUCCESS; error_hndl: XBT_DEBUG( "ERROR_HNDL: node %d file %s line %d error %d\n", rank, __FILE__, line, ret ); return ret; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * JPG following the examples from other coll_tuned implementations. Dec06. */ /* copied function (with appropriate renaming) starts here */ /* * gather_intra * * Function: - basic gather operation * Accepts: - same arguments as MPI_Gather() * Returns: - MPI_SUCCESS or error code */ int Coll_gather_ompi_basic_linear::gather(void* sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int i; int err; int rank; int size; char *ptmp; MPI_Aint incr; MPI_Aint extent; MPI_Aint lb; size = comm->size(); rank = comm->rank(); /* Everyone but root sends data and returns. */ XBT_DEBUG("ompi_coll_tuned_gather_intra_basic_linear rank %d", rank); if (rank != root) { Request::send(sbuf, scount, sdtype, root, COLL_TAG_GATHER, comm); return MPI_SUCCESS; } /* I am the root, loop receiving the data. */ rdtype->extent(&lb, &extent); incr = extent * rcount; for (i = 0, ptmp = (char *) rbuf; i < size; ++i, ptmp += incr) { if (i == rank) { if (MPI_IN_PLACE != sbuf) { err = Datatype::copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); } else { err = MPI_SUCCESS; } } else { Request::recv(ptmp, rcount, rdtype, i, COLL_TAG_GATHER, comm, MPI_STATUS_IGNORE); err = MPI_SUCCESS; } if (MPI_SUCCESS != err) { return err; } } /* All done */ return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/smpi_default_selector.cpp0000644000175000017500000003022613217757320023254 0ustar mquinsonmquinson/* selector with default/naive Simgrid algorithms. These should not be trusted for performance evaluations */ /* Copyright (c) 2009-2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.hpp" #include "smpi_process.hpp" namespace simgrid{ namespace smpi{ int Coll_bcast_default::bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { return Coll_bcast_binomial_tree::bcast(buf, count, datatype, root, comm); } int Coll_barrier_default::barrier(MPI_Comm comm) { return Coll_barrier_ompi_basic_linear::barrier(comm); } int Coll_gather_default::gather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_GATHER; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = comm->rank(); int size = comm->size(); if(rank != root) { // Send buffer to root Request::send(sendbuf, sendcount, sendtype, root, system_tag, comm); } else { recvtype->extent(&lb, &recvext); // Local copy from root Datatype::copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + root * recvcount * recvext, recvcount, recvtype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int src = 0; src < size; src++) { if(src != root) { requests[index] = Request::irecv_init(static_cast(recvbuf) + src * recvcount * recvext, recvcount, recvtype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. Request::startall(size - 1, requests); Request::waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int src = 0; src < size-1; src++) { Request::unref(&requests[src]); } xbt_free(requests); } return MPI_SUCCESS; } int Coll_reduce_scatter_default::reduce_scatter(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank = comm->rank(); /* arbitrarily choose root as rank 0 */ int size = comm->size(); int count = 0; int *displs = xbt_new(int, size); for (int i = 0; i < size; i++) { displs[i] = count; count += recvcounts[i]; } void *tmpbuf = static_cast(smpi_get_tmp_sendbuffer(count*datatype->get_extent())); int ret = MPI_SUCCESS; ret = Coll_reduce_default::reduce(sendbuf, tmpbuf, count, datatype, op, 0, comm); if(ret==MPI_SUCCESS) ret = Colls::scatterv(tmpbuf, recvcounts, displs, datatype, recvbuf, recvcounts[rank], datatype, 0, comm); xbt_free(displs); smpi_free_tmp_buffer(tmpbuf); return ret; } int Coll_allgather_default::allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf,int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLGATHER; MPI_Aint lb = 0; MPI_Aint recvext = 0; MPI_Request *requests; int rank = comm->rank(); int size = comm->size(); // FIXME: check for errors recvtype->extent(&lb, &recvext); // Local copy from self Datatype::copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + rank * recvcount * recvext, recvcount, recvtype); // Send/Recv buffers to/from others; requests = xbt_new(MPI_Request, 2 * (size - 1)); int index = 0; for (int other = 0; other < size; other++) { if(other != rank) { requests[index] = Request::isend_init(sendbuf, sendcount, sendtype, other, system_tag,comm); index++; requests[index] = Request::irecv_init(static_cast(recvbuf) + other * recvcount * recvext, recvcount, recvtype, other, system_tag, comm); index++; } } // Wait for completion of all comms. Request::startall(2 * (size - 1), requests); Request::waitall(2 * (size - 1), requests, MPI_STATUS_IGNORE); for (int other = 0; other < 2*(size-1); other++) { Request::unref(&requests[other]); } xbt_free(requests); return MPI_SUCCESS; } int Coll_allgatherv_default::allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLGATHERV; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = comm->rank(); int size = comm->size(); recvtype->extent(&lb, &recvext); // Local copy from self Datatype::copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + displs[rank] * recvext,recvcounts[rank], recvtype); // Send buffers to others; MPI_Request *requests = xbt_new(MPI_Request, 2 * (size - 1)); int index = 0; for (int other = 0; other < size; other++) { if(other != rank) { requests[index] = Request::isend_init(sendbuf, sendcount, sendtype, other, system_tag, comm); index++; requests[index] = Request::irecv_init(static_cast(recvbuf) + displs[other] * recvext, recvcounts[other], recvtype, other, system_tag, comm); index++; } } // Wait for completion of all comms. Request::startall(2 * (size - 1), requests); Request::waitall(2 * (size - 1), requests, MPI_STATUS_IGNORE); for (int other = 0; other < 2*(size-1); other++) { Request::unref(&requests[other]); } xbt_free(requests); return MPI_SUCCESS; } int Coll_scatter_default::scatter(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_SCATTER; MPI_Aint lb = 0; MPI_Aint sendext = 0; MPI_Request *requests; int rank = comm->rank(); int size = comm->size(); if(rank != root) { // Recv buffer from root Request::recv(recvbuf, recvcount, recvtype, root, system_tag, comm, MPI_STATUS_IGNORE); } else { sendtype->extent(&lb, &sendext); // Local copy from root if(recvbuf!=MPI_IN_PLACE){ Datatype::copy(static_cast(sendbuf) + root * sendcount * sendext, sendcount, sendtype, recvbuf, recvcount, recvtype); } // Send buffers to receivers requests = xbt_new(MPI_Request, size - 1); int index = 0; for(int dst = 0; dst < size; dst++) { if(dst != root) { requests[index] = Request::isend_init(static_cast(sendbuf) + dst * sendcount * sendext, sendcount, sendtype, dst, system_tag, comm); index++; } } // Wait for completion of isend's. Request::startall(size - 1, requests); Request::waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int dst = 0; dst < size-1; dst++) { Request::unref(&requests[dst]); } xbt_free(requests); } return MPI_SUCCESS; } int Coll_reduce_default::reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int system_tag = COLL_TAG_REDUCE; MPI_Aint lb = 0; MPI_Aint dataext = 0; char* sendtmpbuf = static_cast(sendbuf); int rank = comm->rank(); int size = comm->size(); if(size==0) return MPI_ERR_COMM; //non commutative case, use a working algo from openmpi if (op != MPI_OP_NULL && not op->is_commutative()) { return Coll_reduce_ompi_basic_linear::reduce(sendtmpbuf, recvbuf, count, datatype, op, root, comm); } if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = static_cast(smpi_get_tmp_sendbuffer(count*datatype->get_extent())); Datatype::copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); } if(rank != root) { // Send buffer to root Request::send(sendtmpbuf, count, datatype, root, system_tag, comm); } else { datatype->extent(&lb, &dataext); // Local copy from root if (sendtmpbuf != nullptr && recvbuf != nullptr) Datatype::copy(sendtmpbuf, count, datatype, recvbuf, count, datatype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, size - 1); int index = 0; for (int src = 0; src < size; src++) { if (src != root) { if (not smpi_process()->replaying()) tmpbufs[index] = xbt_malloc(count * dataext); else tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = Request::irecv_init(tmpbufs[index], count, datatype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. Request::startall(size - 1, requests); for (int src = 0; src < size - 1; src++) { index = Request::waitany(size - 1, requests, MPI_STATUS_IGNORE); XBT_DEBUG("finished waiting any request with index %d", index); if(index == MPI_UNDEFINED) { break; }else{ Request::unref(&requests[index]); } if(op) /* op can be MPI_OP_NULL that does nothing */ if(op!=MPI_OP_NULL) op->apply( tmpbufs[index], recvbuf, &count, datatype); } for(index = 0; index < size - 1; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } xbt_free(tmpbufs); xbt_free(requests); } if( sendbuf == MPI_IN_PLACE ) { smpi_free_tmp_buffer(sendtmpbuf); } return MPI_SUCCESS; } int Coll_allreduce_default::allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int ret; ret = Colls::reduce(sendbuf, recvbuf, count, datatype, op, 0, comm); if(ret==MPI_SUCCESS) ret = Colls::bcast(recvbuf, count, datatype, 0, comm); return ret; } int Coll_alltoall_default::alltoall( void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm) { return Coll_alltoall_ompi::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } int Coll_alltoallv_default::alltoallv(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = 889; int i; int count; MPI_Aint lb = 0; MPI_Aint sendext = 0; MPI_Aint recvext = 0; MPI_Request *requests; /* Initialize. */ int rank = comm->rank(); int size = comm->size(); XBT_DEBUG("<%d> algorithm basic_alltoallv() called.", rank); sendtype->extent(&lb, &sendext); recvtype->extent(&lb, &recvext); /* Local copy from self */ int err = Datatype::copy(static_cast(sendbuf) + senddisps[rank] * sendext, sendcounts[rank], sendtype, static_cast(recvbuf) + recvdisps[rank] * recvext, recvcounts[rank], recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ requests = xbt_new(MPI_Request, 2 * (size - 1)); count = 0; /* Create all receives that will be posted first */ for (i = 0; i < size; ++i) { if (i != rank && recvcounts[i] != 0) { requests[count] = Request::irecv_init(static_cast(recvbuf) + recvdisps[i] * recvext, recvcounts[i], recvtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [src = %d, recvcounts[src] = %d]", rank, i, recvcounts[i]); } } /* Now create all sends */ for (i = 0; i < size; ++i) { if (i != rank && sendcounts[i] != 0) { requests[count] = Request::isend_init(static_cast(sendbuf) + senddisps[i] * sendext, sendcounts[i], sendtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [dst = %d, sendcounts[dst] = %d]", rank, i, sendcounts[i]); } } /* Wait for them all. */ Request::startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); Request::waitall(count, requests, MPI_STATUS_IGNORE); for(i = 0; i < count; i++) { if(requests[i]!=MPI_REQUEST_NULL) Request::unref(&requests[i]); } xbt_free(requests); } return err; } } } SimGrid-3.18/src/smpi/colls/smpi_mvapich2_selector_stampede.hpp0000644000175000017500000016710213217757321025235 0ustar mquinsonmquinson/* selector for collective algorithms based on mvapich decision logic, with calibration from Stampede cluster at TACC*/ /* This is the tuning used by MVAPICH for Stampede platform based on (MV2_ARCH_INTEL_XEON_E5_2680_16, * MV2_HCA_MLX_CX_FDR) */ /* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /************ Alltoall variables and initializers */ #ifndef SMPI_MVAPICH2_SELECTOR_STAMPEDE_HPP #define SMPI_MVAPICH2_SELECTOR_STAMPEDE_HPP #include #define MV2_MAX_NB_THRESHOLDS 32 XBT_PUBLIC(void) smpi_coll_cleanup_mvapich2(void); struct mv2_alltoall_tuning_element { int min; int max; int (*MV2_pt_Alltoall_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr); }; struct mv2_alltoall_tuning_table { int numproc; int size_table; mv2_alltoall_tuning_element algo_table[MV2_MAX_NB_THRESHOLDS]; mv2_alltoall_tuning_element in_place_algo_table[MV2_MAX_NB_THRESHOLDS]; }; int (*MV2_Alltoall_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr) = NULL; /* Indicates number of processes per node */ int* mv2_alltoall_table_ppn_conf = NULL; /* Indicates total number of configurations */ int mv2_alltoall_num_ppn_conf = 1; int* mv2_size_alltoall_tuning_table = NULL; mv2_alltoall_tuning_table** mv2_alltoall_thresholds_table = NULL; #define MPIR_Alltoall_bruck_MV2 simgrid::smpi::Coll_alltoall_bruck::alltoall #define MPIR_Alltoall_RD_MV2 simgrid::smpi::Coll_alltoall_rdb::alltoall #define MPIR_Alltoall_Scatter_dest_MV2 simgrid::smpi::Coll_alltoall_mvapich2_scatter_dest::alltoall #define MPIR_Alltoall_pairwise_MV2 simgrid::smpi::Coll_alltoall_pair::alltoall #define MPIR_Alltoall_inplace_MV2 simgrid::smpi::Coll_alltoall_ring::alltoall static void init_mv2_alltoall_tables_stampede() { int agg_table_sum = 0; mv2_alltoall_tuning_table** table_ptrs = NULL; mv2_alltoall_num_ppn_conf = 3; if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_alltoall_thresholds_table = new mv2_alltoall_tuning_table*[mv2_alltoall_num_ppn_conf]; table_ptrs = new mv2_alltoall_tuning_table*[mv2_alltoall_num_ppn_conf]; mv2_size_alltoall_tuning_table = new int[mv2_alltoall_num_ppn_conf]; mv2_alltoall_table_ppn_conf = new int[mv2_alltoall_num_ppn_conf]; mv2_alltoall_table_ppn_conf[0] = 1; mv2_size_alltoall_tuning_table[0] = 6; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_1ppn[] = { { 2, 1, { {0, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 4, 2, { {0, 262144, &MPIR_Alltoall_Scatter_dest_MV2}, {262144, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 8, 2, { {0, 8, &MPIR_Alltoall_RD_MV2}, {8, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 16, 3, { {0, 64, &MPIR_Alltoall_RD_MV2}, {64, 512, &MPIR_Alltoall_bruck_MV2}, {512, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 32, 3, { {0, 32, &MPIR_Alltoall_RD_MV2}, {32, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 64, 3, { {0, 8, &MPIR_Alltoall_RD_MV2}, {8, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[0] = mv2_tmp_alltoall_thresholds_table_1ppn; mv2_alltoall_table_ppn_conf[1] = 2; mv2_size_alltoall_tuning_table[1] = 6; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_2ppn[] = { { 4, 2, { {0, 32, &MPIR_Alltoall_RD_MV2}, {32, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 8, 2, { {0, 64, &MPIR_Alltoall_RD_MV2}, {64, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 16, 3, { {0, 64, &MPIR_Alltoall_RD_MV2}, {64, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 32, 3, { {0, 16, &MPIR_Alltoall_RD_MV2}, {16, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 64, 3, { {0, 8, &MPIR_Alltoall_RD_MV2}, {8, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 128, 3, { {0, 4, &MPIR_Alltoall_RD_MV2}, {4, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[1] = mv2_tmp_alltoall_thresholds_table_2ppn; mv2_alltoall_table_ppn_conf[2] = 16; mv2_size_alltoall_tuning_table[2] = 7; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_16ppn[] = { { 16, 2, { {0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {32768, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 32, 2, { {0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, { {16384, -1, &MPIR_Alltoall_inplace_MV2}, }, }, { 64, 3, { {0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, 16384, &MPIR_Alltoall_Scatter_dest_MV2}, {16384, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {32768, 131072, &MPIR_Alltoall_inplace_MV2}, }, }, { 128, 2, { {0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, { 256, 2, { {0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, { 512, 2, { {0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, { 1024, 2, { {0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, { {16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[2] = mv2_tmp_alltoall_thresholds_table_16ppn; agg_table_sum = 0; for (int i = 0; i < mv2_alltoall_num_ppn_conf; i++) { agg_table_sum += mv2_size_alltoall_tuning_table[i]; } mv2_alltoall_thresholds_table[0] = new mv2_alltoall_tuning_table[agg_table_sum]; std::copy_n(table_ptrs[0], mv2_size_alltoall_tuning_table[0], mv2_alltoall_thresholds_table[0]); for (int i = 1; i < mv2_alltoall_num_ppn_conf; i++) { mv2_alltoall_thresholds_table[i] = mv2_alltoall_thresholds_table[i - 1] + mv2_size_alltoall_tuning_table[i - 1]; std::copy_n(table_ptrs[i], mv2_size_alltoall_tuning_table[i], mv2_alltoall_thresholds_table[i]); } delete[] table_ptrs; } /************ Allgather variables and initializers */ struct mv2_allgather_tuning_element { int min; int max; int (*MV2_pt_Allgatherction)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr); }; struct mv2_allgather_tuning_table { int numproc; int two_level[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_allgather_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; }; int (*MV2_Allgatherction)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); int* mv2_allgather_table_ppn_conf = NULL; int mv2_allgather_num_ppn_conf = 1; int* mv2_size_allgather_tuning_table = NULL; mv2_allgather_tuning_table** mv2_allgather_thresholds_table = NULL; static int MPIR_Allgather_RD_Allgather_Comm_MV2(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr) { return 0; } #define MPIR_Allgather_Bruck_MV2 simgrid::smpi::Coll_allgather_bruck::allgather #define MPIR_Allgather_RD_MV2 simgrid::smpi::Coll_allgather_rdb::allgather #define MPIR_Allgather_Ring_MV2 simgrid::smpi::Coll_allgather_ring::allgather #define MPIR_2lvl_Allgather_MV2 simgrid::smpi::Coll_allgather_mvapich2_smp::allgather static void init_mv2_allgather_tables_stampede() { int agg_table_sum = 0; if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_allgather_tuning_table** table_ptrs = NULL; mv2_allgather_num_ppn_conf = 3; mv2_allgather_thresholds_table = new mv2_allgather_tuning_table*[mv2_allgather_num_ppn_conf]; table_ptrs = new mv2_allgather_tuning_table*[mv2_allgather_num_ppn_conf]; mv2_size_allgather_tuning_table = new int[mv2_allgather_num_ppn_conf]; mv2_allgather_table_ppn_conf = new int[mv2_allgather_num_ppn_conf]; mv2_allgather_table_ppn_conf[0] = 1; mv2_size_allgather_tuning_table[0] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_1ppn[] = { { 2, {0}, 1, { {0, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 4, {0, 0}, 2, { {0, 262144, &MPIR_Allgather_RD_MV2}, {262144, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 8, {0, 0}, 2, { {0, 131072, &MPIR_Allgather_RD_MV2}, {131072, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 16, {0, 0}, 2, { {0, 131072, &MPIR_Allgather_RD_MV2}, {131072, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {0, 0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {0, 0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[0] = mv2_tmp_allgather_thresholds_table_1ppn; mv2_allgather_table_ppn_conf[1] = 2; mv2_size_allgather_tuning_table[1] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_2ppn[] = { { 4, {0, 0}, 2, { {0, 524288, &MPIR_Allgather_RD_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 8, {0, 1, 0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 16, {0, 1, 0}, 2, { {0, 16384, &MPIR_Allgather_RD_MV2}, {16384, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {1, 1, 0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {1, 1, 0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 128, {1, 1, 0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[1] = mv2_tmp_allgather_thresholds_table_2ppn; mv2_allgather_table_ppn_conf[2] = 16; mv2_size_allgather_tuning_table[2] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_16ppn[] = { { 16, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 128, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 256, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 512, {0, 0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[2] = mv2_tmp_allgather_thresholds_table_16ppn; agg_table_sum = 0; for (int i = 0; i < mv2_allgather_num_ppn_conf; i++) { agg_table_sum += mv2_size_allgather_tuning_table[i]; } mv2_allgather_thresholds_table[0] = new mv2_allgather_tuning_table[agg_table_sum]; std::copy_n(table_ptrs[0], mv2_size_allgather_tuning_table[0], mv2_allgather_thresholds_table[0]); for (int i = 1; i < mv2_allgather_num_ppn_conf; i++) { mv2_allgather_thresholds_table[i] = mv2_allgather_thresholds_table[i - 1] + mv2_size_allgather_tuning_table[i - 1]; std::copy_n(table_ptrs[i], mv2_size_allgather_tuning_table[i], mv2_allgather_thresholds_table[i]); } delete[] table_ptrs; } /************ Gather variables and initializers */ struct mv2_gather_tuning_element { int min; int max; int (*MV2_pt_Gather_function)(void* sendbuf, int sendcnt, MPI_Datatype sendtype, void* recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr); }; struct mv2_gather_tuning_table { int numproc; int size_inter_table; mv2_gather_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_gather_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; }; int mv2_size_gather_tuning_table = 7; mv2_gather_tuning_table* mv2_gather_thresholds_table = NULL; typedef int (*MV2_Gather_function_ptr)(void* sendbuf, int sendcnt, MPI_Datatype sendtype, void* recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm); MV2_Gather_function_ptr MV2_Gather_inter_leader_function = NULL; MV2_Gather_function_ptr MV2_Gather_intra_node_function = NULL; #define MPIR_Gather_MV2_Direct simgrid::smpi::Coll_gather_ompi_basic_linear::gather #define MPIR_Gather_MV2_two_level_Direct simgrid::smpi::Coll_gather_mvapich2_two_level::gather #define MPIR_Gather_intra simgrid::smpi::Coll_gather_mpich::gather static void init_mv2_gather_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_size_gather_tuning_table = 7; mv2_gather_thresholds_table = new mv2_gather_tuning_table[mv2_size_gather_tuning_table]; mv2_gather_tuning_table mv2_tmp_gather_thresholds_table[] = { {16, 2, {{0, 524288, &MPIR_Gather_MV2_Direct}, {524288, -1, &MPIR_Gather_intra}}, 1, {{0, -1, &MPIR_Gather_MV2_Direct}}}, {32, 3, {{0, 16384, &MPIR_Gather_MV2_Direct}, {16384, 131072, &MPIR_Gather_intra}, {131072, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, {64, 3, {{0, 256, &MPIR_Gather_MV2_two_level_Direct}, {256, 16384, &MPIR_Gather_MV2_Direct}, {256, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, {128, 3, {{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {16384, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, {256, 3, {{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {16384, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, {512, 3, {{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {8196, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, {1024, 3, {{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {8196, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1, {{0, -1, &MPIR_Gather_intra}}}, }; std::copy_n(mv2_tmp_gather_thresholds_table, mv2_size_gather_tuning_table, mv2_gather_thresholds_table); } /************ Allgatherv variables and initializers */ struct mv2_allgatherv_tuning_element { int min; int max; int (*MV2_pt_Allgatherv_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int* recvcounts, int* displs, MPI_Datatype recvtype, MPI_Comm commg); }; struct mv2_allgatherv_tuning_table { int numproc; int size_inter_table; mv2_allgatherv_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; }; int (*MV2_Allgatherv_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int* recvcounts, int* displs, MPI_Datatype recvtype, MPI_Comm comm); int mv2_size_allgatherv_tuning_table = 0; mv2_allgatherv_tuning_table* mv2_allgatherv_thresholds_table = NULL; #define MPIR_Allgatherv_Rec_Doubling_MV2 simgrid::smpi::Coll_allgatherv_mpich_rdb::allgatherv #define MPIR_Allgatherv_Bruck_MV2 simgrid::smpi::Coll_allgatherv_ompi_bruck::allgatherv #define MPIR_Allgatherv_Ring_MV2 simgrid::smpi::Coll_allgatherv_mpich_ring::allgatherv static void init_mv2_allgatherv_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_size_allgatherv_tuning_table = 6; mv2_allgatherv_thresholds_table = new mv2_allgatherv_tuning_table[mv2_size_allgatherv_tuning_table]; mv2_allgatherv_tuning_table mv2_tmp_allgatherv_thresholds_table[] = { { 16, 2, { {0, 512, &MPIR_Allgatherv_Rec_Doubling_MV2}, {512, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 32, 2, { {0, 512, &MPIR_Allgatherv_Rec_Doubling_MV2}, {512, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 64, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 128, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 256, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 512, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, }; std::copy_n(mv2_tmp_allgatherv_thresholds_table, mv2_size_allgatherv_tuning_table, mv2_allgatherv_thresholds_table); } /************ Allreduce variables and initializers */ struct mv2_allreduce_tuning_element { int min; int max; int (*MV2_pt_Allreducection)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); }; struct mv2_allreduce_tuning_table { int numproc; int mcast_enabled; int is_two_level_allreduce[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_allreduce_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_allreduce_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; }; int (*MV2_Allreducection)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) = NULL; int (*MV2_Allreduce_intra_function)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) = NULL; int mv2_size_allreduce_tuning_table = 0; mv2_allreduce_tuning_table* mv2_allreduce_thresholds_table = NULL; static int MPIR_Allreduce_mcst_reduce_two_level_helper_MV2(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { return 0; } static int MPIR_Allreduce_mcst_reduce_redscat_gather_MV2(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { return 0; } static int MPIR_Allreduce_reduce_p2p_MV2(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { simgrid::smpi::Colls::reduce(sendbuf, recvbuf, count, datatype, op, 0, comm); return MPI_SUCCESS; } static int MPIR_Allreduce_reduce_shmem_MV2(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { simgrid::smpi::Colls::reduce(sendbuf, recvbuf, count, datatype, op, 0, comm); return MPI_SUCCESS; } #define MPIR_Allreduce_pt2pt_rd_MV2 simgrid::smpi::Coll_allreduce_rdb::allreduce #define MPIR_Allreduce_pt2pt_rs_MV2 simgrid::smpi::Coll_allreduce_mvapich2_rs::allreduce #define MPIR_Allreduce_two_level_MV2 simgrid::smpi::Coll_allreduce_mvapich2_two_level::allreduce static void init_mv2_allreduce_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_size_allreduce_tuning_table = 8; mv2_allreduce_thresholds_table = new mv2_allreduce_tuning_table[mv2_size_allreduce_tuning_table]; mv2_allreduce_tuning_table mv2_tmp_allreduce_thresholds_table[] = { { 16, 0, {1, 0}, 2, { {0, 1024, &MPIR_Allreduce_pt2pt_rd_MV2}, {1024, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 1024, &MPIR_Allreduce_reduce_shmem_MV2}, {1024, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 32, 0, {1, 1, 0}, 3, { {0, 1024, &MPIR_Allreduce_pt2pt_rd_MV2}, {1024, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 1024, &MPIR_Allreduce_reduce_shmem_MV2}, {1024, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 64, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 128, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 256, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 512, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 1024, 0, {1, 1, 1, 0}, 4, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 8192, &MPIR_Allreduce_pt2pt_rd_MV2}, {8192, 65536, &MPIR_Allreduce_pt2pt_rs_MV2}, {65536, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 2048, 0, {1, 1, 1, 0}, 4, { {0, 64, &MPIR_Allreduce_pt2pt_rd_MV2}, {64, 512, &MPIR_Allreduce_reduce_p2p_MV2}, {512, 4096, &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2}, {4096, 16384, &MPIR_Allreduce_pt2pt_rs_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, }; std::copy_n(mv2_tmp_allreduce_thresholds_table, mv2_size_allreduce_tuning_table, mv2_allreduce_thresholds_table); } struct mv2_bcast_tuning_element { int min; int max; int (*MV2_pt_Bcast_function)(void* buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); int zcpy_pipelined_knomial_factor; }; struct mv2_bcast_tuning_table { int numproc; int bcast_segment_size; int intra_node_knomial_factor; int inter_node_knomial_factor; int is_two_level_bcast[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_bcast_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_bcast_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; }; int mv2_size_bcast_tuning_table = 0; mv2_bcast_tuning_table* mv2_bcast_thresholds_table = NULL; int (*MV2_Bcast_function)(void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr) = NULL; int (*MV2_Bcast_intra_node_function)(void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr) = NULL; int zcpy_knomial_factor = 2; int mv2_pipelined_zcpy_knomial_factor = -1; int bcast_segment_size = 8192; int mv2_inter_node_knomial_factor = 4; int mv2_intra_node_knomial_factor = 4; #define mv2_bcast_two_level_system_size 64 #define mv2_bcast_short_msg 16384 #define mv2_bcast_large_msg 512 * 1024 #define INTRA_NODE_ROOT 0 #define MPIR_Pipelined_Bcast_Zcpy_MV2 simgrid::smpi::Coll_bcast_mpich::bcast #define MPIR_Pipelined_Bcast_MV2 simgrid::smpi::Coll_bcast_mpich::bcast #define MPIR_Bcast_binomial_MV2 simgrid::smpi::Coll_bcast_binomial_tree::bcast #define MPIR_Bcast_scatter_ring_allgather_shm_MV2 simgrid::smpi::Coll_bcast_scatter_LR_allgather::bcast #define MPIR_Bcast_scatter_doubling_allgather_MV2 simgrid::smpi::Coll_bcast_scatter_rdb_allgather::bcast #define MPIR_Bcast_scatter_ring_allgather_MV2 simgrid::smpi::Coll_bcast_scatter_LR_allgather::bcast #define MPIR_Shmem_Bcast_MV2 simgrid::smpi::Coll_bcast_mpich::bcast #define MPIR_Bcast_tune_inter_node_helper_MV2 simgrid::smpi::Coll_bcast_mvapich2_inter_node::bcast #define MPIR_Bcast_inter_node_helper_MV2 simgrid::smpi::Coll_bcast_mvapich2_inter_node::bcast #define MPIR_Knomial_Bcast_intra_node_MV2 simgrid::smpi::Coll_bcast_mvapich2_knomial_intra_node::bcast #define MPIR_Bcast_intra_MV2 simgrid::smpi::Coll_bcast_mvapich2_intra_node::bcast static void init_mv2_bcast_tables_stampede() { // Stampede, if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_size_bcast_tuning_table = 8; mv2_bcast_thresholds_table = new mv2_bcast_tuning_table[mv2_size_bcast_tuning_table]; mv2_bcast_tuning_table mv2_tmp_bcast_thresholds_table[] = { {16, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 11, {{0, 8, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {8, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16, 1024, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {1024, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {8192, 16384, &MPIR_Bcast_binomial_MV2, -1}, {16384, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {32768, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {65536, 131072, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}, {131072, 262144, &MPIR_Bcast_scatter_ring_allgather_MV2, -1}, {262144, 524288, &MPIR_Bcast_scatter_doubling_allgather_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_MV2, -1}}, 11, {{0, 8, &MPIR_Shmem_Bcast_MV2, 2}, {8, 16, &MPIR_Shmem_Bcast_MV2, 4}, {16, 1024, &MPIR_Shmem_Bcast_MV2, 2}, {1024, 8192, &MPIR_Shmem_Bcast_MV2, 4}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, -1}, {16384, 32768, &MPIR_Shmem_Bcast_MV2, 4}, {32768, 65536, &MPIR_Shmem_Bcast_MV2, 2}, {65536, 131072, &MPIR_Shmem_Bcast_MV2, -1}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1}}}, {32, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1}, 8, {{0, 128, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {128, 256, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {256, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {32768, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {65536, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {262144, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}}, 8, {{0, 128, &MPIR_Shmem_Bcast_MV2, 2}, {128, 256, &MPIR_Shmem_Bcast_MV2, 4}, {256, 32768, &MPIR_Shmem_Bcast_MV2, 2}, {32768, 65536, &MPIR_Shmem_Bcast_MV2, 4}, {65536, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, 8}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, &MPIR_Shmem_Bcast_MV2, 8}}}, {64, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1, 1}, 9, {{0, 2, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {2, 4, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16, 32, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {32, 128, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {128, 256, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {256, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {4096, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {32768, -1, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}}, 9, {{0, 2, &MPIR_Shmem_Bcast_MV2, 4}, {2, 4, &MPIR_Shmem_Bcast_MV2, 8}, {4, 16, &MPIR_Shmem_Bcast_MV2, 4}, {16, 32, &MPIR_Shmem_Bcast_MV2, 8}, {32, 128, &MPIR_Shmem_Bcast_MV2, 4}, {128, 256, &MPIR_Shmem_Bcast_MV2, 8}, {256, 4096, &MPIR_Shmem_Bcast_MV2, 4}, {4096, 32768, &MPIR_Shmem_Bcast_MV2, 8}, {32768, -1, &MPIR_Shmem_Bcast_MV2, 2}}}, {128, 8192, 4, 4, {1, 1, 1, 0}, 4, {{0, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {8192, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_MV2, -1}}, 4, {{0, 8192, &MPIR_Shmem_Bcast_MV2, 8}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, NULL, -1}}}, {256, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, {{0, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}, {262144, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}}, 5, {{0, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1}}}, {512, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, {{0, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4096, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Pipelined_Bcast_MV2, -1}, {262144, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}}, 5, {{0, 4096, &MPIR_Shmem_Bcast_MV2, 8}, {4096, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, -1, &MPIR_Shmem_Bcast_MV2, -1}}}, {1024, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, {{0, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {8192, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {65536, 524288, &MPIR_Pipelined_Bcast_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}}, 5, {{0, 8192, &MPIR_Shmem_Bcast_MV2, 8}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 65536, &MPIR_Shmem_Bcast_MV2, 2}, {65536, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1}}}, {2048, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1}, 7, {{0, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {16, 32, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {32, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4096, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {32768, 524288, &MPIR_Pipelined_Bcast_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}}, 7, {{0, 16, &MPIR_Shmem_Bcast_MV2, 8}, {16, 32, &MPIR_Shmem_Bcast_MV2, 4}, {32, 4096, &MPIR_Shmem_Bcast_MV2, 8}, {4096, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 32768, &MPIR_Shmem_Bcast_MV2, 2}, {32768, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1}}}}; std::copy_n(mv2_tmp_bcast_thresholds_table, mv2_size_bcast_tuning_table, mv2_bcast_thresholds_table); } /************ Reduce variables and initializers */ struct mv2_reduce_tuning_element { int min; int max; int (*MV2_pt_Reduce_function)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); }; struct mv2_reduce_tuning_table { int numproc; int inter_k_degree; int intra_k_degree; int is_two_level_reduce[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_reduce_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_reduce_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; }; int mv2_size_reduce_tuning_table = 0; mv2_reduce_tuning_table* mv2_reduce_thresholds_table = NULL; int mv2_reduce_intra_knomial_factor = -1; int mv2_reduce_inter_knomial_factor = -1; int (*MV2_Reduce_function)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr) = NULL; int (*MV2_Reduce_intra_function)(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr) = NULL; #define MPIR_Reduce_inter_knomial_wrapper_MV2 simgrid::smpi::Coll_reduce_mvapich2_knomial::reduce #define MPIR_Reduce_intra_knomial_wrapper_MV2 simgrid::smpi::Coll_reduce_mvapich2_knomial::reduce #define MPIR_Reduce_binomial_MV2 simgrid::smpi::Coll_reduce_binomial::reduce #define MPIR_Reduce_redscat_gather_MV2 simgrid::smpi::Coll_reduce_scatter_gather::reduce #define MPIR_Reduce_shmem_MV2 simgrid::smpi::Coll_reduce_ompi_basic_linear::reduce #define MPIR_Reduce_two_level_helper_MV2 simgrid::smpi::Coll_reduce_mvapich2_two_level::reduce static void init_mv2_reduce_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; /*Stampede*/ mv2_size_reduce_tuning_table = 8; mv2_reduce_thresholds_table = new mv2_reduce_tuning_table[mv2_size_reduce_tuning_table]; mv2_reduce_tuning_table mv2_tmp_reduce_thresholds_table[] = { { 16, 4, 4, {1, 0, 0}, 3, { {0, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 2, { {0, 65536, &MPIR_Reduce_shmem_MV2}, {65536, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 32, 4, 4, {1, 1, 1, 1, 0, 0, 0}, 7, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {32768, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 6, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_shmem_MV2}, {32768, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_shmem_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 64, 4, 4, {1, 1, 1, 1, 0}, 5, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 128, 4, 4, {1, 0, 1, 0, 1, 0}, 6, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 256, 4, 4, {1, 1, 1, 0, 1, 1, 0}, 7, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_binomial_MV2}, {32768, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 6, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_shmem_MV2}, {32768, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 512, 4, 4, {1, 0, 1, 1, 1, 0}, 6, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 1024, 4, 4, {1, 0, 1, 1, 1}, 5, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 2048, 4, 4, {1, 0, 1, 1, 1, 1}, 6, { {0, 2048, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {2048, 4096, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {4096, 16384, &MPIR_Reduce_binomial_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 131072, &MPIR_Reduce_binomial_MV2}, {131072, -1, &MPIR_Reduce_binomial_MV2}, }, 6, { {0, 2048, &MPIR_Reduce_shmem_MV2}, {2048, 4096, &MPIR_Reduce_shmem_MV2}, {4096, 16384, &MPIR_Reduce_shmem_MV2}, {16384, 65536, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {65536, 131072, &MPIR_Reduce_binomial_MV2}, {131072, -1, &MPIR_Reduce_shmem_MV2}, }, }, }; std::copy_n(mv2_tmp_reduce_thresholds_table, mv2_size_reduce_tuning_table, mv2_reduce_thresholds_table); } /************ Reduce scatter variables and initializers */ struct mv2_red_scat_tuning_element { int min; int max; int (*MV2_pt_Red_scat_function)(void* sendbuf, void* recvbuf, int* recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm_ptr); }; struct mv2_red_scat_tuning_table { int numproc; int size_inter_table; mv2_red_scat_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; }; int mv2_size_red_scat_tuning_table = 0; mv2_red_scat_tuning_table* mv2_red_scat_thresholds_table = NULL; int (*MV2_Red_scat_function)(void* sendbuf, void* recvbuf, int* recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm_ptr); static int MPIR_Reduce_Scatter_Basic_MV2(void* sendbuf, void* recvbuf, int* recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { simgrid::smpi::Coll_reduce_scatter_default::reduce_scatter(sendbuf, recvbuf, recvcnts, datatype, op, comm); return MPI_SUCCESS; } #define MPIR_Reduce_scatter_non_comm_MV2 simgrid::smpi::Coll_reduce_scatter_mpich_noncomm::reduce_scatter #define MPIR_Reduce_scatter_Rec_Halving_MV2 \ simgrid::smpi::Coll_reduce_scatter_ompi_basic_recursivehalving::reduce_scatter #define MPIR_Reduce_scatter_Pair_Wise_MV2 simgrid::smpi::Coll_reduce_scatter_mpich_pair::reduce_scatter static void init_mv2_reduce_scatter_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; mv2_size_red_scat_tuning_table = 6; mv2_red_scat_thresholds_table = new mv2_red_scat_tuning_table[mv2_size_red_scat_tuning_table]; mv2_red_scat_tuning_table mv2_tmp_red_scat_thresholds_table[] = { { 16, 3, { {0, 64, &MPIR_Reduce_Scatter_Basic_MV2}, {64, 65536, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {65536, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 32, 3, { {0, 64, &MPIR_Reduce_Scatter_Basic_MV2}, {64, 131072, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {131072, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 64, 3, { {0, 1024, &MPIR_Reduce_Scatter_Basic_MV2}, {1024, 262144, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {262144, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 128, 2, { {0, 128, &MPIR_Reduce_Scatter_Basic_MV2}, {128, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, { 256, 2, { {0, 128, &MPIR_Reduce_Scatter_Basic_MV2}, {128, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, { 512, 2, { {0, 256, &MPIR_Reduce_Scatter_Basic_MV2}, {256, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, }; std::copy_n(mv2_tmp_red_scat_thresholds_table, mv2_size_red_scat_tuning_table, mv2_red_scat_thresholds_table); } /************ Scatter variables and initializers */ struct mv2_scatter_tuning_element { int min; int max; int (*MV2_pt_Scatter_function)(void* sendbuf, int sendcnt, MPI_Datatype sendtype, void* recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm); }; struct mv2_scatter_tuning_table { int numproc; int size_inter_table; mv2_scatter_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_scatter_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; }; int* mv2_scatter_table_ppn_conf = NULL; int mv2_scatter_num_ppn_conf = 1; int* mv2_size_scatter_tuning_table = NULL; mv2_scatter_tuning_table** mv2_scatter_thresholds_table = NULL; int (*MV2_Scatter_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) = NULL; int (*MV2_Scatter_intra_function)(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) = NULL; int MPIR_Scatter_mcst_wrap_MV2(void* sendbuf, int sendcnt, MPI_Datatype sendtype, void* recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr); int MPIR_Scatter_mcst_wrap_MV2(void* sendbuf, int sendcnt, MPI_Datatype sendtype, void* recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr) { return 0; } #define MPIR_Scatter_MV2_Binomial simgrid::smpi::Coll_scatter_ompi_binomial::scatter #define MPIR_Scatter_MV2_Direct simgrid::smpi::Coll_scatter_ompi_basic_linear::scatter #define MPIR_Scatter_MV2_two_level_Binomial simgrid::smpi::Coll_scatter_mvapich2_two_level_binomial::scatter #define MPIR_Scatter_MV2_two_level_Direct simgrid::smpi::Coll_scatter_mvapich2_two_level_direct::scatter static void init_mv2_scatter_tables_stampede() { if (simgrid::smpi::Colls::smpi_coll_cleanup_callback == NULL) simgrid::smpi::Colls::smpi_coll_cleanup_callback = &smpi_coll_cleanup_mvapich2; int agg_table_sum = 0; mv2_scatter_tuning_table** table_ptrs = NULL; mv2_scatter_num_ppn_conf = 3; mv2_scatter_thresholds_table = new mv2_scatter_tuning_table*[mv2_scatter_num_ppn_conf]; table_ptrs = new mv2_scatter_tuning_table*[mv2_scatter_num_ppn_conf]; mv2_size_scatter_tuning_table = new int[mv2_scatter_num_ppn_conf]; mv2_scatter_table_ppn_conf = new int[mv2_scatter_num_ppn_conf]; mv2_scatter_table_ppn_conf[0] = 1; mv2_size_scatter_tuning_table[0] = 6; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_1ppn[] = { { 2, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 4, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 8, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 16, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 32, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 64, 2, { {0, 32, &MPIR_Scatter_MV2_Binomial}, {32, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, }; table_ptrs[0] = mv2_tmp_scatter_thresholds_table_1ppn; mv2_scatter_table_ppn_conf[1] = 2; mv2_size_scatter_tuning_table[1] = 6; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_2ppn[] = { { 4, 2, { {0, 4096, &MPIR_Scatter_MV2_Binomial}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 8, 2, { {0, 512, &MPIR_Scatter_MV2_two_level_Direct}, {512, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 16, 2, { {0, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 32, 2, { {0, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 64, 2, { {0, 8192, &MPIR_Scatter_MV2_two_level_Direct}, {8192, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 128, 4, { {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 128, &MPIR_Scatter_MV2_two_level_Binomial}, {128, 16384, &MPIR_Scatter_MV2_two_level_Direct}, {16384, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, 128, &MPIR_Scatter_MV2_Direct}, {128, -1, &MPIR_Scatter_MV2_Binomial}, }, }, }; table_ptrs[1] = mv2_tmp_scatter_thresholds_table_2ppn; mv2_scatter_table_ppn_conf[2] = 16; mv2_size_scatter_tuning_table[2] = 8; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_16ppn[] = { { 16, 2, { {0, 256, &MPIR_Scatter_MV2_Binomial}, {256, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 32, 2, { {0, 512, &MPIR_Scatter_MV2_Binomial}, {512, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 64, 2, { {0, 1024, &MPIR_Scatter_MV2_two_level_Direct}, {1024, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 128, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 256, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 512, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {16, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 4096, &MPIR_Scatter_MV2_two_level_Direct}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 1024, 5, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 32, &MPIR_Scatter_MV2_Binomial}, {32, 4096, &MPIR_Scatter_MV2_two_level_Direct}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 2048, 7, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Binomial}, {16, 128, &MPIR_Scatter_MV2_two_level_Binomial}, {128, 1024, &MPIR_Scatter_MV2_two_level_Direct}, {1024, 16384, &MPIR_Scatter_MV2_two_level_Direct}, {16384, 65536, &MPIR_Scatter_MV2_Direct}, {65536, -1, &MPIR_Scatter_MV2_two_level_Direct}, }, 6, { {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 128, &MPIR_Scatter_MV2_Binomial}, {128, 1024, &MPIR_Scatter_MV2_Binomial}, {1024, 16384, &MPIR_Scatter_MV2_Direct}, {16384, 65536, &MPIR_Scatter_MV2_Direct}, {65536, -1, &MPIR_Scatter_MV2_Direct}, }, }, }; table_ptrs[2] = mv2_tmp_scatter_thresholds_table_16ppn; agg_table_sum = 0; for (int i = 0; i < mv2_scatter_num_ppn_conf; i++) { agg_table_sum += mv2_size_scatter_tuning_table[i]; } mv2_scatter_thresholds_table[0] = new mv2_scatter_tuning_table[agg_table_sum]; std::copy_n(table_ptrs[0], mv2_size_scatter_tuning_table[0], mv2_scatter_thresholds_table[0]); for (int i = 1; i < mv2_scatter_num_ppn_conf; i++) { mv2_scatter_thresholds_table[i] = mv2_scatter_thresholds_table[i - 1] + mv2_size_scatter_tuning_table[i - 1]; std::copy_n(table_ptrs[i], mv2_size_scatter_tuning_table[i], mv2_scatter_thresholds_table[i]); } delete[] table_ptrs; } #endif SimGrid-3.18/src/smpi/colls/smpi_coll.cpp0000644000175000017500000003064013217757320020661 0ustar mquinsonmquinson/* smpi_coll.c -- various optimized routing for collectives */ /* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_coll.hpp" #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" #include "smpi_request.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_coll, smpi, "Logging specific to SMPI (coll)"); #define COLL_SETTER(cat, ret, args, args2) \ int(*Colls::cat) args; \ void Colls::set_##cat(std::string name) \ { \ int id = find_coll_description(mpi_coll_##cat##_description, name, #cat); \ cat = reinterpret_cast(mpi_coll_##cat##_description[id].coll); \ if (cat == nullptr) \ xbt_die("Collective " #cat " set to nullptr!"); \ } #define SET_COLL(coll) \ name = xbt_cfg_get_string("smpi/" #coll); \ if (name.empty()) \ name = selector_name; \ set_##coll(name); namespace simgrid{ namespace smpi{ void (*Colls::smpi_coll_cleanup_callback)(); /* these arrays must be nullptr terminated */ s_mpi_coll_description_t Colls::mpi_coll_gather_description[] = { COLL_GATHERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_allgather_description[] = { COLL_ALLGATHERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_allgatherv_description[] = { COLL_ALLGATHERVS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_allreduce_description[] ={ COLL_ALLREDUCES(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_reduce_scatter_description[] = { COLL_REDUCE_SCATTERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_scatter_description[] ={ COLL_SCATTERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_barrier_description[] ={ COLL_BARRIERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_alltoall_description[] = { COLL_ALLTOALLS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_alltoallv_description[] = { COLL_ALLTOALLVS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_bcast_description[] = { COLL_BCASTS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; s_mpi_coll_description_t Colls::mpi_coll_reduce_description[] = { COLL_REDUCES(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; /** Displays the long description of all registered models, and quit */ void Colls::coll_help(const char *category, s_mpi_coll_description_t * table) { XBT_WARN("Long description of the %s models accepted by this simulator:\n", category); for (int i = 0; table[i].name; i++) XBT_WARN(" %s: %s\n", table[i].name, table[i].description); } int Colls::find_coll_description(s_mpi_coll_description_t* table, std::string name, const char* desc) { for (int i = 0; table[i].name; i++) if (name == table[i].name) { if (strcmp(table[i].name,"default")) XBT_INFO("Switch to algorithm %s for collective %s",table[i].name,desc); return i; } if (not table[0].name) xbt_die("No collective is valid for '%s'! This is a bug.", name.c_str()); std::string name_list = std::string(table[0].name); for (int i = 1; table[i].name; i++) name_list = name_list + ", " + table[i].name; xbt_die("Collective '%s' is invalid! Valid collectives are: %s.", name.c_str(), name_list.c_str()); return -1; } COLL_APPLY(COLL_SETTER,COLL_GATHER_SIG,""); COLL_APPLY(COLL_SETTER,COLL_ALLGATHER_SIG,""); COLL_APPLY(COLL_SETTER,COLL_ALLGATHERV_SIG,""); COLL_APPLY(COLL_SETTER,COLL_REDUCE_SIG,""); COLL_APPLY(COLL_SETTER,COLL_ALLREDUCE_SIG,""); COLL_APPLY(COLL_SETTER,COLL_REDUCE_SCATTER_SIG,""); COLL_APPLY(COLL_SETTER,COLL_SCATTER_SIG,""); COLL_APPLY(COLL_SETTER,COLL_BARRIER_SIG,""); COLL_APPLY(COLL_SETTER,COLL_BCAST_SIG,""); COLL_APPLY(COLL_SETTER,COLL_ALLTOALL_SIG,""); COLL_APPLY(COLL_SETTER,COLL_ALLTOALLV_SIG,""); void Colls::set_collectives(){ std::string selector_name = xbt_cfg_get_string("smpi/coll-selector"); if (selector_name.empty()) selector_name = "default"; std::string name; SET_COLL(gather); SET_COLL(allgather); SET_COLL(allgatherv); SET_COLL(allreduce); SET_COLL(alltoall); SET_COLL(alltoallv); SET_COLL(reduce); SET_COLL(reduce_scatter); SET_COLL(scatter); SET_COLL(bcast); SET_COLL(barrier); } //Implementations of the single algorith collectives int Colls::gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_GATHERV; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = comm->rank(); int size = comm->size(); if (rank != root) { // Send buffer to root Request::send(sendbuf, sendcount, sendtype, root, system_tag, comm); } else { recvtype->extent(&lb, &recvext); // Local copy from root Datatype::copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + displs[root] * recvext, recvcounts[root], recvtype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int src = 0; src < size; src++) { if(src != root) { requests[index] = Request::irecv_init(static_cast(recvbuf) + displs[src] * recvext, recvcounts[src], recvtype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. Request::startall(size - 1, requests); Request::waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int src = 0; src < size-1; src++) { Request::unref(&requests[src]); } xbt_free(requests); } return MPI_SUCCESS; } int Colls::scatterv(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_SCATTERV; MPI_Aint lb = 0; MPI_Aint sendext = 0; int rank = comm->rank(); int size = comm->size(); if(rank != root) { // Recv buffer from root Request::recv(recvbuf, recvcount, recvtype, root, system_tag, comm, MPI_STATUS_IGNORE); } else { sendtype->extent(&lb, &sendext); // Local copy from root if(recvbuf!=MPI_IN_PLACE){ Datatype::copy(static_cast(sendbuf) + displs[root] * sendext, sendcounts[root], sendtype, recvbuf, recvcount, recvtype); } // Send buffers to receivers MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int dst = 0; dst < size; dst++) { if (dst != root) { requests[index] = Request::isend_init(static_cast(sendbuf) + displs[dst] * sendext, sendcounts[dst], sendtype, dst, system_tag, comm); index++; } } // Wait for completion of isend's. Request::startall(size - 1, requests); Request::waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int dst = 0; dst < size-1; dst++) { Request::unref(&requests[dst]); } xbt_free(requests); } return MPI_SUCCESS; } int Colls::scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int system_tag = -888; MPI_Aint lb = 0; MPI_Aint dataext = 0; int rank = comm->rank(); int size = comm->size(); datatype->extent(&lb, &dataext); // Local copy from self Datatype::copy(sendbuf, count, datatype, recvbuf, count, datatype); // Send/Recv buffers to/from others MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, rank); int index = 0; for (int other = 0; other < rank; other++) { tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = Request::irecv_init(tmpbufs[index], count, datatype, other, system_tag, comm); index++; } for (int other = rank + 1; other < size; other++) { requests[index] = Request::isend_init(sendbuf, count, datatype, other, system_tag, comm); index++; } // Wait for completion of all comms. Request::startall(size - 1, requests); if(op != MPI_OP_NULL && op->is_commutative()){ for (int other = 0; other < size - 1; other++) { index = Request::waitany(size - 1, requests, MPI_STATUS_IGNORE); if(index == MPI_UNDEFINED) { break; } if(index < rank) { // #Request is below rank: it's a irecv op->apply( tmpbufs[index], recvbuf, &count, datatype); } } }else{ //non commutative case, wait in order for (int other = 0; other < size - 1; other++) { Request::wait(&(requests[other]), MPI_STATUS_IGNORE); if(index < rank && op!=MPI_OP_NULL) { op->apply( tmpbufs[other], recvbuf, &count, datatype); } } } for(index = 0; index < rank; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } for(index = 0; index < size-1; index++) { Request::unref(&requests[index]); } xbt_free(tmpbufs); xbt_free(requests); return MPI_SUCCESS; } int Colls::exscan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int system_tag = -888; MPI_Aint lb = 0; MPI_Aint dataext = 0; int recvbuf_is_empty=1; int rank = comm->rank(); int size = comm->size(); datatype->extent(&lb, &dataext); // Send/Recv buffers to/from others MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, rank); int index = 0; for (int other = 0; other < rank; other++) { tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = Request::irecv_init(tmpbufs[index], count, datatype, other, system_tag, comm); index++; } for (int other = rank + 1; other < size; other++) { requests[index] = Request::isend_init(sendbuf, count, datatype, other, system_tag, comm); index++; } // Wait for completion of all comms. Request::startall(size - 1, requests); if(op != MPI_OP_NULL && op->is_commutative()){ for (int other = 0; other < size - 1; other++) { index = Request::waitany(size - 1, requests, MPI_STATUS_IGNORE); if(index == MPI_UNDEFINED) { break; } if(index < rank) { if(recvbuf_is_empty){ Datatype::copy(tmpbufs[index], count, datatype, recvbuf, count, datatype); recvbuf_is_empty=0; } else // #Request is below rank: it's a irecv op->apply( tmpbufs[index], recvbuf, &count, datatype); } } }else{ //non commutative case, wait in order for (int other = 0; other < size - 1; other++) { Request::wait(&(requests[other]), MPI_STATUS_IGNORE); if(index < rank) { if (recvbuf_is_empty) { Datatype::copy(tmpbufs[other], count, datatype, recvbuf, count, datatype); recvbuf_is_empty = 0; } else if(op!=MPI_OP_NULL) op->apply( tmpbufs[other], recvbuf, &count, datatype); } } } for(index = 0; index < rank; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } for(index = 0; index < size-1; index++) { Request::unref(&requests[index]); } xbt_free(tmpbufs); xbt_free(requests); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/0000755000175000017500000000000013217757317020137 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/allreduce/allreduce-mvapich-two-level.cpp0000644000175000017500000001364513217757317026155 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" #define MPIR_Allreduce_pt2pt_rd_MV2 Coll_allreduce_rdb::allreduce #define MPIR_Allreduce_pt2pt_rs_MV2 Coll_allreduce_mvapich2_rs::allreduce extern int (*MV2_Allreducection)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); extern int (*MV2_Allreduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); namespace simgrid{ namespace smpi{ static int MPIR_Allreduce_reduce_p2p_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { Colls::reduce(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } static int MPIR_Allreduce_reduce_shmem_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { Colls::reduce(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } /* general two level allreduce helper function */ int Coll_allreduce_mvapich2_two_level::allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int total_size = 0; MPI_Aint true_lb, true_extent; MPI_Comm shmem_comm = MPI_COMM_NULL, leader_comm = MPI_COMM_NULL; int local_rank = -1, local_size = 0; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Allreduce_intra_function==NULL) MV2_Allreduce_intra_function = Coll_allreduce_mpich::allreduce; if(MV2_Allreducection==NULL) MV2_Allreducection = Coll_allreduce_rdb::allreduce; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } if (count == 0) { return MPI_SUCCESS; } datatype->extent(&true_lb, &true_extent); total_size = comm->size(); shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); leader_comm = comm->get_leaders_comm(); if (local_rank == 0) { if (sendbuf != MPI_IN_PLACE) { Datatype::copy(sendbuf, count, datatype, recvbuf, count, datatype); } } /* Doing the shared memory gather and reduction by the leader */ if (local_rank == 0) { if ((MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_shmem_MV2) || (MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_p2p_MV2) ) { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, shmem_comm); } if (local_size != total_size) { void* sendtmpbuf = (char *)smpi_get_tmp_sendbuffer(count*datatype->get_extent()); Datatype::copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); /* inter-node allreduce */ if(MV2_Allreducection == &MPIR_Allreduce_pt2pt_rd_MV2){ mpi_errno = MPIR_Allreduce_pt2pt_rd_MV2(sendtmpbuf, recvbuf, count, datatype, op, leader_comm); } else { mpi_errno = MPIR_Allreduce_pt2pt_rs_MV2(sendtmpbuf, recvbuf, count, datatype, op, leader_comm); } smpi_free_tmp_buffer(sendtmpbuf); } } else { /* insert the first reduce here */ if ((MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_shmem_MV2) || (MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_p2p_MV2) ) { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, shmem_comm); } } /* Broadcasting the mesage from leader to the rest */ /* Note: shared memory broadcast could improve the performance */ mpi_errno = Colls::bcast(recvbuf, count, datatype, 0, shmem_comm); return (mpi_errno); } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-rsag-rab.cpp0000644000175000017500000001416613217757317025104 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * implemented by Pitch Patarasuk, 07/01/2007 */ #include "../colls_private.hpp" //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_rsag_rab::allreduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } comm_size = comm->size(); if((comm_size&(comm_size-1))) THROWF(arg_error,0, "allreduce smp rsag rab algorithm can't be used with non power of two number of processes ! "); rank = comm->rank(); MPI_Aint extent; extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; Request::sendrecv(sbuf, count, dtype, rank, tag, rbuf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuf, &count, dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send(rbuf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int dst, base_offset, send_base_offset, recv_base_offset, recv_chunk; int curr_count, i, recv_offset, send_offset; // reduce-scatter recv_chunk = extent * count / (comm_size / num_core); mask = 1; curr_count = count / 2; int phase = 0; base_offset = 0; while (mask < (comm_size / num_core)) { dst = inter_rank ^ mask; // compute offsets // right-handside if (inter_rank & mask) { recv_base_offset = base_offset + curr_count; send_base_offset = base_offset; base_offset = recv_base_offset; } // left-handside else { recv_base_offset = base_offset; send_base_offset = base_offset + curr_count; } send_offset = send_base_offset * extent; recv_offset = recv_base_offset * extent; // if (rank==7) // printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_offset, recv_offset, curr_count); Request::sendrecv((char *)rbuf + send_offset, curr_count, dtype, (dst * num_core), tag, tmp_buf, curr_count, dtype, (dst * num_core), tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *)rbuf + recv_offset, &curr_count, dtype); mask *= 2; curr_count /= 2; phase++; } // INTER: allgather int size = (comm_size / num_core) / 2; base_offset = 0; mask = 1; while (mask < (comm_size / num_core)) { if (inter_rank & mask) { base_offset += size; } mask <<= 1; size /= 2; } curr_count *= 2; mask >>= 1; i = 1; phase = 0; while (mask >= 1) { // destination pair for both send and recv dst = inter_rank ^ mask; // compute offsets send_base_offset = base_offset; if (inter_rank & mask) { recv_base_offset = base_offset - i; base_offset -= i; } else { recv_base_offset = base_offset + i; } send_offset = send_base_offset * recv_chunk; recv_offset = recv_base_offset * recv_chunk; // if (rank==7) //printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_offset, recv_offset, curr_count); Request::sendrecv((char *)rbuf + send_offset, curr_count, dtype, (dst * num_core), tag, (char *)rbuf + recv_offset, curr_count, dtype, (dst * num_core), tag, comm, &status); curr_count *= 2; i *= 2; mask >>= 1; phase++; } } // INTER // intra SMP binomial bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); Request::recv(rbuf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); Request::send(rbuf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-rsag.cpp0000644000175000017500000001523513217757317024340 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_rsag::allreduce(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = comm->size(); rank = comm->rank(); MPI_Aint extent; extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; if (not rank) { //printf("intra com size = %d\n",num_core); //printf("inter com size = %d\n",inter_comm_size); } Request::sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int send_offset, recv_offset; int seg_count = count / inter_comm_size; int to = ((inter_rank + 1) % inter_comm_size) * num_core; int from = ((inter_rank + inter_comm_size - 1) % inter_comm_size) * num_core; int i; //printf("node %d to %d from %d\n",rank,to,from); for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; recv_offset = ((inter_rank - 2 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; Request::sendrecv((char *) recv_buf + send_offset, seg_count, dtype, to, tag + i, tmp_buf, seg_count, dtype, from, tag + i, comm, &status); // result is in rbuf if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *) recv_buf + recv_offset, &seg_count, dtype); } // INTER: allgather for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - i + inter_comm_size) % inter_comm_size) * seg_count * extent; recv_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; Request::sendrecv((char *) recv_buf + send_offset, seg_count, dtype, to, tag + i, (char *) recv_buf + recv_offset, seg_count, dtype, from, tag + i, comm, &status); } } // INTER_binomial_reduce // only root node for each SMP // if (intra_rank == 0) { // // mask = 1; // while (mask < inter_comm_size) { // if ((mask & inter_rank) == 0) { // src = (inter_rank | mask) * num_core; // if (src < comm_size) { // Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); // (* uop) (tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // } // } // else { // dst = (inter_rank & (~mask)) * num_core; // Request::send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // break; // } // mask <<=1; // } // } // INTER_binomial_bcast // if (intra_rank == 0) { // mask = 1; // while (mask < inter_comm_size) { // if (inter_rank & mask) { // src = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // Request::recv(recv_buf, count, dtype, src, tag, comm, &status); // break; // } // mask <<= 1; // } // // mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); // while (mask > 0) { // if (inter_rank < inter_comm_size) { // dst = (inter_rank + mask) * num_core; // if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // Request::send(recv_buf, count, dtype, dst, tag, comm); // } // } // mask >>= 1; // } // } // INTRA_binomial_bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); Request::recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); Request::send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-binomial-pipeline.cpp0000644000175000017500000001576613217757317027012 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for both intra-communication inter-communication The communication are done in a pipeline fashion */ /* this is a default segment size for pipelining, but it is typically passed as a command line argument */ int allreduce_smp_binomial_pipeline_segment_size = 4096; /* ** NOTE ** This code is modified from allreduce-smp-binomial.c by wrapping the code with pipeline effect as follow for (loop over pipelength) { smp-binomial main code; } */ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ /* This fucntion performs all-reduce operation as follow. ** in a pipeline fashion ** 1) binomial_tree reduce inside each SMP node 2) binomial_tree reduce intra-communication between root of each SMP node 3) binomial_tree bcast intra-communication between root of each SMP node 4) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_binomial_pipeline::allreduce(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } comm_size = comm->size(); rank = comm->rank(); MPI_Aint extent; extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; int phase; int send_offset; int recv_offset; int pcount = allreduce_smp_binomial_pipeline_segment_size; if (pcount > count) { pcount = count; } /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ Request::sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* compute pipe length */ int pipelength; pipelength = count / pcount; /* pipelining over pipelength (+3 is because we have 4 stages: reduce-intra, reduce-inter, bcast-inter, bcast-intra */ for (phase = 0; phase < pipelength + 3; phase++) { /* start binomial reduce intra communication inside each SMP node */ if (phase < pipelength) { mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { recv_offset = phase * pcount * extent; Request::recv(tmp_buf, pcount, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *)recv_buf + recv_offset, &pcount, dtype); } } else { send_offset = phase * pcount * extent; dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); break; } mask <<= 1; } } /* start binomial reduce inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if ((phase > 0) && (phase < (pipelength + 1))) { if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { recv_offset = (phase - 1) * pcount * extent; Request::recv(tmp_buf, pcount, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *)recv_buf + recv_offset, &pcount, dtype); } } else { dst = (inter_rank & (~mask)) * num_core; send_offset = (phase - 1) * pcount * extent; Request::send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); break; } mask <<= 1; } } } /* start binomial broadcast inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if ((phase > 1) && (phase < (pipelength + 2))) { if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if (inter_rank & mask) { src = (inter_rank - mask) * num_core; recv_offset = (phase - 2) * pcount * extent; Request::recv((char *)recv_buf + recv_offset, pcount, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (inter_rank < inter_comm_size) { dst = (inter_rank + mask) * num_core; if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); send_offset = (phase - 2) * pcount * extent; Request::send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); } } mask >>= 1; } } } /* start binomial broadcast intra-communication inside each SMP nodes */ if (phase > 2) { int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); recv_offset = (phase - 3) * pcount * extent; Request::recv((char *)recv_buf + recv_offset, pcount, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { send_offset = (phase - 3) * pcount * extent; Request::send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); } mask >>= 1; } } } // for phase smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-rab2.cpp0000644000175000017500000000516213217757317023433 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include namespace simgrid{ namespace smpi{ // this requires that count >= NP int Coll_allreduce_rab2::allreduce(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { MPI_Aint s_extent; int i, rank, nprocs; int nbytes, send_size, s_offset, r_offset; void *recv, *send, *tmp; /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ rank = comm->rank(); nprocs = comm->size(); s_extent = dtype->get_extent(); // uneven count if (count % nprocs) { if (count < nprocs) send_size = nprocs; else send_size = (count + nprocs) / nprocs; nbytes = send_size * s_extent; send = (void *) smpi_get_tmp_sendbuffer(s_extent * send_size * nprocs); recv = (void *) smpi_get_tmp_recvbuffer(s_extent * send_size * nprocs); tmp = (void *) smpi_get_tmp_sendbuffer(nbytes); memcpy(send, sbuff, s_extent * count); Colls::alltoall(send, send_size, dtype, recv, send_size, dtype, comm); memcpy(tmp, recv, nbytes); for (i = 1, s_offset = nbytes; i < nprocs; i++, s_offset = i * nbytes) if(op!=MPI_OP_NULL) op->apply( (char *) recv + s_offset, tmp, &send_size, dtype); Colls::allgather(tmp, send_size, dtype, recv, send_size, dtype, comm); memcpy(rbuff, recv, count * s_extent); smpi_free_tmp_buffer(recv); smpi_free_tmp_buffer(tmp); smpi_free_tmp_buffer(send); } else { send = sbuff; send_size = count / nprocs; nbytes = send_size * s_extent; r_offset = rank * nbytes; recv = (void *) smpi_get_tmp_recvbuffer(s_extent * send_size * nprocs); Colls::alltoall(send, send_size, dtype, recv, send_size, dtype, comm); memcpy((char *) rbuff + r_offset, recv, nbytes); for (i = 1, s_offset = nbytes; i < nprocs; i++, s_offset = i * nbytes) if(op!=MPI_OP_NULL) op->apply( (char *) recv + s_offset, (char *) rbuff + r_offset, &send_size, dtype); Colls::allgather((char *) rbuff + r_offset, send_size, dtype, rbuff, send_size, dtype, comm); smpi_free_tmp_buffer(recv); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-rdb.cpp0000644000175000017500000001045613217757317023356 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include namespace simgrid{ namespace smpi{ int Coll_allreduce_rdb::allreduce(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int nprocs, rank, tag = COLL_TAG_ALLREDUCE; int mask, dst, pof2, newrank, rem, newdst; MPI_Aint extent, lb; MPI_Status status; void *tmp_buf = NULL; /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ nprocs=comm->size(); rank=comm->rank(); dtype->extent(&lb, &extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); Request::sendrecv(sbuff, count, dtype, rank, 500, rbuff, count, dtype, rank, 500, comm, &status); // find nearest power-of-two less than or equal to comm_size pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; rem = nprocs - pof2; // In the non-power-of-two case, all even-numbered // processes of rank < 2*rem send their data to // (rank+1). These even-numbered processes no longer // participate in the algorithm until the very end. The // remaining processes form a nice power-of-two. if (rank < 2 * rem) { // even if (rank % 2 == 0) { Request::send(rbuff, count, dtype, rank + 1, tag, comm); // temporarily set the rank to -1 so that this // process does not pariticipate in recursive // doubling newrank = -1; } else // odd { Request::recv(tmp_buf, count, dtype, rank - 1, tag, comm, &status); // do the reduction on received data. since the // ordering is right, it doesn't matter whether // the operation is commutative or not. if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuff, &count, dtype); // change the rank newrank = rank / 2; } } else // rank >= 2 * rem newrank = rank - rem; // If op is user-defined or count is less than pof2, use // recursive doubling algorithm. Otherwise do a reduce-scatter // followed by allgather. (If op is user-defined, // derived datatypes are allowed and the user could pass basic // datatypes on one process and derived on another as long as // the type maps are the same. Breaking up derived // datatypes to do the reduce-scatter is tricky, therefore // using recursive doubling in that case.) if (newrank != -1) { mask = 0x1; while (mask < pof2) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; // Send the most current data, which is in recvbuf. Recv // into tmp_buf Request::sendrecv(rbuff, count, dtype, dst, tag, tmp_buf, count, dtype, dst, tag, comm, &status); // tmp_buf contains data received in this step. // recvbuf contains data accumulated so far // op is commutative OR the order is already right // we assume it is commuttive op // if (op -> op_commute || (dst < rank)) if ((dst < rank)) { if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuff, &count, dtype); } else // op is noncommutative and the order is not right { if(op!=MPI_OP_NULL) op->apply( rbuff, tmp_buf, &count, dtype); // copy result back into recvbuf Request::sendrecv(tmp_buf, count, dtype, rank, tag, rbuff, count, dtype, rank, tag, comm, &status); } mask <<= 1; } } // In the non-power-of-two case, all odd-numbered processes of // rank < 2 * rem send the result to (rank-1), the ranks who didn't // participate above. if (rank < 2 * rem) { if (rank % 2) // odd Request::send(rbuff, count, dtype, rank - 1, tag, comm); else // even Request::recv(rbuff, count, dtype, rank + 1, tag, comm, &status); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-rab-rdb.cpp0000644000175000017500000001450213217757317024114 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allreduce_rab_rdb::allreduce(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int tag = COLL_TAG_ALLREDUCE; unsigned int mask, pof2, i, recv_idx, last_idx, send_idx, send_cnt; int dst, newrank, rem, newdst, recv_cnt, *cnts, *disps; MPI_Aint extent; MPI_Status status; void *tmp_buf = NULL; unsigned int nprocs = comm->size(); int rank = comm->rank(); extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); Datatype::copy(sbuff, count, dtype, rbuff, count, dtype); // find nearest power-of-two less than or equal to comm_size pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; rem = nprocs - pof2; // In the non-power-of-two case, all even-numbered // processes of rank < 2*rem send their data to // (rank+1). These even-numbered processes no longer // participate in the algorithm until the very end. The // remaining processes form a nice power-of-two. if (rank < 2 * rem) { // even if (rank % 2 == 0) { Request::send(rbuff, count, dtype, rank + 1, tag, comm); // temporarily set the rank to -1 so that this // process does not pariticipate in recursive // doubling newrank = -1; } else // odd { Request::recv(tmp_buf, count, dtype, rank - 1, tag, comm, &status); // do the reduction on received data. since the // ordering is right, it doesn't matter whether // the operation is commutative or not. if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuff, &count, dtype); // change the rank newrank = rank / 2; } } else // rank >= 2 * rem newrank = rank - rem; // If op is user-defined or count is less than pof2, use // recursive doubling algorithm. Otherwise do a reduce-scatter // followed by allgather. (If op is user-defined, // derived datatypes are allowed and the user could pass basic // datatypes on one process and derived on another as long as // the type maps are the same. Breaking up derived // datatypes to do the reduce-scatter is tricky, therefore // using recursive doubling in that case.) if (newrank != -1) { // do a reduce-scatter followed by allgather. for the // reduce-scatter, calculate the count that each process receives // and the displacement within the buffer cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } // Send data from recvbuf. Recv into tmp_buf Request::sendrecv((char *) rbuff + disps[send_idx] * extent, send_cnt, dtype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, dtype, dst, tag, comm, &status); // tmp_buf contains data received in this step. // recvbuf contains data accumulated so far // This algorithm is used only for predefined ops // and predefined ops are always commutative. if(op!=MPI_OP_NULL) op->apply( (char *) tmp_buf + disps[recv_idx] * extent, (char *) rbuff + disps[recv_idx] * extent, &recv_cnt, dtype); // update send_idx for next iteration send_idx = recv_idx; mask <<= 1; // update last_idx, but not in last iteration because the value // is needed in the allgather step below. if (mask < pof2) last_idx = recv_idx + pof2 / mask; } // now do the allgather mask >>= 1; while (mask > 0) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { // update last_idx except on first iteration if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } Request::sendrecv((char *) rbuff + disps[send_idx] * extent, send_cnt, dtype, dst, tag, (char *) rbuff + disps[recv_idx] * extent, recv_cnt, dtype, dst, tag, comm, &status); if (newrank > newdst) send_idx = recv_idx; mask >>= 1; } free(cnts); free(disps); } // In the non-power-of-two case, all odd-numbered processes of // rank < 2 * rem send the result to (rank-1), the ranks who didn't // participate above. if (rank < 2 * rem) { if (rank % 2) // odd Request::send(rbuff, count, dtype, rank - 1, tag, comm); else // even Request::recv(rbuff, count, dtype, rank + 1, tag, comm, &status); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-redbcast.cpp0000644000175000017500000000125413217757317024372 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allreduce_redbcast::allreduce(void *buf, void *buf2, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { Colls::reduce(buf, buf2, count, datatype, op, 0, comm); Colls::bcast(buf2, count, datatype, 0, comm); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-ompi-ring-segmented.cpp0000644000175000017500000004311313217757317026455 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * - Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * - Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer listed * in this license in the documentation and/or other materials * provided with the distribution. * - Neither the name of the copyright holders nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * The copyright holders provide no reassurances that the source code * provided does not infringe any patent, copyright, or any other * intellectual property rights of third parties. The copyright holders * disclaim any liability to any recipient for claims brought against * recipient by any third party for infringement of that parties * intellectual property rights. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * ompi_coll_tuned_allreduce_intra_ring_segmented * * Function: Pipelined ring algorithm for allreduce operation * Accepts: Same as MPI_Allreduce(), segment size * Returns: MPI_SUCCESS or error code * * Description: Implements pipelined ring algorithm for allreduce: * user supplies suggested segment size for the pipelining of * reduce operation. * The segment size determines the number of phases, np, for * the algorithm execution. * The message is automatically divided into blocks of * approximately (count / (np * segcount)) elements. * At the end of reduction phase, allgather like step is * executed. * Algorithm requires (np + 1)*(N - 1) steps. * * Limitations: The algorithm DOES NOT preserve order of operations so it * can be used only for commutative operations. * In addition, algorithm cannot work if the total size is * less than size * segment size. * Example on 3 nodes with 2 phases * Initial state * # 0 1 2 * [00a] [10a] [20a] * [00b] [10b] [20b] * [01a] [11a] [21a] * [01b] [11b] [21b] * [02a] [12a] [22a] * [02b] [12b] [22b] * * COMPUTATION PHASE 0 (a) * Step 0: rank r sends block ra to rank (r+1) and receives bloc (r-1)a * from rank (r-1) [with wraparound]. * # 0 1 2 * [00a] [00a+10a] [20a] * [00b] [10b] [20b] * [01a] [11a] [11a+21a] * [01b] [11b] [21b] * [22a+02a] [12a] [22a] * [02b] [12b] [22b] * * Step 1: rank r sends block (r-1)a to rank (r+1) and receives bloc * (r-2)a from rank (r-1) [with wraparound]. * # 0 1 2 * [00a] [00a+10a] [00a+10a+20a] * [00b] [10b] [20b] * [11a+21a+01a] [11a] [11a+21a] * [01b] [11b] [21b] * [22a+02a] [22a+02a+12a] [22a] * [02b] [12b] [22b] * * COMPUTATION PHASE 1 (b) * Step 0: rank r sends block rb to rank (r+1) and receives bloc (r-1)b * from rank (r-1) [with wraparound]. * # 0 1 2 * [00a] [00a+10a] [20a] * [00b] [00b+10b] [20b] * [01a] [11a] [11a+21a] * [01b] [11b] [11b+21b] * [22a+02a] [12a] [22a] * [22b+02b] [12b] [22b] * * Step 1: rank r sends block (r-1)b to rank (r+1) and receives bloc * (r-2)b from rank (r-1) [with wraparound]. * # 0 1 2 * [00a] [00a+10a] [00a+10a+20a] * [00b] [10b] [0bb+10b+20b] * [11a+21a+01a] [11a] [11a+21a] * [11b+21b+01b] [11b] [21b] * [22a+02a] [22a+02a+12a] [22a] * [02b] [22b+01b+12b] [22b] * * * DISTRIBUTION PHASE: ring ALLGATHER with ranks shifted by 1 (same as * in regular ring algorithm. * */ #define COLL_TUNED_COMPUTED_SEGCOUNT(SEGSIZE, TYPELNG, SEGCOUNT) \ if( ((SEGSIZE) >= (TYPELNG)) && \ ((SEGSIZE) < ((TYPELNG) * (SEGCOUNT))) ) { \ size_t residual; \ (SEGCOUNT) = (int)((SEGSIZE) / (TYPELNG)); \ residual = (SEGSIZE) - (SEGCOUNT) * (TYPELNG); \ if( residual > ((TYPELNG) >> 1) ) \ (SEGCOUNT)++; \ } \ #define COLL_TUNED_COMPUTE_BLOCKCOUNT( COUNT, NUM_BLOCKS, SPLIT_INDEX, \ EARLY_BLOCK_COUNT, LATE_BLOCK_COUNT ) \ EARLY_BLOCK_COUNT = LATE_BLOCK_COUNT = COUNT / NUM_BLOCKS; \ SPLIT_INDEX = COUNT % NUM_BLOCKS; \ if (0 != SPLIT_INDEX) { \ EARLY_BLOCK_COUNT = EARLY_BLOCK_COUNT + 1; \ } \ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allreduce_ompi_ring_segmented::allreduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int ret = MPI_SUCCESS; int line; int k, recv_from, send_to; int early_blockcount, late_blockcount, split_rank; int segcount, max_segcount; int num_phases, phase; int block_count; unsigned int inbi; size_t typelng; char *tmpsend = NULL, *tmprecv = NULL; char *inbuf[2] = {NULL, NULL}; ptrdiff_t true_extent, extent; ptrdiff_t block_offset, max_real_segsize; MPI_Request reqs[2] = {NULL, NULL}; const size_t segsize = 1 << 20; /* 1 MB */ int size = comm->size(); int rank = comm->rank(); XBT_DEBUG("coll:tuned:allreduce_intra_ring_segmented rank %d, count %d", rank, count); /* Special case for size == 1 */ if (1 == size) { if (MPI_IN_PLACE != sbuf) { ret= Datatype::copy(sbuf, count, dtype,rbuf, count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } return MPI_SUCCESS; } /* Determine segment count based on the suggested segment size */ extent = dtype->get_extent(); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } true_extent = dtype->get_extent(); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } typelng = dtype->size(); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } segcount = count; COLL_TUNED_COMPUTED_SEGCOUNT(segsize, typelng, segcount) /* Special case for count less than size * segcount - use regular ring */ if (count < size * segcount) { XBT_DEBUG( "coll:tuned:allreduce_ring_segmented rank %d/%d, count %d, switching to regular ring", rank, size, count); return (Coll_allreduce_lr::allreduce(sbuf, rbuf, count, dtype, op, comm)); } /* Determine the number of phases of the algorithm */ num_phases = count / (size * segcount); if ((count % (size * segcount) >= size) && (count % (size * segcount) > ((size * segcount) / 2))) { num_phases++; } /* Determine the number of elements per block and corresponding block sizes. The blocks are divided into "early" and "late" ones: blocks 0 .. (split_rank - 1) are "early" and blocks (split_rank) .. (size - 1) are "late". Early blocks are at most 1 element larger than the late ones. Note, these blocks will be split into num_phases segments, out of the largest one will have max_segcount elements. */ COLL_TUNED_COMPUTE_BLOCKCOUNT( count, size, split_rank, early_blockcount, late_blockcount ) COLL_TUNED_COMPUTE_BLOCKCOUNT( early_blockcount, num_phases, inbi, max_segcount, k) max_real_segsize = true_extent + (max_segcount - 1) * extent; /* Allocate and initialize temporary buffers */ inbuf[0] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf[0]) { ret = -1; line = __LINE__; goto error_hndl; } if (size > 2) { inbuf[1] = (char*)smpi_get_tmp_recvbuffer(max_real_segsize); if (NULL == inbuf[1]) { ret = -1; line = __LINE__; goto error_hndl; } } /* Handle MPI_IN_PLACE */ if (MPI_IN_PLACE != sbuf) { ret= Datatype::copy(sbuf, count, dtype,rbuf, count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } /* Computation loop: for each phase, repeat ring allreduce computation loop */ for (phase = 0; phase < num_phases; phase ++) { ptrdiff_t phase_offset; int early_phase_segcount, late_phase_segcount, split_phase, phase_count; /* For each of the remote nodes: - post irecv for block (r-1) - send block (r) To do this, first compute block offset and count, and use block offset to compute phase offset. - in loop for every step k = 2 .. n - post irecv for block (r + n - k) % n - wait on block (r + n - k + 1) % n to arrive - compute on block (r + n - k + 1) % n - send block (r + n - k + 1) % n - wait on block (r + 1) - compute on block (r + 1) - send block (r + 1) to rank (r + 1) Note that we must be careful when computing the begining of buffers and for send operations and computation we must compute the exact block size. */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; inbi = 0; /* Initialize first receive from the neighbor on the left */ reqs[inbi] = Request::irecv(inbuf[inbi], max_segcount, dtype, recv_from, 666, comm); /* Send first block (my block) to the neighbor on the right: - compute my block and phase offset - send data */ block_offset = ((rank < split_rank)? (rank * early_blockcount) : (rank * late_blockcount + split_rank)); block_count = ((rank < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmpsend = ((char*)rbuf) + (block_offset + phase_offset) * extent; Request::send(tmpsend, phase_count, dtype, send_to, 666, comm); for (k = 2; k < size; k++) { const int prevblock = (rank + size - k + 1) % size; inbi = inbi ^ 0x1; /* Post irecv for the current block */ reqs[inbi] = Request::irecv(inbuf[inbi], max_segcount, dtype, recv_from, 666, comm); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } /* Wait on previous block to arrive */ Request::wait(&reqs[inbi ^ 0x1], MPI_STATUS_IGNORE); /* Apply operation on previous block: result goes to rbuf rbuf[prevblock] = inbuf[inbi ^ 0x1] (op) rbuf[prevblock] */ block_offset = ((prevblock < split_rank)? (prevblock * early_blockcount) : (prevblock * late_blockcount + split_rank)); block_count = ((prevblock < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmprecv = ((char*)rbuf) + (block_offset + phase_offset) * extent; if(op!=MPI_OP_NULL) op->apply( inbuf[inbi ^ 0x1], tmprecv, &phase_count, dtype); /* send previous block to send_to */ Request::send(tmprecv, phase_count, dtype, send_to, 666, comm); } /* Wait on the last block to arrive */ Request::wait(&reqs[inbi], MPI_STATUS_IGNORE); /* Apply operation on the last block (from neighbor (rank + 1) rbuf[rank+1] = inbuf[inbi] (op) rbuf[rank + 1] */ recv_from = (rank + 1) % size; block_offset = ((recv_from < split_rank)? (recv_from * early_blockcount) : (recv_from * late_blockcount + split_rank)); block_count = ((recv_from < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmprecv = ((char*)rbuf) + (block_offset + phase_offset) * extent; if(op!=MPI_OP_NULL) op->apply( inbuf[inbi], tmprecv, &phase_count, dtype); } /* Distribution loop - variation of ring allgather */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; for (k = 0; k < size - 1; k++) { const int recv_data_from = (rank + size - k) % size; const int send_data_from = (rank + 1 + size - k) % size; const int send_block_offset = ((send_data_from < split_rank)? (send_data_from * early_blockcount) : (send_data_from * late_blockcount + split_rank)); const int recv_block_offset = ((recv_data_from < split_rank)? (recv_data_from * early_blockcount) : (recv_data_from * late_blockcount + split_rank)); block_count = ((send_data_from < split_rank)? early_blockcount : late_blockcount); tmprecv = (char*)rbuf + recv_block_offset * extent; tmpsend = (char*)rbuf + send_block_offset * extent; Request::sendrecv(tmpsend, block_count, dtype, send_to, 666, tmprecv, early_blockcount, dtype, recv_from, 666, comm, MPI_STATUS_IGNORE); } if (NULL != inbuf[0]) smpi_free_tmp_buffer(inbuf[0]); if (NULL != inbuf[1]) smpi_free_tmp_buffer(inbuf[1]); return MPI_SUCCESS; error_hndl: XBT_DEBUG("%s:%4d\tRank %d Error occurred %d\n", __FILE__, line, rank, ret); if (NULL != inbuf[0]) smpi_free_tmp_buffer(inbuf[0]); if (NULL != inbuf[1]) smpi_free_tmp_buffer(inbuf[1]); return ret; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-rdb.cpp0000644000175000017500000001375413217757317024157 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for intra-communication and rdb for inter-communication*/ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) Recursive doubling intra-communication between root of each SMP node 3) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_rdb::allreduce(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = comm->size(); rank = comm->rank(); MPI_Aint extent; extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); /* compute intra and inter ranking */ int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ Request::sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* start binomial reduce intra communication inside each SMP node */ mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } /* end binomial reduce intra-communication */ /* start rdb (recursive doubling) all-reduce inter-communication between each SMP nodes : each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { /* find nearest power-of-two less than or equal to inter_comm_size */ int pof2, rem, newrank, newdst; pof2 = 1; while (pof2 <= inter_comm_size) pof2 <<= 1; pof2 >>= 1; rem = inter_comm_size - pof2; /* In the non-power-of-two case, all even-numbered processes of rank < 2*rem send their data to (rank+1). These even-numbered processes no longer participate in the algorithm until the very end. */ if (inter_rank < 2 * rem) { if (inter_rank % 2 == 0) { dst = rank + num_core; Request::send(recv_buf, count, dtype, dst, tag, comm); newrank = -1; } else { src = rank - num_core; Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); newrank = inter_rank / 2; } } else { newrank = inter_rank - rem; } /* example inter-communication RDB rank change algorithm 0,4,8,12..36 <= true rank (assume 4 core per SMP) 0123 4567 89 <= inter_rank 1 3 4567 89 (1,3 got data from 0,2 : 0,2 will be idle until the end) 0 1 4567 89 0 1 2345 67 => newrank */ if (newrank != -1) { mask = 1; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; dst *= num_core; /* exchange data in rdb manner */ Request::sendrecv(recv_buf, count, dtype, dst, tag, tmp_buf, count, dtype, dst, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); mask <<= 1; } } /* non pof2 case left-over processes (all even ranks: < 2 * rem) get the result */ if (inter_rank < 2 * rem) { if (inter_rank % 2) { Request::send(recv_buf, count, dtype, rank - num_core, tag, comm); } else { Request::recv(recv_buf, count, dtype, rank + num_core, tag, comm, &status); } } } /* start binomial broadcast intra-communication inside each SMP nodes */ int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); Request::recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { Request::send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-binomial.cpp0000644000175000017500000001161113217757317025170 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for both intra-communication inter-communication*/ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) binomial_tree reduce intra-communication between root of each SMP node 3) binomial_tree bcast intra-communication between root of each SMP node 4) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_binomial::allreduce(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } MPI_Status status; comm_size=comm->size(); rank=comm->rank(); MPI_Aint extent, lb; dtype->extent(&lb, &extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); /* compute intra and inter ranking */ int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ Request::sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* start binomial reduce intra communication inside each SMP node */ mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } /* start binomial reduce inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); } } else { dst = (inter_rank & (~mask)) * num_core; Request::send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } } /* start binomial broadcast inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if (inter_rank & mask) { src = (inter_rank - mask) * num_core; Request::recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (inter_rank < inter_comm_size) { dst = (inter_rank + mask) * num_core; if (dst < comm_size) { Request::send(recv_buf, count, dtype, dst, tag, comm); } } mask >>= 1; } } /* start binomial broadcast intra-communication inside each SMP nodes */ int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); Request::recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { Request::send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-mvapich-rs.cpp0000644000175000017500000002655313217757317024665 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "../colls_private.hpp" #include namespace simgrid{ namespace smpi{ int Coll_allreduce_mvapich2_rs::allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int newrank = 0; int mask, pof2, i, send_idx, recv_idx, last_idx, send_cnt; int dst, is_commutative, rem, newdst, recv_cnt, *cnts, *disps; MPI_Aint true_lb, true_extent, extent; void *tmp_buf, *tmp_buf_free; if (count == 0) { return MPI_SUCCESS; } /* homogeneous */ int comm_size = comm->size(); int rank = comm->rank(); is_commutative = (op==MPI_OP_NULL || op->is_commutative()); /* need to allocate temporary buffer to store incoming data */ datatype->extent(&true_lb, &true_extent); extent = datatype->get_extent(); tmp_buf_free = smpi_get_tmp_recvbuffer(count * std::max(extent, true_extent)); /* adjust for potential negative lower bound in datatype */ tmp_buf = (void *) ((char *) tmp_buf_free - true_lb); /* copy local data into recvbuf */ if (sendbuf != MPI_IN_PLACE) { mpi_errno = Datatype::copy(sendbuf, count, datatype, recvbuf, count, datatype); } /* find nearest power-of-two less than or equal to comm_size */ for( pof2 = 1; pof2 <= comm_size; pof2 <<= 1 ); pof2 >>=1; rem = comm_size - pof2; /* In the non-power-of-two case, all even-numbered processes of rank < 2*rem send their data to (rank+1). These even-numbered processes no longer participate in the algorithm until the very end. The remaining processes form a nice power-of-two. */ if (rank < 2 * rem) { if (rank % 2 == 0) { /* even */ Request::send(recvbuf, count, datatype, rank + 1, COLL_TAG_ALLREDUCE, comm); /* temporarily set the rank to -1 so that this process does not pariticipate in recursive doubling */ newrank = -1; } else { /* odd */ Request::recv(tmp_buf, count, datatype, rank - 1, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* do the reduction on received data. since the ordering is right, it doesn't matter whether the operation is commutative or not. */ if(op!=MPI_OP_NULL) op->apply( tmp_buf, recvbuf, &count, datatype); /* change the rank */ newrank = rank / 2; } } else { /* rank >= 2*rem */ newrank = rank - rem; } /* If op is user-defined or count is less than pof2, use recursive doubling algorithm. Otherwise do a reduce-scatter followed by allgather. (If op is user-defined, derived datatypes are allowed and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky, therefore using recursive doubling in that case.) */ if (newrank != -1) { if (/*(HANDLE_GET_KIND(op) != HANDLE_KIND_BUILTIN) ||*/ (count < pof2)) { /* use recursive doubling */ mask = 0x1; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; /* Send the most current data, which is in recvbuf. Recv into tmp_buf */ Request::sendrecv(recvbuf, count, datatype, dst, COLL_TAG_ALLREDUCE, tmp_buf, count, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ if (is_commutative || (dst < rank)) { /* op is commutative OR the order is already right */ if(op!=MPI_OP_NULL) op->apply( tmp_buf, recvbuf, &count, datatype); } else { /* op is noncommutative and the order is not right */ if(op!=MPI_OP_NULL) op->apply( recvbuf, tmp_buf, &count, datatype); /* copy result back into recvbuf */ mpi_errno = Datatype::copy(tmp_buf, count, datatype, recvbuf, count, datatype); } mask <<= 1; } } else { /* do a reduce-scatter followed by allgather */ /* for the reduce-scatter, calculate the count that each process receives and the displacement within the buffer */ cnts = (int *)xbt_malloc(pof2 * sizeof (int)); disps = (int *)xbt_malloc(pof2 * sizeof (int)); for (i = 0; i < (pof2 - 1); i++) { cnts[i] = count / pof2; } cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) { disps[i] = disps[i - 1] + cnts[i - 1]; } mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ Request::sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, COLL_TAG_ALLREDUCE, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ /* This algorithm is used only for predefined ops and predefined ops are always commutative. */ if(op!=MPI_OP_NULL) op->apply( (char *) tmp_buf + disps[recv_idx] * extent, (char *) recvbuf + disps[recv_idx] * extent, &recv_cnt, datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; /* update last_idx, but not in last iteration because the value is needed in the allgather step below. */ if (mask < pof2) last_idx = recv_idx + pof2 / mask; } /* now do the allgather */ mask >>= 1; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) { last_idx = last_idx + pof2 / (mask * 2); } recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) { send_cnt += cnts[i]; } for (i = recv_idx; i < last_idx; i++) { recv_cnt += cnts[i]; } } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) { send_cnt += cnts[i]; } for (i = recv_idx; i < send_idx; i++) { recv_cnt += cnts[i]; } } Request::sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, COLL_TAG_ALLREDUCE, (char *) recvbuf + disps[recv_idx] * extent, recv_cnt, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); if (newrank > newdst) { send_idx = recv_idx; } mask >>= 1; } xbt_free(disps); xbt_free(cnts); } } /* In the non-power-of-two case, all odd-numbered processes of rank < 2*rem send the result to (rank-1), the ranks who didn't participate above. */ if (rank < 2 * rem) { if (rank % 2) { /* odd */ Request::send(recvbuf, count, datatype, rank - 1, COLL_TAG_ALLREDUCE, comm); } else { /* even */ Request::recv(recvbuf, count, datatype, rank + 1, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); } } smpi_free_tmp_buffer(tmp_buf_free); return (mpi_errno); } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-rab1.cpp0000644000175000017500000000640013217757317023426 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include namespace simgrid{ namespace smpi{ // NP pow of 2 for now int Coll_allreduce_rab1::allreduce(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { MPI_Status status; MPI_Aint extent; int tag = COLL_TAG_ALLREDUCE, send_size, newcnt, share; unsigned int pof2 = 1, mask; int send_idx, recv_idx, dst, send_cnt, recv_cnt; void *recv, *tmp_buf; int rank = comm->rank(); unsigned int nprocs = comm->size(); if((nprocs&(nprocs-1))) THROWF(arg_error,0, "allreduce rab1 algorithm can't be used with non power of two number of processes ! "); extent = dtype->get_extent(); pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; send_idx = recv_idx = 0; // uneven count if ((count % nprocs)) { send_size = (count + nprocs) / nprocs; newcnt = send_size * nprocs; recv = (void *) smpi_get_tmp_recvbuffer(extent * newcnt); tmp_buf = (void *) smpi_get_tmp_sendbuffer(extent * newcnt); memcpy(recv, sbuff, extent * count); mask = pof2 / 2; share = newcnt / pof2; while (mask > 0) { dst = rank ^ mask; send_cnt = recv_cnt = newcnt / (pof2 / mask); if (rank < dst) send_idx = recv_idx + (mask * share); else recv_idx = send_idx + (mask * share); Request::sendrecv((char *) recv + send_idx * extent, send_cnt, dtype, dst, tag, tmp_buf, recv_cnt, dtype, dst, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *) recv + recv_idx * extent, &recv_cnt, dtype); // update send_idx for next iteration send_idx = recv_idx; mask >>= 1; } memcpy(tmp_buf, (char *) recv + recv_idx * extent, recv_cnt * extent); Colls::allgather(tmp_buf, recv_cnt, dtype, recv, recv_cnt, dtype, comm); memcpy(rbuff, recv, count * extent); smpi_free_tmp_buffer(recv); smpi_free_tmp_buffer(tmp_buf); } else { tmp_buf = (void *) smpi_get_tmp_sendbuffer(extent * count); memcpy(rbuff, sbuff, count * extent); mask = pof2 / 2; share = count / pof2; while (mask > 0) { dst = rank ^ mask; send_cnt = recv_cnt = count / (pof2 / mask); if (rank < dst) send_idx = recv_idx + (mask * share); else recv_idx = send_idx + (mask * share); Request::sendrecv((char *) rbuff + send_idx * extent, send_cnt, dtype, dst, tag, tmp_buf, recv_cnt, dtype, dst, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *) rbuff + recv_idx * extent, &recv_cnt, dtype); // update send_idx for next iteration send_idx = recv_idx; mask >>= 1; } memcpy(tmp_buf, (char *) rbuff + recv_idx * extent, recv_cnt * extent); Colls::allgather(tmp_buf, recv_cnt, dtype, rbuff, recv_cnt, dtype, comm); smpi_free_tmp_buffer(tmp_buf); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-lr.cpp0000644000175000017500000000761113217757317023223 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific all-reduce operation designed bandwidth optimally Bug fixing by Xin Yuan, 04/04/2008 */ /* ** NOTE ** Use -DMPICH2_REDUCTION if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include namespace simgrid{ namespace smpi{ int Coll_allreduce_lr::allreduce(void *sbuf, void *rbuf, int rcount, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int tag = COLL_TAG_ALLREDUCE; MPI_Status status; int rank, i, size, count; int send_offset, recv_offset; int remainder, remainder_flag, remainder_offset; rank = comm->rank(); size = comm->size(); /* make it compatible with all data type */ MPI_Aint extent; extent = dtype->get_extent(); /* when communication size is smaller than number of process (not support) */ if (rcount < size) { XBT_WARN("MPI_allreduce_lr use default MPI_allreduce."); Coll_allreduce_default::allreduce(sbuf, rbuf, rcount, dtype, op, comm); return MPI_SUCCESS; } /* when communication size is not divisible by number of process: call the native implementation for the remain chunk at the end of the operation */ if (rcount % size != 0) { remainder = rcount % size; remainder_flag = 1; remainder_offset = (rcount / size) * size * extent; } else { remainder = remainder_flag = remainder_offset = 0; } /* size of each point-to-point communication is equal to the size of the whole message divided by number of processes */ count = rcount / size; /* our ALL-REDUCE implementation 1. copy (partial of)send_buf to recv_buf 2. use logical ring reduce-scatter 3. use logical ring all-gather */ // copy partial data send_offset = ((rank - 1 + size) % size) * count * extent; recv_offset = ((rank - 1 + size) % size) * count * extent; Request::sendrecv((char *) sbuf + send_offset, count, dtype, rank, tag - 1, (char *) rbuf + recv_offset, count, dtype, rank, tag - 1, comm, &status); // reduce-scatter for (i = 0; i < (size - 1); i++) { send_offset = ((rank - 1 - i + 2 * size) % size) * count * extent; recv_offset = ((rank - 2 - i + 2 * size) % size) * count * extent; // recv_offset = ((rank-i+2*size)%size)*count*extent; Request::sendrecv((char *) rbuf + send_offset, count, dtype, ((rank + 1) % size), tag + i, (char *) rbuf + recv_offset, count, dtype, ((rank + size - 1) % size), tag + i, comm, &status); // compute result to rbuf+recv_offset if(op!=MPI_OP_NULL) op->apply( (char *) sbuf + recv_offset, (char *) rbuf + recv_offset, &count, dtype); } // all-gather for (i = 0; i < (size - 1); i++) { send_offset = ((rank - i + 2 * size) % size) * count * extent; recv_offset = ((rank - 1 - i + 2 * size) % size) * count * extent; Request::sendrecv((char *) rbuf + send_offset, count, dtype, ((rank + 1) % size), tag + i, (char *) rbuf + recv_offset, count, dtype, ((rank + size - 1) % size), tag + i, comm, &status); } /* when communication size is not divisible by number of process: call the native implementation for the remain chunk at the end of the operation */ if (remainder_flag) { return Colls::allreduce((char *) sbuf + remainder_offset, (char *) rbuf + remainder_offset, remainder, dtype, op, comm); } return 0; } } } SimGrid-3.18/src/smpi/colls/allreduce/allreduce-smp-rsag-lr.cpp0000644000175000017500000001700513217757317024750 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ namespace simgrid{ namespace smpi{ int Coll_allreduce_smp_rsag_lr::allreduce(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } int num_core=1; if (comm->is_uniform()){ num_core = comm->get_intra_comm()->size(); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = comm->size(); rank = comm->rank(); MPI_Aint extent; extent = dtype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; if (not rank) { //printf("intra com size = %d\n",num_core); //printf("inter com size = %d\n",inter_comm_size); } Request::sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_buf, &count, dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); Request::send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int send_offset, recv_offset; int send_count, recv_count; int curr_size = count / inter_comm_size; int curr_remainder = count % inter_comm_size; int to = ((inter_rank + 1) % inter_comm_size) * num_core; int from = ((inter_rank + inter_comm_size - 1) % inter_comm_size) * num_core; int i; //printf("node %d to %d from %d\n",rank,to,from); /* last segment may have a larger size since it also include the remainder */ int last_segment_ptr = (inter_comm_size - 1) * (count / inter_comm_size) * extent; for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; recv_offset = ((inter_rank - 2 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; /* adjust size */ if (send_offset != last_segment_ptr) send_count = curr_size; else send_count = curr_size + curr_remainder; if (recv_offset != last_segment_ptr) recv_count = curr_size; else recv_count = curr_size + curr_remainder; Request::sendrecv((char *) recv_buf + send_offset, send_count, dtype, to, tag + i, tmp_buf, recv_count, dtype, from, tag + i, comm, &status); // result is in rbuf if(op!=MPI_OP_NULL) op->apply( tmp_buf, (char *) recv_buf + recv_offset, &recv_count, dtype); } // INTER: allgather for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - i + inter_comm_size) % inter_comm_size) * curr_size * extent; recv_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; /* adjust size */ if (send_offset != last_segment_ptr) send_count = curr_size; else send_count = curr_size + curr_remainder; if (recv_offset != last_segment_ptr) recv_count = curr_size; else recv_count = curr_size + curr_remainder; Request::sendrecv((char *) recv_buf + send_offset, send_count, dtype, to, tag + i, (char *) recv_buf + recv_offset, recv_count, dtype, from, tag + i, comm, &status); } } // INTER_binomial_reduce // only root node for each SMP /* if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { Request::recv(tmp_buf, count, dtype, src, tag, comm, &status); (* uop) (tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank & (~mask)) * num_core; Request::send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<=1; } } */ // INTER_binomial_bcast // if (intra_rank == 0) { // mask = 1; // while (mask < inter_comm_size) { // if (inter_rank & mask) { // src = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // Request::recv(recv_buf, count, dtype, src, tag, comm, &status); // break; // } // mask <<= 1; // } // // mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); // while (mask > 0) { // if (inter_rank < inter_comm_size) { // dst = (inter_rank + mask) * num_core; // if (dst < comm_size) { // //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // Request::send(recv_buf, count, dtype, dst, tag, comm); // } // } // mask >>= 1; // } // } // INTRA_binomial_bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); Request::recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); Request::send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/colls_global.cpp0000644000175000017500000000064613217757320021337 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_colls, smpi, "Logging specific to SMPI collectives"); SimGrid-3.18/src/smpi/colls/smpi_intel_mpi_selector.cpp0000644000175000017500000011443013217757320023610 0ustar mquinsonmquinson/* selector for collective algorithms based on openmpi's default coll_tuned_decision_fixed selector */ /* Copyright (c) 2009-2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.hpp" // This selector is based on information gathered on the Stampede cluster, with Intel MPI 4.1.3.049, and from the intel reference manual. The data was gathered launching runs with 1,2,4,8,16 processes per node. #define INTEL_MAX_NB_THRESHOLDS 32 #define INTEL_MAX_NB_NUMPROCS 12 #define INTEL_MAX_NB_PPN 5 /* 1 2 4 8 16 ppn */ struct intel_tuning_table_size_element { unsigned int max_size; int algo; }; struct intel_tuning_table_numproc_element { int max_num_proc; int num_elems; intel_tuning_table_size_element elems[INTEL_MAX_NB_THRESHOLDS]; }; struct intel_tuning_table_element { int ppn; intel_tuning_table_numproc_element elems[INTEL_MAX_NB_NUMPROCS]; }; /* I_MPI_ADJUST_ALLREDUCE MPI_Allreduce 1 - Recursive doubling algorithm 2 - Rabenseifner's algorithm 3 - Reduce + Bcast algorithm 4 - Topology aware Reduce + Bcast algorithm 5 - Binomial gather + scatter algorithm 6 - Topology aware binominal gather + scatter algorithm 7 - Shumilin's ring algorithm 8 - Ring algorithm as Shumilin's ring algorithm is unknown, default to ring' */ namespace simgrid{ namespace smpi{ int (*intel_allreduce_functions_table[])(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) ={ Coll_allreduce_rdb::allreduce, Coll_allreduce_rab1::allreduce, Coll_allreduce_redbcast::allreduce, Coll_allreduce_mvapich2_two_level::allreduce, Coll_allreduce_smp_binomial::allreduce, Coll_allreduce_mvapich2_two_level::allreduce, Coll_allreduce_ompi_ring_segmented::allreduce, Coll_allreduce_ompi_ring_segmented::allreduce }; intel_tuning_table_element intel_allreduce_table[] = { {1,{ { 2,9,{ {6,7}, {85,1}, {192,7}, {853,1}, {1279,7}, {16684,1}, {34279,8}, {1681224,3}, {2147483647,7} } }, { 4, 8,{ {16,7}, {47,1}, {2062,7}, {16699,1}, {33627,7}, {70732,8}, {1300705,3}, {2147483647,8} } }, {8,8,{ {118,1}, {146,4}, {16760,1}, {36364,6}, {136239,8}, {315710,7}, {3220366,3}, {2147483647,8} } }, {16,7,{ {934,1}, {1160,6}, {15505,1}, {52730,2}, {300705,8}, {563680,7}, {2147483647,3} } }, {2147483647,11,{ {5,6}, {11,4}, {182,1}, {700,6}, {1450,4}, {11146,1}, {25539,6}, {37634,4}, {93784,6}, {817658,2}, {2147483647,3} } } } }, {2,{ { 4,6,{ {2084,7}, {15216,1}, {99715,7}, {168666,3}, {363889,2}, {2147483647,7} } }, { 8,6,{ {14978,1}, {66879,2}, {179296,8}, {304801,3}, {704509,7}, {2147483647,2} } }, { 16,6,{ {16405,1}, {81784,2}, {346385,8}, {807546,7}, {1259854,2}, {2147483647,3} } }, { 32,4,{ {8913,1}, {103578,2}, {615876,8}, {2147483647,2} } }, { 64,7,{ {1000,1}, {2249,2}, {6029,1}, {325357,2}, {1470976,8}, {2556670,7}, {2147483647,3} } }, { 128,5,{ {664,1}, {754706,2}, {1663862,4}, {3269097,2}, {2147483647,7} } }, { 2147483647,3,{ {789,1}, {2247589,2}, {2147483647,8} } } } }, {4,{ { 4,4,{ {5738,1}, {197433,2}, {593742,7}, {2147483647,2} } }, { 8,7,{ {5655,1}, {75166,2}, {177639,8}, {988014,3}, {1643869,2}, {2494859,8}, {2147483647,2} } }, { 16,7,{ {587,1}, {3941,2}, {9003,1}, {101469,2}, {355768,8}, {3341814,3}, {2147483647,8} } }, { 32,4,{ {795,1}, {146567,2}, {732118,8}, {2147483647,3} } }, { 64,4,{ {528,1}, {221277,2}, {1440737,8}, {2147483647,3} } }, { 128,4,{ {481,1}, {593833,2}, {2962021,8}, {2147483647,7} } }, { 256,2,{ {584,1}, {2147483647,2} } }, { 2147483647,3,{ {604,1}, {2997006,2}, {2147483647,8} } } } }, {8,{ { 8,6,{ {2560,1}, {114230,6}, {288510,8}, {664038,2}, {1339913,6}, {2147483647,4} } }, { 16,5,{ {497,1}, {54201,2}, {356217,8}, {3413609,3}, {2147483647,8} } }, { 32,5,{ {377,1}, {109745,2}, {716514,8}, {3976768,3}, {2147483647,8} } }, { 64,6,{ {109,1}, {649,5}, {266080,2}, {1493331,8}, {2541403,7}, {2147483647,3} } }, { 128,4,{ {7,1}, {751,5}, {408808,2}, {2147483647,8} } }, { 256,3,{ {828,5}, {909676,2}, {2147483647,8} } }, { 512,5,{ {847,5}, {1007066,2}, {1068775,4}, {2803389,2}, {2147483647,8} } }, { 2147483647,3,{ {1974,5}, {4007876,2}, {2147483647,8} } } } }, {16,{ { 16,12,{ {409,1}, {768,6}, {1365,4}, {3071,6}, {11299,2}, {21746,6}, {55629,2}, {86065,4}, {153867,2}, {590560,6}, {1448760,2}, {2147483647,8}, } }, { 32,8,{ {6,1}, {24,5}, {86,1}, {875,5}, {74528,2}, {813050,8}, {1725981,7}, {2147483647,8}, } }, { 64,6,{ {1018,5}, {1217,6}, {2370,5}, {160654,2}, {1885487,8}, {2147483647,3}, } }, { 128,4,{ {2291,5}, {434465,2}, {3525103,8}, {2147483647,7}, } }, { 256,3,{ {2189,5}, {713154,2}, {2147483647,8}, } }, { 512,3,{ {2140,5}, {1235056,2}, {2147483647,8}, } }, { 2147483647,3,{ {2153,5}, {2629855,2}, {2147483647,8}, } } } } }; /*I_MPI_ADJUST_ALLTOALL MPI_Alltoall 1. Bruck's algorithm 2. Isend/Irecv + waitall algorithm 3. Pair wise exchange algorithm 4. Plum's algorithm */ intel_tuning_table_element intel_alltoall_table[] = { {1,{ { 2,1, { {2147483647,3} } }, { 4,2, { {0,4}, {2147483647,2} } }, {8,1, { {2147483647,2} } }, {16,5, { {0,3}, {84645,2}, {167570,3}, {413152,4}, {2147483647,2} } }, {32,6, { {61,1}, {164,2}, {696,1}, {143254,2}, {387024,3}, {2147483647,2} }, }, {64,4, { {523,1}, {146088,2}, {488989,4}, {2147483647,2} } }, {2147483647,3, { {270,1}, {628,4}, {2147483647,2} } } } }, {2, { { 4,4,{ {1,2}, {75,3}, {131072,2}, {2147483647,2} } }, { 8,3,{ {709,1}, {131072,2}, {2147483647,2} } }, { 16,4,{ {40048,2}, {131072,3}, {155927,3}, {2147483647,4} } }, { 32,7,{ {105,1}, {130,2}, {1030,1}, {58893,2}, {131072,2}, {271838,3}, {2147483647,2} } }, { 2147483647,8,{ {521,1}, {2032,4}, {2412,2}, {4112,4}, {61620,2}, {131072,3}, {427408,3}, {2147483647,4} } } } }, {4,{ { 8,3,{ {512,1}, {32768,2}, {2147483647,2} } }, { 64,8,{ {7,1}, {199,4}, {764,1}, {6409,4}, {20026,2}, {32768,3}, {221643,4}, {2147483647,3} } }, { 2147483647,7,{ {262,1}, {7592,4}, {22871,2}, {32768,3}, {47538,3}, {101559,4}, {2147483647,3} } } } }, {8,{ { 16,6,{ {973,1}, {5126,4}, {16898,2}, {32768,4}, {65456,4}, {2147483647,2} } }, { 32,7,{ {874,1}, {6727,4}, {17912,2}, {32768,3}, {41513,3}, {199604,4}, {2147483647,3} } }, { 64,8,{ {5,1}, {114,4}, {552,1}, {8130,4}, {32768,3}, {34486,3}, {160113,4}, {2147483647,3} } }, { 128,6,{ {270,1}, {3679,4}, {32768,3}, {64367,3}, {146595,4}, {2147483647,3} } }, { 2147483647,4,{ {133,1}, {4017,4}, {32768,3}, {76351,4}, {2147483647,3} } } } }, {16,{ { 32,7,{ {963,1}, {1818,4}, {20007,2}, {32768,4}, {54296,4}, {169735,3}, {2147483647,2} } }, { 64,11,{ {17,1}, {42,4}, {592,1}, {2015,4}, {2753,2}, {6496,3}, {20402,4}, {32768,3}, {36246,3}, {93229,4}, {2147483647,3} } }, { 128,9,{ {18,1}, {40,4}, {287,1}, {1308,4}, {6842,1}, {32768,3}, {36986,3}, {129081,4}, {2147483647,3} } }, { 256,7,{ {135,1}, {1538,4}, {3267,1}, {4132,3}, {31469,4}, {32768,3}, {2147483647,3} } }, { 2147483647,8,{ {66,1}, {1637,4}, {2626,1}, {4842,4}, {32768,3}, {33963,3}, {72978,4}, {2147483647,3} } } } } }; int (*intel_alltoall_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm) ={ Coll_alltoall_bruck::alltoall, Coll_alltoall_mvapich2_scatter_dest::alltoall, Coll_alltoall_pair::alltoall, Coll_alltoall_mvapich2::alltoall//Plum is proprietary ? (and super efficient) }; /*I_MPI_ADJUST_BARRIER MPI_Barrier 1. Dissemination algorithm 2. Recursive doubling algorithm 3. Topology aware dissemination algorithm 4. Topology aware recursive doubling algorithm 5. Binominal gather + scatter algorithm 6. Topology aware binominal gather + scatter algorithm */ static int intel_barrier_gather_scatter(MPI_Comm comm){ //our default barrier performs a antibcast/bcast Coll_barrier_default::barrier(comm); return MPI_SUCCESS; } int (*intel_barrier_functions_table[])(MPI_Comm comm) ={ Coll_barrier_ompi_basic_linear::barrier, Coll_barrier_ompi_recursivedoubling::barrier, Coll_barrier_ompi_basic_linear::barrier, Coll_barrier_ompi_recursivedoubling::barrier, intel_barrier_gather_scatter, intel_barrier_gather_scatter }; intel_tuning_table_element intel_barrier_table[] = { {1,{ {2,1, { {2147483647,2} } }, {4,1, { {2147483647,6} } }, {8,1, { {2147483647,1} } }, {64,1, { {2147483647,2} } }, {2147483647,1, { {2147483647,6} } } } }, {2,{ { 2,1,{ {2147483647,1} } }, { 4,1,{ {2147483647,3} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 128,1,{ {2147483647,3} } }, { 2147483647,1,{ {2147483647,4} } } } }, {4,{ { 4,1,{ {2147483647,2} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } }, {8,{ { 8,1,{ {2147483647,1} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } }, {16,{ { 4,1,{ {2147483647,2} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } } }; /*I_MPI_ADJUST_BCAST MPI_Bcast 1. Binomial algorithm 2. Recursive doubling algorithm 3. Ring algorithm 4. Topology aware binomial algorithm 5. Topology aware recursive doubling algorithm 6. Topology aware ring algorithm 7. Shumilin's bcast algorithm */ int (*intel_bcast_functions_table[])(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm) ={ Coll_bcast_binomial_tree::bcast, //Coll_bcast_scatter_rdb_allgather::bcast, Coll_bcast_NTSL::bcast, Coll_bcast_NTSL::bcast, Coll_bcast_SMP_binomial::bcast, //Coll_bcast_scatter_rdb_allgather::bcast, Coll_bcast_NTSL::bcast, Coll_bcast_SMP_linear::bcast, Coll_bcast_mvapich2::bcast,//we don't know shumilin's algo' }; intel_tuning_table_element intel_bcast_table[] = { {1,{ {2,9, { {1,2}, {402,7}, {682,5}, {1433,4}, {5734,7}, {21845,1}, {95963,6}, {409897,5}, {2147483647,1} } }, {4,1, { {2147483647,7} } }, {8,11, { {3,6}, {4,7}, {25,6}, {256,1}, {682,6}, {1264,1}, {2234,6}, {6655,5}, {16336,1}, {3998434,7}, {2147483647,6} } }, {2147483647,1, { {2147483647,7} } } } }, {2,{ { 4,6,{ {806,4}, {18093,7}, {51366,6}, {182526,4}, {618390,1}, {2147483647,7} } }, { 8,6,{ {24,1}, {74,4}, {18137,1}, {614661,7}, {1284626,1}, {2147483647,2} } }, { 16,4,{ {1,1}, {158,7}, {16955,1}, {2147483647,7} } }, { 32,3,{ {242,7}, {10345,1}, {2147483647,7} } }, { 2147483647,4,{ {1,1}, {737,7}, {5340,1}, {2147483647,7} } } } }, {4,{ { 8,4,{ {256,4}, {17181,1}, {1048576,7}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } }, {8,{ { 16,5,{ {3,1}, {318,7}, {1505,1}, {1048576,7}, {2147483647,7} } }, { 32,3,{ {422,7}, {851,1}, {2147483647,7} } }, { 64,3,{ {468,7}, {699,1}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } }, {16,{ { 8,4,{ {256,4}, {17181,1}, {1048576,7}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } } }; /*I_MPI_ADJUST_REDUCE MPI_Reduce 1. Shumilin's algorithm 2. Binomial algorithm 3. Topology aware Shumilin's algorithm 4. Topology aware binomial algorithm 5. Rabenseifner's algorithm 6. Topology aware Rabenseifner's algorithm */ int (*intel_reduce_functions_table[])(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) ={ Coll_reduce_mvapich2::reduce, Coll_reduce_binomial::reduce, Coll_reduce_mvapich2::reduce, Coll_reduce_mvapich2_two_level::reduce, Coll_reduce_rab::reduce, Coll_reduce_rab::reduce }; intel_tuning_table_element intel_reduce_table[] = { {1,{ {2147483647,1, { {2147483647,1} } } } }, {2,{ { 2,1,{ {2147483647,1} } }, { 4,2,{ {10541,3}, {2147483647,1} } }, { 2147483647,1,{ {2147483647,1} } } } }, {4,{ { 256,1,{ {2147483647,1} } }, { 2147483647,2,{ {45,3}, {2147483647,1} } } } }, {8,{ { 512,1,{ {2147483647,1} } }, { 2147483647,3,{ {5,1}, {11882,3}, {2147483647,1} } } } }, {16,{ { 256,1,{ {2147483647,1} } }, { 2147483647,2,{ {45,3}, {2147483647,1} } } } } }; /* I_MPI_ADJUST_REDUCE_SCATTER MPI_Reduce_scatter 1. Recursive having algorithm 2. Pair wise exchange algorithm 3. Recursive doubling algorithm 4. Reduce + Scatterv algorithm 5. Topology aware Reduce + Scatterv algorithm */ static int intel_reduce_scatter_reduce_scatterv(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { Coll_reduce_scatter_default::reduce_scatter(sbuf, rbuf, rcounts,dtype, op,comm); return MPI_SUCCESS; } static int intel_reduce_scatter_recursivehalving(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { if(op==MPI_OP_NULL || op->is_commutative()) return Coll_reduce_scatter_ompi_basic_recursivehalving::reduce_scatter(sbuf, rbuf, rcounts,dtype, op,comm); else return Coll_reduce_scatter_mvapich2::reduce_scatter(sbuf, rbuf, rcounts,dtype, op,comm); } int (*intel_reduce_scatter_functions_table[])( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) ={ intel_reduce_scatter_recursivehalving, Coll_reduce_scatter_mpich_pair::reduce_scatter, Coll_reduce_scatter_mpich_rdb::reduce_scatter, intel_reduce_scatter_reduce_scatterv, intel_reduce_scatter_reduce_scatterv }; intel_tuning_table_element intel_reduce_scatter_table[] = { {1,{ {2,5, { {5,4}, {522429,2}, {1375877,5}, {2932736,2}, {2147483647,5} } }, {4,9, { {4,4}, {15,1}, {120,3}, {651,1}, {12188,3}, {33890,1}, {572117,2}, {1410202,5}, {2147483647,2} } }, {8,7, { {4,4}, {2263,1}, {25007,3}, {34861,1}, {169625,2}, {2734000,4}, {2147483647,2} } }, {16,5, { {4,4}, {14228,1}, {46084,3}, {522139,2}, {2147483647,5} } }, {32,5, { {4,4}, {27516,1}, {61693,3}, {2483469,2}, {2147483647,5} } }, {64,4, { {0,3}, {4,4}, {100396,1}, {2147483647,2} } }, {2147483647,6, { {0,3}, {4,4}, {186926,1}, {278259,3}, {1500100,2}, {2147483647,5} } } } }, {2,{ { 2,2,{ {6,1}, {2147483647,2} } }, { 4,7,{ {5,4}, {13,5}, {59,3}, {76,1}, {91488,3}, {680063,4}, {2147483647,2} } }, { 8,8,{ {4,4}, {11,5}, {31,1}, {69615,3}, {202632,2}, {396082,5}, {1495696,4}, {2147483647,2} } }, { 16,1,{ {4,4}, {345,1}, {79523,3}, {2147483647,2} } }, { 32,5,{ {0,3}, {4,4}, {992,1}, {71417,3}, {2147483647,2} } }, { 64,4,{ {4,4}, {1472,1}, {196592,3}, {2147483647,2} } }, { 128,5,{ {0,3}, {4,4}, {32892,1}, {381072,3}, {2147483647,2} } }, { 2147483647,6,{ {0,2}, {4,4}, {33262,1}, {1571397,3}, {2211398,5}, {2147483647,4} } } } }, {4,{ { 4,7,{ {12,4}, {27,5}, {49,3}, {187,1}, {405673,3}, {594687,4}, {2147483647,2} } }, { 8,5,{ {24,5}, {155,1}, {204501,3}, {274267,5}, {2147483647,4} } }, { 16,6,{ {63,1}, {72,3}, {264,1}, {168421,3}, {168421,4}, {2147483647,2} } }, { 32,10,{ {0,3}, {4,4}, {12,1}, {18,5}, {419,1}, {188739,3}, {716329,4}, {1365841,5}, {2430194,2}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {17,5}, {635,1}, {202937,3}, {308253,5}, {1389874,4}, {2147483647,2} } }, { 128,8,{ {0,3}, {4,4}, {16,5}, {1238,1}, {280097,3}, {631434,5}, {2605072,4}, {2147483647,2} } }, { 256,5,{ {0,2}, {4,4}, {16,5}, {2418,1}, {2147483647,3} } }, { 2147483647,6,{ {0,2}, {4,4}, {16,5}, {33182,1}, {3763779,3}, {2147483647,4} } } } }, {8,{ { 8,6,{ {5,4}, {494,1}, {97739,3}, {522836,2}, {554174,5}, {2147483647,2} } }, { 16,8,{ {5,4}, {62,1}, {94,3}, {215,1}, {185095,3}, {454784,4}, {607911,5}, {2147483647,4} } }, { 32,7,{ {0,3}, {4,4}, {302,1}, {250841,3}, {665822,4}, {1760980,5}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {41,5}, {306,1}, {332405,3}, {1269189,4}, {3712421,5}, {2147483647,4} } }, { 128,6,{ {0,3}, {4,4}, {39,5}, {526,1}, {487878,3}, {2147483647,4} } }, { 256,8,{ {0,2}, {4,4}, {36,5}, {1382,1}, {424162,3}, {632881,5}, {1127566,3}, {2147483647,4} } }, { 512,4,{ {4,4}, {34,5}, {5884,1}, {2147483647,3} } }, { 2147483647,4,{ {5,4}, {32,5}, {25105,1}, {2147483647,3} } } } }, {16,{ { 4,7,{ {12,4}, {27,5}, {49,3}, {187,1}, {405673,3}, {594687,4}, {2147483647,2} } }, { 8,5,{ {24,5}, {155,1}, {204501,3}, {274267,5}, {2147483647,4} } }, { 16,6,{ {63,1}, {72,3}, {264,1}, {168421,3}, {168421,4}, {2147483647,2} } }, { 32,10,{ {0,3}, {4,4}, {12,1}, {18,5}, {419,1}, {188739,3}, {716329,4}, {1365841,5}, {2430194,2}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {17,5}, {635,1}, {202937,3}, {308253,5}, {1389874,4}, {2147483647,2} } }, { 128,8,{ {0,3}, {4,4}, {16,5}, {1238,1}, {280097,3}, {631434,5}, {2605072,4}, {2147483647,2} } }, { 256,5,{ {0,2}, {4,4}, {16,5}, {2418,1}, {2147483647,3} } }, { 2147483647,6,{ {0,2}, {4,4}, {16,5}, {33182,1}, {3763779,3}, {2147483647,4} } } } } }; /* I_MPI_ADJUST_ALLGATHER MPI_Allgather 1. Recursive doubling algorithm 2. Bruck's algorithm 3. Ring algorithm 4. Topology aware Gatherv + Bcast algorithm */ int (*intel_allgather_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) ={ Coll_allgather_rdb::allgather, Coll_allgather_bruck::allgather, Coll_allgather_ring::allgather, Coll_allgather_GB::allgather }; intel_tuning_table_element intel_allgather_table[] = { {1,{ {4,11, { {1,4}, {384,1}, {1533,4}, {3296,1}, {10763,4}, {31816,3}, {193343,4}, {405857,3}, {597626,4}, {1844323,3}, {2147483647,4} } }, {8,10, { {12,4}, {46,1}, {205,4}, {3422,2}, {4200,4}, {8748,1}, {24080,3}, {33244,4}, {371159,1}, {2147483647,3} } }, {16, 8, { {3,4}, {53,1}, {100,4}, {170,1}, {6077,4}, {127644,1}, {143741,4}, {2147483647,3} } }, {2147483647,10, { {184,1}, {320,4}, {759,1}, {1219,4}, {2633,1}, {8259,4}, {123678,1}, {160801,4}, {284341,1}, {2147483647,4} } } } }, {2,{ { 8,6,{ {490,1}, {558,2}, {2319,1}, {46227,3}, {2215101,1}, {2147483647,3} } }, { 16,4,{ {1005,1}, {1042,2}, {2059,1}, {2147483647,3} } }, { 2147483647,2,{ {2454,1}, {2147483647,3} } } } }, {4,{ { 8,2,{ {2861,1}, {2147483647,3} } }, { 2147483647,2,{ {605,1}, {2147483647,3} } } } }, {8,{ { 16,4,{ {66,1}, {213,4}, {514,1}, {2147483647,3} } }, { 32,4,{ {91,1}, {213,4}, {514,1}, {2147483647,3} } }, { 64,4,{ {71,1}, {213,4}, {514,1}, {2147483647,3} } }, { 128,2,{ {305,1}, {2147483647,3} } }, { 2147483647,2,{ {213,1}, {2147483647,3} } } } }, {16,{ { 8,2,{ {2861,1}, {2147483647,3} } }, { 2147483647,2,{ {605,1}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_ALLGATHERV MPI_Allgatherv 1. Recursive doubling algorithm 2. Bruck's algorithm 3. Ring algorithm 4. Topology aware Gatherv + Bcast algorithm */ int (*intel_allgatherv_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) ={ Coll_allgatherv_mpich_rdb::allgatherv, Coll_allgatherv_ompi_bruck::allgatherv, Coll_allgatherv_ring::allgatherv, Coll_allgatherv_GB::allgatherv }; intel_tuning_table_element intel_allgatherv_table[] = { {1,{ {2,3, { {259668,3}, {635750,4}, {2147483647,3} } }, {4,7, { {1,1}, {5,4}, {46,1}, {2590,2}, {1177259,3}, {2767234,4}, {2147483647,3} } }, {8, 6, { {99,2}, {143,1}, {4646,2}, {63522,3}, {2187806,4}, {2147483647,3} } }, {2147483647,7, { {1,1}, {5,4}, {46,1}, {2590,2}, {1177259,3}, {2767234,4}, {2147483647,3} } } } }, {2,{ { 4,3,{ {3147,1}, {5622,2}, {2147483647,3} } }, { 8,3,{ {975,1}, {4158,2}, {2147483647,3} } }, { 16,2,{ {2146,1}, {2147483647,3} } }, { 32,4,{ {81,1}, {414,2}, {1190,1}, {2147483647,3} } }, { 2147483647,5,{ {1,2}, {3,1}, {783,2}, {1782,4}, {2147483647,3} } } } }, {4,{ { 8,2,{ {2554,1}, {2147483647,3} } }, { 16,4,{ {272,1}, {657,2}, {2078,1}, {2147483647,3} } }, { 32,2,{ {1081,1}, {2147483647,3} } }, { 64,2,{ {547,1}, {2147483647,3} } }, { 2147483647,5,{ {19,1}, {239,2}, {327,1}, {821,4}, {2147483647,3} } } } }, {8,{ { 16,3,{ {55,1}, {514,2}, {2147483647,3} } }, { 32,4,{ {53,1}, {167,4}, {514,2}, {2147483647,3} } }, { 64,3,{ {13,1}, {319,4}, {2147483647,3} } }, { 128,7,{ {2,1}, {11,2}, {48,1}, {201,2}, {304,1}, {1048,4}, {2147483647,3} } }, { 2147483647,5,{ {5,1}, {115,4}, {129,1}, {451,4}, {2147483647,3} } } } }, {16,{ { 8,2,{ {2554,1}, {2147483647,3} } }, { 16,4,{ {272,1}, {657,2}, {2078,1}, {2147483647,3} } }, { 32,2,{ {1081,1}, {2147483647,3} } }, { 64,2,{ {547,1}, {2147483647,3} } }, { 2147483647,5,{ {19,1}, {239,2}, {327,1}, {821,4}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_GATHER MPI_Gather 1. Binomial algorithm 2. Topology aware binomial algorithm 3. Shumilin's algorithm */ int (*intel_gather_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) ={ Coll_gather_ompi_binomial::gather, Coll_gather_ompi_binomial::gather, Coll_gather_mvapich2::gather }; intel_tuning_table_element intel_gather_table[] = { {1,{ {8,3, { {17561,3}, {44791,2}, {2147483647,3} } }, {16,7, { {16932,3}, {84425,2}, {158363,3}, {702801,2}, {1341444,3}, {2413569,2}, {2147483647,3} } }, {2147483647,4, { {47187,3}, {349696,2}, {2147483647,3}, {2147483647,1} } } } }, {2,{ {2147483647,1,{ {2147483647,3} } } } }, {4,{ {2147483647,1,{ {2147483647,3} } } } }, {8,{ { 16,1,{ {2147483647,3} } }, { 32,2,{ {9,2}, {2147483647,3} } }, { 64,2,{ {784,2}, {2147483647,3} } }, { 128,3,{ {160,3}, {655,2}, {2147483647,3} } }, { 2147483647,1,{ {2147483647,3} } } } }, {16,{ {2147483647,1,{ {2147483647,3} } } } } }; /* I_MPI_ADJUST_SCATTER MPI_Scatter 1. Binomial algorithm 2. Topology aware binomial algorithm 3. Shumilin's algorithm */ int (*intel_scatter_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) ={ Coll_scatter_ompi_binomial::scatter, Coll_scatter_ompi_binomial::scatter, Coll_scatter_mvapich2::scatter }; intel_tuning_table_element intel_scatter_table[] = { {1,{ {2,2, { {16391,1}, {2147483647,3} } }, {4,6, { {16723,3}, {153541,2}, {425631,3}, {794142,2}, {1257027,3}, {2147483647,2} } }, {8,7, { {2633,3}, {6144,2}, {14043,3}, {24576,2}, {107995,3}, {1752729,2}, {2147483647,3} } }, {16,7, { {2043,3}, {2252,2}, {17749,3}, {106020,2}, {628654,3}, {3751354,2}, {2147483647,3} } }, {2147483647,4, { {65907,3}, {245132,2}, {1042439,3}, {2147483647,2}, {2147483647,1} } } } }, {2,{ {2147483647,1,{ {2147483647,3} } } } }, {4,{ { 8,1,{ {2147483647,3} } }, { 16,2,{ {140,3}, {1302,1}, {2147483647,3} } }, { 32,2,{ {159,3}, {486,1}, {2147483647,3} } }, { 64,2,{ {149,1}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } }, {8,{ { 16,4,{ {587,1}, {1370,2}, {2102,1}, {2147483647,3} } }, { 32,3,{ {1038,1}, {2065,2}, {2147483647,3} } }, { 64,3,{ {515,1}, {2069,2}, {2147483647,3} } }, { 128,3,{ {284,1}, {796,2}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } }, {16,{ { 8,1,{ {2147483647,3} } }, { 16,3,{ {140,3}, {1302,1}, {2147483647,3} } }, { 32,3,{ {159,3}, {486,1}, {2147483647,3} } }, { 64,2,{ {149,1}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_ALLTOALLV MPI_Alltoallv 1. Isend/Irecv + waitall algorithm 2. Plum's algorithm */ int (*intel_alltoallv_functions_table[])(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) ={ Coll_alltoallv_ompi_basic_linear::alltoallv, Coll_alltoallv_bruck::alltoallv }; intel_tuning_table_element intel_alltoallv_table[] = { {1,{ {2147483647,1, { {2147483647,1} } } } }, {2,{ {2147483647,1, { {2147483647,1} } } } }, {4,{ { 8,1,{ {2147483647,1}//weirdly, intel reports the use of algo 0 here } }, { 2147483647,2,{ {4,1},//0 again {2147483647,2} } } } }, {8,{ { 16,1,{ {2147483647,1} } }, { 2147483647,2,{ {0,1},//weird again, only for 0-sized messages {2147483647,2} } } } }, {16,{ { 8,1,{ {2147483647,1}//0 } }, { 2147483647,2,{ {4,1},//0 {2147483647,2} } } } } }; //These are collected from table 3.5-2 of the Intel MPI Reference Manual #define SIZECOMP_reduce_scatter\ int total_message_size = 0;\ for (i = 0; i < comm_size; i++) { \ total_message_size += rcounts[i];\ }\ size_t block_dsize = total_message_size*dtype->size();\ #define SIZECOMP_allreduce\ size_t block_dsize =rcount * dtype->size(); #define SIZECOMP_alltoall\ size_t block_dsize =send_count * send_type->size(); #define SIZECOMP_bcast\ size_t block_dsize =count * datatype->size(); #define SIZECOMP_reduce\ size_t block_dsize =count * datatype->size(); #define SIZECOMP_barrier\ size_t block_dsize = 1; #define SIZECOMP_allgather\ size_t block_dsize =recv_count * recv_type->size(); #define SIZECOMP_allgatherv\ int total_message_size = 0;\ for (i = 0; i < comm_size; i++) { \ total_message_size += recv_count[i];\ }\ size_t block_dsize = total_message_size*recv_type->size(); #define SIZECOMP_gather\ int rank = comm->rank();\ size_t block_dsize = (send_buff == MPI_IN_PLACE || rank ==root) ?\ recv_count * recv_type->size() :\ send_count * send_type->size(); #define SIZECOMP_scatter\ int rank = comm->rank();\ size_t block_dsize = (sendbuf == MPI_IN_PLACE || rank !=root ) ?\ recvcount * recvtype->size() :\ sendcount * sendtype->size(); #define SIZECOMP_alltoallv\ size_t block_dsize = 1; #define IMPI_COLL_SELECT(cat, ret, args, args2)\ ret Coll_ ## cat ## _impi:: cat (COLL_UNPAREN args)\ {\ int comm_size = comm->size();\ int i =0;\ SIZECOMP_ ## cat\ i=0;\ int j =0, k=0;\ if(comm->get_leaders_comm()==MPI_COMM_NULL){\ comm->init_smp();\ }\ int local_size=1;\ if (comm->is_uniform()) {\ local_size = comm->get_intra_comm()->size();\ }\ while(i < INTEL_MAX_NB_PPN &&\ local_size!=intel_ ## cat ## _table[i].ppn)\ i++;\ if(i==INTEL_MAX_NB_PPN) i=0;\ while(comm_size>intel_ ## cat ## _table[i].elems[j].max_num_proc\ && j < INTEL_MAX_NB_THRESHOLDS)\ j++;\ while(block_dsize >=intel_ ## cat ## _table[i].elems[j].elems[k].max_size\ && k< intel_ ## cat ## _table[i].elems[j].num_elems)\ k++;\ return (intel_ ## cat ## _functions_table[intel_ ## cat ## _table[i].elems[j].elems[k].algo-1]\ args2);\ } COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLGATHERV_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_disps, recv_type, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLREDUCE_SIG, (sbuf, rbuf, rcount, dtype, op, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_GATHER_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_type, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLGATHER_SIG, (send_buff,send_count,send_type,recv_buff,recv_count,recv_type,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLTOALL_SIG,(send_buff, send_count, send_type, recv_buff, recv_count, recv_type,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLTOALLV_SIG, (send_buff, send_counts, send_disps, send_type, recv_buff, recv_counts, recv_disps, recv_type, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_BCAST_SIG , (buf, count, datatype, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_REDUCE_SIG,(buf,rbuf, count, datatype, op, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_REDUCE_SCATTER_SIG ,(sbuf,rbuf, rcounts,dtype,op,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_SCATTER_SIG ,(sendbuf, sendcount, sendtype,recvbuf, recvcount, recvtype,root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_BARRIER_SIG,(comm)); } } SimGrid-3.18/src/smpi/colls/smpi_openmpi_selector.cpp0000644000175000017500000006247613217757320023313 0ustar mquinsonmquinson/* selector for collective algorithms based on openmpi's default coll_tuned_decision_fixed selector */ /* Copyright (c) 2009-2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allreduce_ompi::allreduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { size_t dsize, block_dsize; int comm_size = comm->size(); const size_t intermediate_message = 10000; /** * Decision function based on MX results from the Grig cluster at UTK. * * Currently, linear, recursive doubling, and nonoverlapping algorithms * can handle both commutative and non-commutative operations. * Ring algorithm does not support non-commutative operations. */ dsize = dtype->size(); block_dsize = dsize * count; if (block_dsize < intermediate_message) { return (Coll_allreduce_rdb::allreduce (sbuf, rbuf, count, dtype, op, comm)); } if( ((op==MPI_OP_NULL) || op->is_commutative()) && (count > comm_size) ) { const size_t segment_size = 1 << 20; /* 1 MB */ if ((comm_size * segment_size >= block_dsize)) { //FIXME: ok, these are not the right algorithms, try to find closer ones // lr is a good match for allreduce_ring (difference is mainly the use of sendrecv) return Coll_allreduce_lr::allreduce(sbuf, rbuf, count, dtype, op, comm); } else { return (Coll_allreduce_ompi_ring_segmented::allreduce (sbuf, rbuf, count, dtype, op, comm /*segment_size*/)); } } return (Coll_allreduce_redbcast::allreduce(sbuf, rbuf, count, dtype, op, comm)); } int Coll_alltoall_ompi::alltoall( void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm) { int communicator_size; size_t dsize, block_dsize; communicator_size = comm->size(); /* Decision function based on measurement on Grig cluster at the University of Tennessee (2GB MX) up to 64 nodes. Has better performance for messages of intermediate sizes than the old one */ /* determine block size */ dsize = sdtype->size(); block_dsize = dsize * scount; if ((block_dsize < 200) && (communicator_size > 12)) { return Coll_alltoall_bruck::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (block_dsize < 3000) { return Coll_alltoall_basic_linear::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return Coll_alltoall_ring::alltoall (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } int Coll_alltoallv_ompi::alltoallv(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { /* For starters, just keep the original algorithm. */ return Coll_alltoallv_ompi_basic_linear::alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int Coll_barrier_ompi::barrier(MPI_Comm comm) { int communicator_size = comm->size(); if( 2 == communicator_size ) return Coll_barrier_ompi_two_procs::barrier(comm); /* * Basic optimisation. If we have a power of 2 number of nodes*/ /* * the use the recursive doubling algorithm, otherwise*/ /* * bruck is the one we want.*/ { int has_one = 0; for( ; communicator_size > 0; communicator_size >>= 1 ) { if( communicator_size & 0x1 ) { if( has_one ) return Coll_barrier_ompi_bruck::barrier(comm); has_one = 1; } } } return Coll_barrier_ompi_recursivedoubling::barrier(comm); } int Coll_bcast_ompi::bcast(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm ) { /* Decision function based on MX results for messages up to 36MB and communicator sizes up to 64 nodes */ const size_t small_message_size = 2048; const size_t intermediate_message_size = 370728; const double a_p16 = 3.2118e-6; /* [1 / byte] */ const double b_p16 = 8.7936; const double a_p64 = 2.3679e-6; /* [1 / byte] */ const double b_p64 = 1.1787; const double a_p128 = 1.6134e-6; /* [1 / byte] */ const double b_p128 = 2.1102; int communicator_size; //int segsize = 0; size_t message_size, dsize; communicator_size = comm->size(); /* else we need data size for decision function */ dsize = datatype->size(); message_size = dsize * (unsigned long)count; /* needed for decision */ /* Handle messages of small and intermediate size, and single-element broadcasts */ if ((message_size < small_message_size) || (count <= 1)) { /* Binomial without segmentation */ return Coll_bcast_binomial_tree::bcast (buff, count, datatype, root, comm); } else if (message_size < intermediate_message_size) { // SplittedBinary with 1KB segments return Coll_bcast_ompi_split_bintree::bcast(buff, count, datatype, root, comm); } //Handle large message sizes else if (communicator_size < (a_p128 * message_size + b_p128)) { //Pipeline with 128KB segments //segsize = 1024 << 7; return Coll_bcast_ompi_pipeline::bcast (buff, count, datatype, root, comm); } else if (communicator_size < 13) { // Split Binary with 8KB segments return Coll_bcast_ompi_split_bintree::bcast(buff, count, datatype, root, comm); } else if (communicator_size < (a_p64 * message_size + b_p64)) { // Pipeline with 64KB segments //segsize = 1024 << 6; return Coll_bcast_ompi_pipeline::bcast (buff, count, datatype, root, comm); } else if (communicator_size < (a_p16 * message_size + b_p16)) { //Pipeline with 16KB segments //segsize = 1024 << 4; return Coll_bcast_ompi_pipeline::bcast (buff, count, datatype, root, comm); } /* Pipeline with 8KB segments */ //segsize = 1024 << 3; return Coll_bcast_flattree_pipeline::bcast (buff, count, datatype, root, comm /*segsize*/); #if 0 /* this is based on gige measurements */ if (communicator_size < 4) { return Coll_bcast_intra_basic_linear::bcast (buff, count, datatype, root, comm, module); } if (communicator_size == 4) { if (message_size < 524288) segsize = 0; else segsize = 16384; return Coll_bcast_intra_bintree::bcast (buff, count, datatype, root, comm, module, segsize); } if (communicator_size <= 8 && message_size < 4096) { return Coll_bcast_intra_basic_linear::bcast (buff, count, datatype, root, comm, module); } if (communicator_size > 8 && message_size >= 32768 && message_size < 524288) { segsize = 16384; return Coll_bcast_intra_bintree::bcast (buff, count, datatype, root, comm, module, segsize); } if (message_size >= 524288) { segsize = 16384; return Coll_bcast_intra_pipeline::bcast (buff, count, datatype, root, comm, module, segsize); } segsize = 0; /* once tested can swap this back in */ /* return Coll_bcast_intra_bmtree::bcast (buff, count, datatype, root, comm, segsize); */ return Coll_bcast_intra_bintree::bcast (buff, count, datatype, root, comm, module, segsize); #endif /* 0 */ } int Coll_reduce_ompi::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { int communicator_size=0; //int segsize = 0; size_t message_size, dsize; const double a1 = 0.6016 / 1024.0; /* [1/B] */ const double b1 = 1.3496; const double a2 = 0.0410 / 1024.0; /* [1/B] */ const double b2 = 9.7128; const double a3 = 0.0422 / 1024.0; /* [1/B] */ const double b3 = 1.1614; //const double a4 = 0.0033 / 1024.0; [1/B] //const double b4 = 1.6761; /* no limit on # of outstanding requests */ //const int max_requests = 0; communicator_size = comm->size(); /* need data size for decision function */ dsize=datatype->size(); message_size = dsize * count; /* needed for decision */ /** * If the operation is non commutative we currently have choice of linear * or in-order binary tree algorithm. */ if ((op != MPI_OP_NULL) && not op->is_commutative()) { if ((communicator_size < 12) && (message_size < 2048)) { return Coll_reduce_ompi_basic_linear::reduce(sendbuf, recvbuf, count, datatype, op, root, comm /*, module*/); } return Coll_reduce_ompi_in_order_binary::reduce(sendbuf, recvbuf, count, datatype, op, root, comm /*, module, 0, max_requests*/); } if ((communicator_size < 8) && (message_size < 512)){ /* Linear_0K */ return Coll_reduce_ompi_basic_linear::reduce (sendbuf, recvbuf, count, datatype, op, root, comm); } else if (((communicator_size < 8) && (message_size < 20480)) || (message_size < 2048) || (count <= 1)) { /* Binomial_0K */ //segsize = 0; return Coll_reduce_ompi_binomial::reduce(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a1 * message_size + b1)) { // Binomial_1K //segsize = 1024; return Coll_reduce_ompi_binomial::reduce(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a2 * message_size + b2)) { // Pipeline_1K //segsize = 1024; return Coll_reduce_ompi_pipeline::reduce (sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a3 * message_size + b3)) { // Binary_32K //segsize = 32*1024; return Coll_reduce_ompi_binary::reduce( sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } // if (communicator_size > (a4 * message_size + b4)) { // Pipeline_32K // segsize = 32*1024; // } else { // Pipeline_64K // segsize = 64*1024; // } return Coll_reduce_ompi_pipeline::reduce (sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); #if 0 /* for small messages use linear algorithm */ if (message_size <= 4096) { segsize = 0; fanout = communicator_size - 1; /* when linear implemented or taken from basic put here, right now using chain as a linear system */ /* it is implemented and I shouldn't be calling a chain with a fanout bigger than MAXTREEFANOUT from topo.h! */ return Coll_reduce_intra_basic_linear::reduce (sendbuf, recvbuf, count, datatype, op, root, comm, module); /* return Coll_reduce_intra_chain::reduce (sendbuf, recvbuf, count, datatype, op, root, comm, segsize, fanout); */ } if (message_size < 524288) { if (message_size <= 65536 ) { segsize = 32768; fanout = 8; } else { segsize = 1024; fanout = communicator_size/2; } /* later swap this for a binary tree */ /* fanout = 2; */ return Coll_reduce_intra_chain::reduce (sendbuf, recvbuf, count, datatype, op, root, comm, module, segsize, fanout, max_requests); } segsize = 1024; return Coll_reduce_intra_pipeline::reduce (sendbuf, recvbuf, count, datatype, op, root, comm, module, segsize, max_requests); #endif /* 0 */ } int Coll_reduce_scatter_ompi::reduce_scatter( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int comm_size, i, pow2; size_t total_message_size, dsize; const double a = 0.0012; const double b = 8.0; const size_t small_message_size = 12 * 1024; const size_t large_message_size = 256 * 1024; int zerocounts = 0; XBT_DEBUG("Coll_reduce_scatter_ompi::reduce_scatter"); comm_size = comm->size(); // We need data size for decision function dsize=dtype->size(); total_message_size = 0; for (i = 0; i < comm_size; i++) { total_message_size += rcounts[i]; if (0 == rcounts[i]) { zerocounts = 1; } } if (((op != MPI_OP_NULL) && not op->is_commutative()) || (zerocounts)) { Coll_reduce_scatter_default::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); return MPI_SUCCESS; } total_message_size *= dsize; // compute the nearest power of 2 for (pow2 = 1; pow2 < comm_size; pow2 <<= 1); if ((total_message_size <= small_message_size) || ((total_message_size <= large_message_size) && (pow2 == comm_size)) || (comm_size >= a * total_message_size + b)) { return Coll_reduce_scatter_ompi_basic_recursivehalving::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); } return Coll_reduce_scatter_ompi_ring::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); } int Coll_allgather_ompi::allgather(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size; size_t dsize, total_dsize; communicator_size = comm->size(); /* Special case for 2 processes */ if (communicator_size == 2) { return Coll_allgather_pair::allgather (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm/*, module*/); } /* Determine complete data size */ dsize=sdtype->size(); total_dsize = dsize * scount * communicator_size; for (pow2_size = 1; pow2_size < communicator_size; pow2_size <<=1); /* Decision based on MX 2Gb results from Grig cluster at The University of Tennesse, Knoxville - if total message size is less than 50KB use either bruck or recursive doubling for non-power of two and power of two nodes, respectively. - else use ring and neighbor exchange algorithms for odd and even number of nodes, respectively. */ if (total_dsize < 50000) { if (pow2_size == communicator_size) { return Coll_allgather_rdb::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else { return Coll_allgather_bruck::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } } else { if (communicator_size % 2) { return Coll_allgather_ring::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else { return Coll_allgather_ompi_neighborexchange::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } } #if defined(USE_MPICH2_DECISION) /* Decision as in MPICH-2 presented in Thakur et.al. "Optimization of Collective Communication Operations in MPICH", International Journal of High Performance Computing Applications, Vol. 19, No. 1, 49-66 (2005) - for power-of-two processes and small and medium size messages (up to 512KB) use recursive doubling - for non-power-of-two processes and small messages (80KB) use bruck, - for everything else use ring. */ if ((pow2_size == communicator_size) && (total_dsize < 524288)) { return Coll_allgather_rdb::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (total_dsize <= 81920) { return Coll_allgather_bruck::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return Coll_allgather_ring::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); #endif /* defined(USE_MPICH2_DECISION) */ } int Coll_allgatherv_ompi::allgatherv(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) { int i; int communicator_size; size_t dsize, total_dsize; communicator_size = comm->size(); /* Special case for 2 processes */ if (communicator_size == 2) { return Coll_allgatherv_pair::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } /* Determine complete data size */ dsize=sdtype->size(); total_dsize = 0; for (i = 0; i < communicator_size; i++) { total_dsize += dsize * rcounts[i]; } /* Decision based on allgather decision. */ if (total_dsize < 50000) { /* return Coll_allgatherv_intra_bruck::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm, module);*/ return Coll_allgatherv_ring::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } else { if (communicator_size % 2) { return Coll_allgatherv_ring::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } else { return Coll_allgatherv_ompi_neighborexchange::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } } } int Coll_gather_ompi::gather(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { //const int large_segment_size = 32768; //const int small_segment_size = 1024; //const size_t large_block_size = 92160; const size_t intermediate_block_size = 6000; const size_t small_block_size = 1024; const int large_communicator_size = 60; const int small_communicator_size = 10; int communicator_size, rank; size_t dsize, block_size; XBT_DEBUG("smpi_coll_tuned_gather_ompi"); communicator_size = comm->size(); rank = comm->rank(); // Determine block size if (rank == root) { dsize = rdtype->size(); block_size = dsize * rcount; } else { dsize = sdtype->size(); block_size = dsize * scount; } /* if (block_size > large_block_size) {*/ /* return smpi_coll_tuned_gather_ompi_linear_sync (sbuf, scount, sdtype, */ /* rbuf, rcount, rdtype, */ /* root, comm);*/ /* } else*/ if (block_size > intermediate_block_size) { return Coll_gather_ompi_linear_sync::gather (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } else if ((communicator_size > large_communicator_size) || ((communicator_size > small_communicator_size) && (block_size < small_block_size))) { return Coll_gather_ompi_binomial::gather (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } // Otherwise, use basic linear return Coll_gather_ompi_basic_linear::gather (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } int Coll_scatter_ompi::scatter(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { const size_t small_block_size = 300; const int small_comm_size = 10; int communicator_size, rank; size_t dsize, block_size; XBT_DEBUG("Coll_scatter_ompi::scatter"); communicator_size = comm->size(); rank = comm->rank(); // Determine block size if (root == rank) { dsize=sdtype->size(); block_size = dsize * scount; } else { dsize=rdtype->size(); block_size = dsize * rcount; } if ((communicator_size > small_comm_size) && (block_size < small_block_size)) { if(rank!=root){ sbuf=xbt_malloc(rcount*rdtype->get_extent()); scount=rcount; sdtype=rdtype; } int ret=Coll_scatter_ompi_binomial::scatter (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); if(rank!=root){ xbt_free(sbuf); } return ret; } return Coll_scatter_ompi_basic_linear::scatter (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } } } SimGrid-3.18/src/smpi/colls/barrier/0000755000175000017500000000000013217757320017617 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/barrier/barrier-ompi.cpp0000644000175000017500000002547513217757320022730 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * * Additional copyrights may follow */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" /* * Barrier is ment to be a synchronous operation, as some BTLs can mark * a request done before its passed to the NIC and progress might not be made * elsewhere we cannot allow a process to exit the barrier until its last * [round of] sends are completed. * * It is last round of sends rather than 'last' individual send as each pair of * peers can use different channels/devices/btls and the receiver of one of * these sends might be forced to wait as the sender * leaves the collective and does not make progress until the next mpi call * */ /* * Simple double ring version of barrier * * synchronous gurantee made by last ring of sends are synchronous * */ namespace simgrid{ namespace smpi{ int Coll_barrier_ompi_doublering::barrier(MPI_Comm comm) { int rank, size; int left, right; rank = comm->rank(); size = comm->size(); XBT_DEBUG("ompi_coll_tuned_barrier_ompi_doublering rank %d", rank); left = ((rank-1+size)%size); right = ((rank+1)%size); if (rank > 0) { /* receive message from the left */ Request::recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* Send message to the right */ Request::send((void*)NULL, 0, MPI_BYTE, right, COLL_TAG_BARRIER, comm); /* root needs to receive from the last node */ if (rank == 0) { Request::recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* Allow nodes to exit */ if (rank > 0) { /* post Receive from left */ Request::recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* send message to the right one */ Request::send((void*)NULL, 0, MPI_BYTE, right, COLL_TAG_BARRIER, comm); /* rank 0 post receive from the last node */ if (rank == 0) { Request::recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ int Coll_barrier_ompi_recursivedoubling::barrier(MPI_Comm comm) { int rank, size, adjsize; int mask, remote; rank = comm->rank(); size = comm->size(); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_recursivedoubling rank %d", rank); /* do nearest power of 2 less than size calc */ for( adjsize = 1; adjsize <= size; adjsize <<= 1 ); adjsize >>= 1; /* if size is not exact power of two, perform an extra step */ if (adjsize != size) { if (rank >= adjsize) { /* send message to lower ranked node */ remote = rank - adjsize; Request::sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } else if (rank < (size - adjsize)) { /* receive message from high level rank */ Request::recv((void*)NULL, 0, MPI_BYTE, rank+adjsize, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } /* exchange messages */ if ( rank < adjsize ) { mask = 0x1; while ( mask < adjsize ) { remote = rank ^ mask; mask <<= 1; if (remote >= adjsize) continue; /* post receive from the remote node */ Request::sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } /* non-power of 2 case */ if (adjsize != size) { if (rank < (size - adjsize)) { /* send enter message to higher ranked node */ remote = rank + adjsize; Request::send((void*)NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm); } } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ int Coll_barrier_ompi_bruck::barrier(MPI_Comm comm) { int rank, size; int distance, to, from; rank = comm->rank(); size = comm->size(); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_bruck rank %d", rank); /* exchange data with rank-2^k and rank+2^k */ for (distance = 1; distance < size; distance <<= 1) { from = (rank + size - distance) % size; to = (rank + distance) % size; /* send message to lower ranked node */ Request::sendrecv(NULL, 0, MPI_BYTE, to, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, from, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ /* special case for two processes */ int Coll_barrier_ompi_two_procs::barrier(MPI_Comm comm) { int remote; remote = comm->rank(); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_two_procs rank %d", remote); remote = (remote + 1) & 0x1; Request::sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); return (MPI_SUCCESS); } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * GEF Oct05 after asking Jeff. */ /* copied function (with appropriate renaming) starts here */ int Coll_barrier_ompi_basic_linear::barrier(MPI_Comm comm) { int i; int size = comm->size(); int rank = comm->rank(); /* All non-root send & receive zero-length message. */ if (rank > 0) { Request::send (NULL, 0, MPI_BYTE, 0, COLL_TAG_BARRIER, comm); Request::recv (NULL, 0, MPI_BYTE, 0, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* The root collects and broadcasts the messages. */ else { MPI_Request* requests; requests = new MPI_Request[size]; for (i = 1; i < size; ++i) { requests[i] = Request::irecv(NULL, 0, MPI_BYTE, MPI_ANY_SOURCE, COLL_TAG_BARRIER, comm ); } Request::waitall( size-1, requests+1, MPI_STATUSES_IGNORE ); for (i = 1; i < size; ++i) { requests[i] = Request::isend(NULL, 0, MPI_BYTE, i, COLL_TAG_BARRIER, comm ); } Request::waitall( size-1, requests+1, MPI_STATUSES_IGNORE ); delete[] requests; } /* All done */ return MPI_SUCCESS; } /* copied function (with appropriate renaming) ends here */ /* * Another recursive doubling type algorithm, but in this case * we go up the tree and back down the tree. */ int Coll_barrier_ompi_tree::barrier(MPI_Comm comm) { int rank, size, depth; int jump, partner; rank = comm->rank(); size = comm->size(); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_tree %d", rank); /* Find the nearest power of 2 of the communicator size. */ for(depth = 1; depth < size; depth <<= 1 ); for (jump=1; jump rank) { Request::recv (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } else if (partner < rank) { Request::send (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm); } } } depth>>=1; for (jump = depth; jump>0; jump>>=1) { partner = rank ^ jump; if (!(partner & (jump-1)) && partner < size) { if (partner > rank) { Request::send (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm); } else if (partner < rank) { Request::recv (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/barrier/barrier-mvapich2-pair.cpp0000644000175000017500000000663013217757320024416 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_barrier_mvapich2_pair::barrier(MPI_Comm comm) { int size, rank; int d, dst, src; int mpi_errno = MPI_SUCCESS; size = comm->size(); /* Trivial barriers return immediately */ if (size == 1) return MPI_SUCCESS; rank = comm->rank(); int N2_prev = 1; /* N2_prev = greatest power of two < size of Comm */ for( N2_prev = 1; N2_prev <= size; N2_prev <<= 1 ); N2_prev >>= 1; int surfeit = size - N2_prev; /* Perform a combine-like operation */ if (rank < N2_prev) { if (rank < surfeit) { /* get the fanin letter from the upper "half" process: */ dst = N2_prev + rank; Request::recv(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* combine on embedded N2_prev power-of-two processes */ for (d = 1; d < N2_prev; d <<= 1) { dst = (rank ^ d); Request::sendrecv(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* fanout data to nodes above N2_prev... */ if (rank < surfeit) { dst = N2_prev + rank; Request::send(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm); } } else { /* fanin data to power of 2 subset */ src = rank - N2_prev; Request::sendrecv(NULL, 0, MPI_BYTE, src, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, src, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/allgatherv/0000755000175000017500000000000013217757317020330 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-mpich-ring.cpp0000644000175000017500000001004413217757317025217 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" /***************************************************************************** * Function: allgather_mpich_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgatherv_mpich_ring::allgatherv(void *sendbuf, int sendcount, MPI_Datatype send_type, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { char * sbuf = NULL, * rbuf = NULL; int soffset, roffset; int torecv=0, tosend=0, min, rank, comm_size; int sendnow, recvnow; int sidx, ridx; MPI_Status status; MPI_Aint recvtype_extent; int right, left, total_count, i; rank= comm->rank(); comm_size=comm->size(); recvtype_extent= recvtype->get_extent(); total_count = 0; for (i=0; i recvcounts[i]) min = recvcounts[i]; if (min * recvtype_extent < 32768*8) min = 32768*8 / recvtype_extent; /* Handle the case where the datatype extent is larger than * the pipeline size. */ if (not min) min = 1; sidx = rank; ridx = left; soffset = 0; roffset = 0; while (tosend || torecv) { /* While we have data to send or receive */ sendnow = ((recvcounts[sidx] - soffset) > min) ? min : (recvcounts[sidx] - soffset); recvnow = ((recvcounts[ridx] - roffset) > min) ? min : (recvcounts[ridx] - roffset); sbuf = (char *)recvbuf + ((displs[sidx] + soffset) * recvtype_extent); rbuf = (char *)recvbuf + ((displs[ridx] + roffset) * recvtype_extent); /* Protect against wrap-around of indices */ if (not tosend) sendnow = 0; if (not torecv) recvnow = 0; /* Communicate */ if (not sendnow && not recvnow) { /* Don't do anything. This case is possible if two * consecutive processes contribute 0 bytes each. */ } else if (not sendnow) { /* If there's no data to send, just do a recv call */ Request::recv(rbuf, recvnow, recvtype, left, COLL_TAG_ALLGATHERV, comm, &status); torecv -= recvnow; } else if (not recvnow) { /* If there's no data to receive, just do a send call */ Request::send(sbuf, sendnow, recvtype, right, COLL_TAG_ALLGATHERV, comm); tosend -= sendnow; } else { /* There's data to be sent and received */ Request::sendrecv(sbuf, sendnow, recvtype, right, COLL_TAG_ALLGATHERV, rbuf, recvnow, recvtype, left, COLL_TAG_ALLGATHERV, comm, &status); tosend -= sendnow; torecv -= recvnow; } soffset += sendnow; roffset += recvnow; if (soffset == recvcounts[sidx]) { soffset = 0; sidx = (sidx + comm_size - 1) % comm_size; } if (roffset == recvcounts[ridx]) { roffset = 0; ridx = (ridx + comm_size - 1) % comm_size; } } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-ompi-bruck.cpp0000644000175000017500000001615113217757317025237 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ #include "../colls_private.hpp" /* * ompi_coll_tuned_allgatherv_intra_bruck * * Function: allgather using O(log(N)) steps. * Accepts: Same arguments as MPI_Allgather * Returns: MPI_SUCCESS or error code * * Description: Variation to All-to-all algorithm described by Bruck et al.in * "Efficient Algorithms for All-to-all Communications * in Multiport Message-Passing Systems" * Note: Unlike in case of allgather implementation, we relay on * indexed datatype to select buffers appropriately. * The only additional memory requirement is for creation of * temporary datatypes. * Example on 7 nodes (memory lay out need not be in-order) * Initial set up: * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] [ ] * [ ] [ ] [ ] [ ] [ ] [5] [ ] * [ ] [ ] [ ] [ ] [ ] [ ] [6] * Step 0: send message to (rank - 2^0), receive message from (rank + 2^0) * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [ ] [ ] [0] * [1] [1] [ ] [ ] [ ] [ ] [ ] * [ ] [2] [2] [ ] [ ] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] [ ] * [ ] [ ] [ ] [4] [4] [ ] [ ] * [ ] [ ] [ ] [ ] [5] [5] [ ] * [ ] [ ] [ ] [ ] [ ] [6] [6] * Step 1: send message to (rank - 2^1), receive message from (rank + 2^1). * message contains all blocks from (rank) .. (rank + 2^2) with * wrap around. * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [0] [0] [0] * [1] [1] [ ] [ ] [ ] [1] [1] * [2] [2] [2] [ ] [ ] [ ] [2] * [3] [3] [3] [3] [ ] [ ] [ ] * [ ] [4] [4] [4] [4] [ ] [ ] * [ ] [ ] [5] [5] [5] [5] [ ] * [ ] [ ] [ ] [6] [6] [6] [6] * Step 2: send message to (rank - 2^2), receive message from (rank + 2^2). * message size is "all remaining blocks" * # 0 1 2 3 4 5 6 * [0] [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] [5] * [6] [6] [6] [6] [6] [6] [6] */ namespace simgrid{ namespace smpi{ int Coll_allgatherv_ompi_bruck::allgatherv(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm) { int sendto, recvfrom, blockcount, i; unsigned int distance; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; MPI_Datatype new_rdtype = MPI_DATATYPE_NULL, new_sdtype = MPI_DATATYPE_NULL; unsigned int size = comm->size(); unsigned int rank = comm->rank(); XBT_DEBUG("coll:tuned:allgather_ompi_bruck rank %u", rank); sdtype->extent(&slb, &sext); rdtype->extent(&rlb, &rext); /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to block rank of the receive buffer. */ tmprecv = (char*) rbuf + rdispls[rank] * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; Datatype::copy(tmpsend, scount, sdtype, tmprecv, rcounts[rank], rdtype); } /* Communication step: At every step i, rank r: - doubles the distance - sends message with blockcount blocks, (rbuf[rank] .. rbuf[rank + 2^i]) to rank (r - distance) - receives message of blockcount blocks, (rbuf[r + distance] ... rbuf[(r+distance) + 2^i]) from rank (r + distance) - blockcount doubles until the last step when only the remaining data is exchanged. */ int* new_rcounts = new int[4 * size]; int* new_rdispls = new_rcounts + size; int* new_scounts = new_rdispls + size; int* new_sdispls = new_scounts + size; for (distance = 1; distance < size; distance<<=1) { recvfrom = (rank + distance) % size; sendto = (rank - distance + size) % size; if (distance <= (size >> 1)) { blockcount = distance; } else { blockcount = size - distance; } /* create send and receive datatypes */ for (i = 0; i < blockcount; i++) { const int tmp_srank = (rank + i) % size; const int tmp_rrank = (recvfrom + i) % size; new_scounts[i] = rcounts[tmp_srank]; new_sdispls[i] = rdispls[tmp_srank]; new_rcounts[i] = rcounts[tmp_rrank]; new_rdispls[i] = rdispls[tmp_rrank]; } Datatype::create_indexed(blockcount, new_scounts, new_sdispls, rdtype, &new_sdtype); Datatype::create_indexed(blockcount, new_rcounts, new_rdispls, rdtype, &new_rdtype); new_sdtype->commit(); new_rdtype->commit(); /* Sendreceive */ Request::sendrecv(rbuf, 1, new_sdtype, sendto, COLL_TAG_ALLGATHERV, rbuf, 1, new_rdtype, recvfrom, COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); Datatype::unref(new_sdtype); Datatype::unref(new_rdtype); } delete[] new_rcounts; return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-mpich-rdb.cpp0000644000175000017500000001613413217757317025035 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Short or medium size message and power-of-two no. of processes. Use * recursive doubling algorithm */ #include "../colls_private.hpp" #include "smpi_status.hpp" #include namespace simgrid{ namespace smpi{ int Coll_allgatherv_mpich_rdb::allgatherv ( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { unsigned int j, i; MPI_Status status; MPI_Aint recvtype_extent, recvtype_true_extent, recvtype_true_lb; unsigned int curr_cnt, dst, total_count; void *tmp_buf, *tmp_buf_rl; unsigned int mask, dst_tree_root, my_tree_root, position, send_offset, recv_offset, last_recv_cnt=0, nprocs_completed, k, offset, tmp_mask, tree_root; unsigned int comm_size = comm->size(); unsigned int rank = comm->rank(); total_count = 0; for (i=0; iget_extent(); /* need to receive contiguously into tmp_buf because displs could make the recvbuf noncontiguous */ recvtype->extent(&recvtype_true_lb, &recvtype_true_extent); tmp_buf_rl = (void*)smpi_get_tmp_sendbuffer(total_count * std::max(recvtype_true_extent, recvtype_extent)); /* adjust for potential negative lower bound in datatype */ tmp_buf = (void *)((char*)tmp_buf_rl - recvtype_true_lb); /* copy local data into right location in tmp_buf */ position = 0; for (i=0; i> i; dst_tree_root <<= i; my_tree_root = rank >> i; my_tree_root <<= i; if (dst < comm_size) { send_offset = 0; for (j=0; j comm_size) { nprocs_completed = comm_size - my_tree_root - mask; /* nprocs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + nprocs_completed) && (dst >= tree_root + nprocs_completed)) { offset = 0; for (j=0; j<(my_tree_root+mask); j++) offset += recvcounts[j]; offset *= recvtype_extent; Request::send(((char *)tmp_buf + offset), last_recv_cnt, recvtype, dst, COLL_TAG_ALLGATHERV, comm); /* last_recv_cnt was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + nprocs_completed) && (rank >= tree_root + nprocs_completed)) { offset = 0; for (j=0; j<(my_tree_root+mask); j++) offset += recvcounts[j]; Request::recv(((char *)tmp_buf + offset * recvtype_extent), total_count - offset, recvtype, dst, COLL_TAG_ALLGATHERV, comm, &status); /* for convenience, recv is posted for a bigger amount than will be sent */ last_recv_cnt=Status::get_count(&status, recvtype); curr_cnt += last_recv_cnt; } tmp_mask >>= 1; k--; } } /* --END EXPERIMENTAL-- */ mask <<= 1; i++; } /* copy data from tmp_buf to recvbuf */ position = 0; for (j=0; jrank(); unsigned int num_procs = comm->size(); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgatherv pair algorithm can't be used with non power of two number of processes ! "); extent = send_type->get_extent(); // local send/recv Request::sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + recv_disps[rank] * extent, recv_counts[rank], recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; Request::sendrecv(send_ptr, send_count, send_type, dst, tag, recv_ptr + recv_disps[src] * extent, recv_counts[src], recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-GB.cpp0000644000175000017500000000204213217757317023451 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ // Allgather - gather/bcast algorithm int Coll_allgatherv_GB::allgatherv(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { Colls::gatherv(send_buff, send_count, send_type, recv_buff, recv_counts, recv_disps, recv_type, 0, comm); int num_procs, i, current, max = 0; num_procs = comm->size(); for (i = 0; i < num_procs; i++) { current = recv_disps[i] + recv_counts[i]; if (current > max) max = current; } Colls::bcast(recv_buff, max, recv_type, 0, comm); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-ring.cpp0000644000175000017500000001063213217757317024124 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j+ i. * Auther: Ahmad Faraj ****************************************************************************/ namespace simgrid{ namespace smpi{ int Coll_allgatherv_ring::allgatherv(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLGATHERV; MPI_Status status; char *sendptr = (char *) send_buff; char *recvptr = (char *) recv_buff; rank = comm->rank(); num_procs = comm->size(); extent = send_type->get_extent(); // local send/recv Request::sendrecv(sendptr, send_count, send_type, rank, tag, recvptr + recv_disps[rank] * extent, recv_counts[rank], recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; Request::sendrecv(sendptr, send_count, send_type, dst, tag, recvptr + recv_disps[src] * extent, recv_counts[src], recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/allgatherv/allgatherv-ompi-neighborexchange.cpp0000644000175000017500000002055013217757317027427 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * ompi_coll_tuned_allgatherv_intra_neighborexchange * * Function: allgatherv using N/2 steps (O(N)) * Accepts: Same arguments as MPI_Allgatherv * Returns: MPI_SUCCESS or error code * * Description: Neighbor Exchange algorithm for allgather adapted for * allgatherv. * Described by Chen et.al. in * "Performance Evaluation of Allgather Algorithms on * Terascale Linux Cluster with Fast Ethernet", * Proceedings of the Eighth International Conference on * High-Performance Computing inn Asia-Pacific Region * (HPCASIA'05), 2005 * * Rank r exchanges message with one of its neighbors and * forwards the data further in the next step. * * No additional memory requirements. * * Limitations: Algorithm works only on even number of processes. * For odd number of processes we switch to ring algorithm. * * Example on 6 nodes: * Initial state * # 0 1 2 3 4 5 * [0] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] * [ ] [ ] [ ] [ ] [ ] [5] * Step 0: * # 0 1 2 3 4 5 * [0] [0] [ ] [ ] [ ] [ ] * [1] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [2] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [4] * [ ] [ ] [ ] [ ] [5] [5] * Step 1: * # 0 1 2 3 4 5 * [0] [0] [0] [ ] [ ] [0] * [1] [1] [1] [ ] [ ] [1] * [ ] [2] [2] [2] [2] [ ] * [ ] [3] [3] [3] [3] [ ] * [4] [ ] [ ] [4] [4] [4] * [5] [ ] [ ] [5] [5] [5] * Step 2: * # 0 1 2 3 4 5 * [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] */ #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int Coll_allgatherv_ompi_neighborexchange::allgatherv(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm) { int line = -1; int rank, size; int neighbor[2], offset_at_step[2], recv_data_from[2], send_data_from; int i, even_rank; int err = 0; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; size = comm->size(); rank = comm->rank(); if (size % 2) { XBT_DEBUG( "coll:tuned:allgatherv_ompi_neighborexchange WARNING: odd size %d, switching to ring algorithm", size); return Coll_allgatherv_ring::allgatherv(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } XBT_DEBUG( "coll:tuned:allgatherv_ompi_neighborexchange rank %d", rank); err = sdtype->extent(&slb, &sext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = rdtype->extent(&rlb, &rext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to the appropriate block of receive buffer */ tmprecv = (char*) rbuf + rdispls[rank] * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; err = Datatype::copy(tmpsend, scount, sdtype, tmprecv, rcounts[rank], rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } /* Determine neighbors, order in which blocks will arrive, etc. */ even_rank = !(rank % 2); if (even_rank) { neighbor[0] = (rank + 1) % size; neighbor[1] = (rank - 1 + size) % size; recv_data_from[0] = rank; recv_data_from[1] = rank; offset_at_step[0] = (+2); offset_at_step[1] = (-2); } else { neighbor[0] = (rank - 1 + size) % size; neighbor[1] = (rank + 1) % size; recv_data_from[0] = neighbor[0]; recv_data_from[1] = neighbor[0]; offset_at_step[0] = (-2); offset_at_step[1] = (+2); } /* Communication loop: - First step is special: exchange a single block with neighbor[0]. - Rest of the steps: update recv_data_from according to offset, and exchange two blocks with appropriate neighbor. the send location becomes previous receve location. Note, we need to create indexed datatype to send and receive these blocks properly. */ tmprecv = (char*)rbuf + rdispls[neighbor[0]] * rext; tmpsend = (char*)rbuf + rdispls[rank] * rext; Request::sendrecv(tmpsend, rcounts[rank], rdtype, neighbor[0], COLL_TAG_ALLGATHERV, tmprecv, rcounts[neighbor[0]], rdtype, neighbor[0], COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); /* Determine initial sending counts and displacements*/ if (even_rank) { send_data_from = rank; } else { send_data_from = recv_data_from[0]; } for (i = 1; i < (size / 2); i++) { MPI_Datatype new_rdtype, new_sdtype; int new_scounts[2], new_sdispls[2], new_rcounts[2], new_rdispls[2]; const int i_parity = i % 2; recv_data_from[i_parity] = (recv_data_from[i_parity] + offset_at_step[i_parity] + size) % size; /* Create new indexed types for sending and receiving. We are sending data from ranks (send_data_from) and (send_data_from+1) We are receiving data from ranks (recv_data_from[i_parity]) and (recv_data_from[i_parity]+1). */ new_scounts[0] = rcounts[send_data_from]; new_scounts[1] = rcounts[(send_data_from + 1)]; new_sdispls[0] = rdispls[send_data_from]; new_sdispls[1] = rdispls[(send_data_from + 1)]; err = Datatype::create_indexed(2, new_scounts, new_sdispls, rdtype, &new_sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } new_sdtype->commit(); new_rcounts[0] = rcounts[recv_data_from[i_parity]]; new_rcounts[1] = rcounts[(recv_data_from[i_parity] + 1)]; new_rdispls[0] = rdispls[recv_data_from[i_parity]]; new_rdispls[1] = rdispls[(recv_data_from[i_parity] + 1)]; err = Datatype::create_indexed(2, new_rcounts, new_rdispls, rdtype, &new_rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } new_rdtype->commit(); tmprecv = (char*)rbuf; tmpsend = (char*)rbuf; /* Sendreceive */ Request::sendrecv(tmpsend, 1, new_sdtype, neighbor[i_parity], COLL_TAG_ALLGATHERV, tmprecv, 1, new_rdtype, neighbor[i_parity], COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); send_data_from = recv_data_from[i_parity]; Datatype::unref(new_sdtype); Datatype::unref(new_rdtype); } return MPI_SUCCESS; err_hndl: XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } } } SimGrid-3.18/src/smpi/colls/reduce/0000755000175000017500000000000013217757320017440 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/colls/reduce/reduce-scatter-gather.cpp0000644000175000017500000003040513217757320024330 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" /* reduce Author: MPICH */ namespace simgrid{ namespace smpi{ int Coll_reduce_scatter_gather::reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { MPI_Status status; int comm_size, rank, pof2, rem, newrank; int mask, *cnts, *disps, i, j, send_idx = 0; int recv_idx, last_idx = 0, newdst; int dst, send_cnt, recv_cnt, newroot, newdst_tree_root; int newroot_tree_root, new_count; int tag = COLL_TAG_REDUCE,temporary_buffer=0; void *send_ptr, *recv_ptr, *tmp_buf; cnts = NULL; disps = NULL; MPI_Aint extent; if (count == 0) return 0; rank = comm->rank(); comm_size = comm->size(); extent = datatype->get_extent(); /* If I'm not the root, then my recvbuf may not be valid, therefore I have to allocate a temporary one */ if (rank != root && not recvbuf) { temporary_buffer=1; recvbuf = (void *)smpi_get_tmp_recvbuffer(count * extent); } /* find nearest power-of-two less than or equal to comm_size */ pof2 = 1; while (pof2 <= comm_size) pof2 <<= 1; pof2 >>= 1; if (count < comm_size) { new_count = comm_size; send_ptr = (void *) smpi_get_tmp_sendbuffer(new_count * extent); recv_ptr = (void *) smpi_get_tmp_recvbuffer(new_count * extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(new_count * extent); memcpy(send_ptr, sendbuf != MPI_IN_PLACE ? sendbuf : recvbuf, extent * count); //if ((rank != root)) Request::sendrecv(send_ptr, new_count, datatype, rank, tag, recv_ptr, new_count, datatype, rank, tag, comm, &status); rem = comm_size - pof2; if (rank < 2 * rem) { if (rank % 2 != 0) { /* odd */ Request::send(recv_ptr, new_count, datatype, rank - 1, tag, comm); newrank = -1; } else { Request::recv(tmp_buf, count, datatype, rank + 1, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recv_ptr, &new_count, datatype); newrank = rank / 2; } } else /* rank >= 2*rem */ newrank = rank - rem; cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); if (newrank != -1) { for (i = 0; i < (pof2 - 1); i++) cnts[i] = new_count / pof2; cnts[pof2 - 1] = new_count - (new_count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ Request::sendrecv((char *) recv_ptr + disps[send_idx] * extent, send_cnt, datatype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ if(op!=MPI_OP_NULL) op->apply( (char *) tmp_buf + disps[recv_idx] * extent, (char *) recv_ptr + disps[recv_idx] * extent, &recv_cnt, datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; if (mask < pof2) last_idx = recv_idx + pof2 / mask; } } /* now do the gather to root */ if (root < 2 * rem) { if (root % 2 != 0) { if (rank == root) { /* recv */ for (i = 0; i < (pof2 - 1); i++) cnts[i] = new_count / pof2; cnts[pof2 - 1] = new_count - (new_count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; Request::recv(recv_ptr, cnts[0], datatype, 0, tag, comm, &status); newrank = 0; send_idx = 0; last_idx = 2; } else if (newrank == 0) { Request::send(recv_ptr, cnts[0], datatype, root, tag, comm); newrank = -1; } newroot = 0; } else newroot = root / 2; } else newroot = root - rem; if (newrank != -1) { j = 0; mask = 0x1; while (mask < pof2) { mask <<= 1; j++; } mask >>= 1; j--; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; if ((newdst == 0) && (root < 2 * rem) && (root % 2 != 0)) dst = root; newdst_tree_root = newdst >> j; newdst_tree_root <<= j; newroot_tree_root = newroot >> j; newroot_tree_root <<= j; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } if (newdst_tree_root == newroot_tree_root) { Request::send((char *) recv_ptr + disps[send_idx] * extent, send_cnt, datatype, dst, tag, comm); break; } else { Request::recv((char *) recv_ptr + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); } if (newrank > newdst) send_idx = recv_idx; mask >>= 1; j--; } } memcpy(recvbuf, recv_ptr, extent * count); smpi_free_tmp_buffer(send_ptr); smpi_free_tmp_buffer(recv_ptr); } else /* (count >= comm_size) */ { tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); //if ((rank != root)) Request::sendrecv(sendbuf != MPI_IN_PLACE ? sendbuf : recvbuf, count, datatype, rank, tag, recvbuf, count, datatype, rank, tag, comm, &status); rem = comm_size - pof2; if (rank < 2 * rem) { if (rank % 2 != 0) { /* odd */ Request::send(recvbuf, count, datatype, rank - 1, tag, comm); newrank = -1; } else { Request::recv(tmp_buf, count, datatype, rank + 1, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, recvbuf, &count, datatype); newrank = rank / 2; } } else /* rank >= 2*rem */ newrank = rank - rem; cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); if (newrank != -1) { for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ Request::sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ if(op!=MPI_OP_NULL) op->apply( (char *) tmp_buf + disps[recv_idx] * extent, (char *) recvbuf + disps[recv_idx] * extent, &recv_cnt, datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; if (mask < pof2) last_idx = recv_idx + pof2 / mask; } } /* now do the gather to root */ if (root < 2 * rem) { if (root % 2 != 0) { if (rank == root) { /* recv */ for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; Request::recv(recvbuf, cnts[0], datatype, 0, tag, comm, &status); newrank = 0; send_idx = 0; last_idx = 2; } else if (newrank == 0) { Request::send(recvbuf, cnts[0], datatype, root, tag, comm); newrank = -1; } newroot = 0; } else newroot = root / 2; } else newroot = root - rem; if (newrank != -1) { j = 0; mask = 0x1; while (mask < pof2) { mask <<= 1; j++; } mask >>= 1; j--; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; if ((newdst == 0) && (root < 2 * rem) && (root % 2 != 0)) dst = root; newdst_tree_root = newdst >> j; newdst_tree_root <<= j; newroot_tree_root = newroot >> j; newroot_tree_root <<= j; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } if (newdst_tree_root == newroot_tree_root) { Request::send((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, tag, comm); break; } else { Request::recv((char *) recvbuf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); } if (newrank > newdst) send_idx = recv_idx; mask >>= 1; j--; } } } if (tmp_buf) smpi_free_tmp_buffer(tmp_buf); if(temporary_buffer==1) smpi_free_tmp_buffer(recvbuf); if (cnts) free(cnts); if (disps) free(disps); return 0; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-ompi.cpp0000644000175000017500000006451313217757320022366 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" namespace simgrid{ namespace smpi{ int smpi_coll_tuned_ompi_reduce_generic( void* sendbuf, void* recvbuf, int original_count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, ompi_coll_tree_t* tree, int count_by_segment, int max_outstanding_reqs ); /** * This is a generic implementation of the reduce protocol. It used the tree * provided as an argument and execute all operations using a segment of * count times a datatype. * For the last communication it will update the count in order to limit * the number of datatype to the original count (original_count) * * Note that for non-commutative operations we cannot save memory copy * for the first block: thus we must copy sendbuf to accumbuf on intermediate * to keep the optimized loop happy. */ int smpi_coll_tuned_ompi_reduce_generic( void* sendbuf, void* recvbuf, int original_count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, ompi_coll_tree_t* tree, int count_by_segment, int max_outstanding_reqs ) { char *inbuf[2] = {NULL, NULL}, *inbuf_free[2] = {NULL, NULL}; char *accumbuf = NULL, *accumbuf_free = NULL; char *local_op_buffer = NULL, *sendtmpbuf = NULL; ptrdiff_t extent, lower_bound, segment_increment; MPI_Request reqs[2] = {MPI_REQUEST_NULL, MPI_REQUEST_NULL}; int num_segments, line, ret, segindex, i, rank; int recvcount, prevcount, inbi; /** * Determine number of segments and number of elements * sent per operation */ datatype->extent(&lower_bound, &extent); num_segments = (original_count + count_by_segment - 1) / count_by_segment; segment_increment = count_by_segment * extent; sendtmpbuf = (char*) sendbuf; if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = (char *)recvbuf; } XBT_DEBUG("coll:tuned:reduce_generic count %d, msg size %lu, segsize %lu, max_requests %d", original_count, (unsigned long)(num_segments * segment_increment), (unsigned long)segment_increment, max_outstanding_reqs); rank = comm->rank(); /* non-leaf nodes - wait for children to send me data & forward up (if needed) */ if( tree->tree_nextsize > 0 ) { ptrdiff_t true_extent, real_segment_size; true_extent=datatype->get_extent(); /* handle non existant recv buffer (i.e. its NULL) and protect the recv buffer on non-root nodes */ accumbuf = (char*)recvbuf; if( (NULL == accumbuf) || (root != rank) ) { /* Allocate temporary accumulator buffer. */ accumbuf_free = (char*)smpi_get_tmp_sendbuffer(true_extent + (original_count - 1) * extent); if (accumbuf_free == NULL) { line = __LINE__; ret = -1; goto error_hndl; } accumbuf = accumbuf_free - lower_bound; } /* If this is a non-commutative operation we must copy sendbuf to the accumbuf, in order to simplfy the loops */ if ((op != MPI_OP_NULL && not op->is_commutative())) { Datatype::copy((char*)sendtmpbuf, original_count, datatype, (char*)accumbuf, original_count, datatype); } /* Allocate two buffers for incoming segments */ real_segment_size = true_extent + (count_by_segment - 1) * extent; inbuf_free[0] = (char*) smpi_get_tmp_recvbuffer(real_segment_size); if( inbuf_free[0] == NULL ) { line = __LINE__; ret = -1; goto error_hndl; } inbuf[0] = inbuf_free[0] - lower_bound; /* if there is chance to overlap communication - allocate second buffer */ if( (num_segments > 1) || (tree->tree_nextsize > 1) ) { inbuf_free[1] = (char*) smpi_get_tmp_recvbuffer(real_segment_size); if( inbuf_free[1] == NULL ) { line = __LINE__; ret = -1; goto error_hndl; } inbuf[1] = inbuf_free[1] - lower_bound; } /* reset input buffer index and receive count */ inbi = 0; recvcount = 0; /* for each segment */ for( segindex = 0; segindex <= num_segments; segindex++ ) { prevcount = recvcount; /* recvcount - number of elements in current segment */ recvcount = count_by_segment; if( segindex == (num_segments-1) ) recvcount = original_count - count_by_segment * segindex; /* for each child */ for( i = 0; i < tree->tree_nextsize; i++ ) { /** * We try to overlap communication: * either with next segment or with the next child */ /* post irecv for current segindex on current child */ if( segindex < num_segments ) { void* local_recvbuf = inbuf[inbi]; if( 0 == i ) { /* for the first step (1st child per segment) and * commutative operations we might be able to irecv * directly into the accumulate buffer so that we can * reduce(op) this with our sendbuf in one step as * ompi_op_reduce only has two buffer pointers, * this avoids an extra memory copy. * * BUT if the operation is non-commutative or * we are root and are USING MPI_IN_PLACE this is wrong! */ if( (op==MPI_OP_NULL || op->is_commutative()) && !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_recvbuf = accumbuf + segindex * segment_increment; } } reqs[inbi]=Request::irecv(local_recvbuf, recvcount, datatype, tree->tree_next[i], COLL_TAG_REDUCE, comm ); } /* wait for previous req to complete, if any. if there are no requests reqs[inbi ^1] will be MPI_REQUEST_NULL. */ /* wait on data from last child for previous segment */ Request::waitall( 1, &reqs[inbi ^ 1], MPI_STATUSES_IGNORE ); local_op_buffer = inbuf[inbi ^ 1]; if( i > 0 ) { /* our first operation is to combine our own [sendbuf] data * with the data we recvd from down stream (but only * the operation is commutative and if we are not root and * not using MPI_IN_PLACE) */ if( 1 == i ) { if( (op==MPI_OP_NULL || op->is_commutative())&& !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_op_buffer = sendtmpbuf + segindex * segment_increment; } } /* apply operation */ if(op!=MPI_OP_NULL) op->apply( local_op_buffer, accumbuf + segindex * segment_increment, &recvcount, datatype ); } else if ( segindex > 0 ) { void* accumulator = accumbuf + (segindex-1) * segment_increment; if( tree->tree_nextsize <= 1 ) { if( (op==MPI_OP_NULL || op->is_commutative()) && !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_op_buffer = sendtmpbuf + (segindex-1) * segment_increment; } } if(op!=MPI_OP_NULL) op->apply( local_op_buffer, accumulator, &prevcount, datatype ); /* all reduced on available data this step (i) complete, * pass to the next process unless you are the root. */ if (rank != tree->tree_root) { /* send combined/accumulated data to parent */ Request::send( accumulator, prevcount, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm); } /* we stop when segindex = number of segments (i.e. we do num_segment+1 steps for pipelining */ if (segindex == num_segments) break; } /* update input buffer index */ inbi = inbi ^ 1; } /* end of for each child */ } /* end of for each segment */ /* clean up */ smpi_free_tmp_buffer(inbuf_free[0]); smpi_free_tmp_buffer(inbuf_free[1]); smpi_free_tmp_buffer(accumbuf_free); } /* leaf nodes Depending on the value of max_outstanding_reqs and the number of segments we have two options: - send all segments using blocking send to the parent, or - avoid overflooding the parent nodes by limiting the number of outstanding requests to max_oustanding_reqs. TODO/POSSIBLE IMPROVEMENT: If there is a way to determine the eager size for the current communication, synchronization should be used only when the message/segment size is smaller than the eager size. */ else { /* If the number of segments is less than a maximum number of oustanding requests or there is no limit on the maximum number of outstanding requests, we send data to the parent using blocking send */ if ((0 == max_outstanding_reqs) || (num_segments <= max_outstanding_reqs)) { segindex = 0; while ( original_count > 0) { if (original_count < count_by_segment) { count_by_segment = original_count; } Request::send((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm) ; segindex++; original_count -= count_by_segment; } } /* Otherwise, introduce flow control: - post max_outstanding_reqs non-blocking synchronous send, - for remaining segments - wait for a ssend to complete, and post the next one. - wait for all outstanding sends to complete. */ else { int creq = 0; MPI_Request* sreq = new (std::nothrow) MPI_Request[max_outstanding_reqs]; if (NULL == sreq) { line = __LINE__; ret = -1; goto error_hndl; } /* post first group of requests */ for (segindex = 0; segindex < max_outstanding_reqs; segindex++) { sreq[segindex]=Request::isend((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm); original_count -= count_by_segment; } creq = 0; while ( original_count > 0 ) { /* wait on a posted request to complete */ Request::wait(&sreq[creq], MPI_STATUS_IGNORE); sreq[creq] = MPI_REQUEST_NULL; if( original_count < count_by_segment ) { count_by_segment = original_count; } sreq[creq]=Request::isend((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm ); creq = (creq + 1) % max_outstanding_reqs; segindex++; original_count -= count_by_segment; } /* Wait on the remaining request to complete */ Request::waitall( max_outstanding_reqs, sreq, MPI_STATUSES_IGNORE ); /* free requests */ delete[] sreq; } } ompi_coll_tuned_topo_destroy_tree(&tree); return MPI_SUCCESS; error_hndl: /* error handler */ XBT_DEBUG("ERROR_HNDL: node %d file %s line %d error %d\n", rank, __FILE__, line, ret ); if( inbuf_free[0] != NULL ) free(inbuf_free[0]); if( inbuf_free[1] != NULL ) free(inbuf_free[1]); if( accumbuf_free != NULL ) free(accumbuf); return ret; } /* Attention: this version of the reduce operations does not work for: - non-commutative operations - segment sizes which are not multiplies of the extent of the datatype meaning that at least one datatype must fit in the segment ! */ int Coll_reduce_ompi_chain::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { uint32_t segsize=64*1024; int segcount = count; size_t typelng; int fanout = comm->size()/2; XBT_DEBUG("coll:tuned:reduce_intra_chain rank %d fo %d ss %5u", comm->rank(), fanout, segsize); /** * Determine number of segments and number of elements * sent per operation */ typelng = datatype->size(); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_chain(fanout, comm, root), segcount, 0 ); } int Coll_reduce_ompi_pipeline::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { uint32_t segsize; int segcount = count; size_t typelng; // COLL_TUNED_UPDATE_PIPELINE( comm, tuned_module, root ); /** * Determine number of segments and number of elements * sent per operation */ const double a2 = 0.0410 / 1024.0; /* [1/B] */ const double b2 = 9.7128; const double a4 = 0.0033 / 1024.0; /* [1/B] */ const double b4 = 1.6761; typelng= datatype->size(); int communicator_size = comm->size(); size_t message_size = typelng * count; if (communicator_size > (a2 * message_size + b2)) { // Pipeline_1K segsize = 1024; }else if (communicator_size > (a4 * message_size + b4)) { // Pipeline_32K segsize = 32*1024; } else { // Pipeline_64K segsize = 64*1024; } XBT_DEBUG("coll:tuned:reduce_intra_pipeline rank %d ss %5u", comm->rank(), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_chain( 1, comm, root), segcount, 0); } int Coll_reduce_ompi_binary::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize; int segcount = count; size_t typelng; /** * Determine number of segments and number of elements * sent per operation */ typelng=datatype->size(); // Binary_32K segsize = 32*1024; XBT_DEBUG("coll:tuned:reduce_intra_binary rank %d ss %5u", comm->rank(), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_tree(2, comm, root), segcount, 0); } int Coll_reduce_ompi_binomial::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize=0; int segcount = count; size_t typelng; const double a1 = 0.6016 / 1024.0; /* [1/B] */ const double b1 = 1.3496; // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); /** * Determine number of segments and number of elements * sent per operation */ typelng= datatype->size(); int communicator_size = comm->size(); size_t message_size = typelng * count; if (((communicator_size < 8) && (message_size < 20480)) || (message_size < 2048) || (count <= 1)) { /* Binomial_0K */ segsize = 0; } else if (communicator_size > (a1 * message_size + b1)) { // Binomial_1K segsize = 1024; } XBT_DEBUG("coll:tuned:reduce_intra_binomial rank %d ss %5u", comm->rank(), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_in_order_bmtree(comm, root), segcount, 0); } /* * reduce_intra_in_order_binary * * Function: Logarithmic reduce operation for non-commutative operations. * Acecpts: same as MPI_Reduce() * Returns: MPI_SUCCESS or error code */ int Coll_reduce_ompi_in_order_binary::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize=0; int ret; int rank, size, io_root; int segcount = count; void *use_this_sendbuf = NULL, *use_this_recvbuf = NULL; size_t typelng; rank = comm->rank(); size = comm->size(); XBT_DEBUG("coll:tuned:reduce_intra_in_order_binary rank %d ss %5u", rank, segsize); /** * Determine number of segments and number of elements * sent per operation */ typelng=datatype->size(); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); /* An in-order binary tree must use root (size-1) to preserve the order of operations. Thus, if root is not rank (size - 1), then we must handle 1. MPI_IN_PLACE option on real root, and 2. we must allocate temporary recvbuf on rank (size - 1). Note that generic function must be careful not to switch order of operations for non-commutative ops. */ io_root = size - 1; use_this_sendbuf = sendbuf; use_this_recvbuf = recvbuf; if (io_root != root) { ptrdiff_t text, ext; char *tmpbuf = NULL; ext=datatype->get_extent(); text=datatype->get_extent(); if ((root == rank) && (MPI_IN_PLACE == sendbuf)) { tmpbuf = (char *) smpi_get_tmp_sendbuffer(text + (count - 1) * ext); if (NULL == tmpbuf) { return MPI_ERR_INTERN; } Datatype::copy ( (char*)recvbuf, count, datatype, (char*)tmpbuf, count, datatype); use_this_sendbuf = tmpbuf; } else if (io_root == rank) { tmpbuf = (char *) smpi_get_tmp_recvbuffer(text + (count - 1) * ext); if (NULL == tmpbuf) { return MPI_ERR_INTERN; } use_this_recvbuf = tmpbuf; } } /* Use generic reduce with in-order binary tree topology and io_root */ ret = smpi_coll_tuned_ompi_reduce_generic( use_this_sendbuf, use_this_recvbuf, count, datatype, op, io_root, comm, ompi_coll_tuned_topo_build_in_order_bintree(comm), segcount, 0 ); if (MPI_SUCCESS != ret) { return ret; } /* Clean up */ if (io_root != root) { if (root == rank) { /* Receive result from rank io_root to recvbuf */ Request::recv(recvbuf, count, datatype, io_root, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); if (MPI_IN_PLACE == sendbuf) { smpi_free_tmp_buffer(use_this_sendbuf); } } else if (io_root == rank) { /* Send result from use_this_recvbuf to root */ Request::send(use_this_recvbuf, count, datatype, root, COLL_TAG_REDUCE, comm); smpi_free_tmp_buffer(use_this_recvbuf); } } return MPI_SUCCESS; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * GEF Oct05 after asking Jeff. */ /* copied function (with appropriate renaming) starts here */ /* * reduce_lin_intra * * Function: - reduction using O(N) algorithm * Accepts: - same as MPI_Reduce() * Returns: - MPI_SUCCESS or error code */ int Coll_reduce_ompi_basic_linear::reduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, int root, MPI_Comm comm) { int i, rank, size; ptrdiff_t true_extent, lb, extent; char *free_buffer = NULL; char *pml_buffer = NULL; char *inplace_temp = NULL; char *inbuf; /* Initialize */ rank = comm->rank(); size = comm->size(); XBT_DEBUG("coll:tuned:reduce_intra_basic_linear rank %d", rank); /* If not root, send data to the root. */ if (rank != root) { Request::send(sbuf, count, dtype, root, COLL_TAG_REDUCE, comm); return MPI_SUCCESS; } /* see discussion in ompi_coll_basic_reduce_lin_intra about extent and true extent */ /* for reducing buffer allocation lengths.... */ dtype->extent(&lb, &extent); true_extent = dtype->get_extent(); if (MPI_IN_PLACE == sbuf) { sbuf = rbuf; inplace_temp = (char*)smpi_get_tmp_recvbuffer(true_extent + (count - 1) * extent); if (NULL == inplace_temp) { return -1; } rbuf = inplace_temp - lb; } if (size > 1) { free_buffer = (char*)smpi_get_tmp_recvbuffer(true_extent + (count - 1) * extent); pml_buffer = free_buffer - lb; } /* Initialize the receive buffer. */ if (rank == (size - 1)) { Datatype::copy((char*)sbuf, count, dtype,(char*)rbuf, count, dtype); } else { Request::recv(rbuf, count, dtype, size - 1, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); } /* Loop receiving and calling reduction function (C or Fortran). */ for (i = size - 2; i >= 0; --i) { if (rank == i) { inbuf = (char*)sbuf; } else { Request::recv(pml_buffer, count, dtype, i, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); inbuf = pml_buffer; } /* Perform the reduction */ if(op!=MPI_OP_NULL) op->apply( inbuf, rbuf, &count, dtype); } if (NULL != inplace_temp) { Datatype::copy(inplace_temp, count, dtype,(char*)sbuf ,count , dtype); smpi_free_tmp_buffer(inplace_temp); } if (NULL != free_buffer) { smpi_free_tmp_buffer(free_buffer); } /* All done */ return MPI_SUCCESS; } /* copied function (with appropriate renaming) ends here */ } } SimGrid-3.18/src/smpi/colls/reduce/reduce-flat-tree.cpp0000644000175000017500000000342513217757320023300 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include namespace simgrid{ namespace smpi{ int Coll_reduce_flat_tree::reduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, int root, MPI_Comm comm) { int i, tag = COLL_TAG_REDUCE; int size; int rank; MPI_Aint extent; char *origin = 0; char *inbuf; MPI_Status status; rank = comm->rank(); size = comm->size(); /* If not root, send data to the root. */ extent = dtype->get_extent(); if (rank != root) { Request::send(sbuf, count, dtype, root, tag, comm); return 0; } /* Root receives and reduces messages. Allocate buffer to receive messages. */ if (size > 1) origin = (char *) smpi_get_tmp_recvbuffer(count * extent); /* Initialize the receive buffer. */ if (rank == (size - 1)) Request::sendrecv(sbuf, count, dtype, rank, tag, rbuf, count, dtype, rank, tag, comm, &status); else Request::recv(rbuf, count, dtype, size - 1, tag, comm, &status); /* Loop receiving and calling reduction function (C or Fortran). */ for (i = size - 2; i >= 0; --i) { if (rank == i) inbuf = static_cast(sbuf); else { Request::recv(origin, count, dtype, i, tag, comm, &status); inbuf = origin; } /* Call reduction function. */ if(op!=MPI_OP_NULL) op->apply( inbuf, rbuf, &count, dtype); } if (origin) smpi_free_tmp_buffer(origin); /* All done */ return 0; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-binomial.cpp0000644000175000017500000000543513217757320023212 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" #include //#include namespace simgrid{ namespace smpi{ int Coll_reduce_binomial::reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { MPI_Status status; int comm_size, rank; int mask, relrank, source; int dst; int tag = COLL_TAG_REDUCE; MPI_Aint extent; void *tmp_buf; MPI_Aint true_lb, true_extent; if (count == 0) return 0; rank = comm->rank(); comm_size = comm->size(); extent = datatype->get_extent(); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int is_commutative = (op==MPI_OP_NULL || op->is_commutative()); mask = 1; int lroot; if (is_commutative) lroot = root; else lroot = 0; relrank = (rank - lroot + comm_size) % comm_size; datatype->extent(&true_lb, &true_extent); /* adjust for potential negative lower bound in datatype */ tmp_buf = (void *)((char*)tmp_buf - true_lb); /* If I'm not the root, then my recvbuf may not be valid, therefore I have to allocate a temporary one */ if (rank != root) { recvbuf = (void*)smpi_get_tmp_recvbuffer(count * std::max(extent, true_extent)); recvbuf = (void *)((char*)recvbuf - true_lb); } if ((rank != root) || (sendbuf != MPI_IN_PLACE)) { Datatype::copy(sendbuf, count, datatype, recvbuf,count, datatype); } while (mask < comm_size) { /* Receive */ if ((mask & relrank) == 0) { source = (relrank | mask); if (source < comm_size) { source = (source + lroot) % comm_size; Request::recv(tmp_buf, count, datatype, source, tag, comm, &status); if (is_commutative) { if(op!=MPI_OP_NULL) op->apply( tmp_buf, recvbuf, &count, datatype); } else { if(op!=MPI_OP_NULL) op->apply( recvbuf, tmp_buf, &count, datatype); Datatype::copy(tmp_buf, count, datatype,recvbuf, count, datatype); } } } else { dst = ((relrank & (~mask)) + lroot) % comm_size; Request::send(recvbuf, count, datatype, dst, tag, comm); break; } mask <<= 1; } if (not is_commutative && (root != 0)) { if (rank == 0){ Request::send(recvbuf, count, datatype, root,tag, comm); }else if (rank == root){ Request::recv(recvbuf, count, datatype, 0, tag, comm, &status); } } if (rank != root) { smpi_free_tmp_buffer(recvbuf); } smpi_free_tmp_buffer(tmp_buf); return 0; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-NTSL.cpp0000644000175000017500000001261013217757320022171 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include int reduce_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ namespace simgrid{ namespace smpi{ int Coll_reduce_NTSL::reduce(void *buf, void *rbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int tag = COLL_TAG_REDUCE; MPI_Status status; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = datatype->get_extent(); rank = comm->rank(); size = comm->size(); /* source node and destination nodes (same through out the functions) */ int to = (rank - 1 + size) % size; int from = (rank + 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = reduce_NTSL_segment_size_in_byte / extent; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ /* if (root != 0) { if (rank == root){ Request::send(buf,count,datatype,0,tag,comm); } else if (rank == 0) { Request::recv(buf,count,datatype,root,tag,comm,&status); } } */ char *tmp_buf; tmp_buf = (char *) smpi_get_tmp_sendbuffer(count * extent); Request::sendrecv(buf, count, datatype, rank, tag, rbuf, count, datatype, rank, tag, comm, &status); /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == root) { Request::recv(tmp_buf, count, datatype, from, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuf, &count, datatype); } else if (rank == ((root - 1 + size) % size)) { Request::send(rbuf, count, datatype, to, tag, comm); } else { Request::recv(tmp_buf, count, datatype, from, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuf, &count, datatype); Request::send(rbuf, count, datatype, to, tag, comm); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } /* pipeline */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root recv data */ if (rank == root) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) tmp_buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, datatype); } } /* last node only sends data */ else if (rank == ((root - 1 + size) % size)) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = Request::isend((char *)rbuf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* intermediate nodes relay (receive, reduce, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = Request::irecv((char *) tmp_buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, datatype); send_request_array[i] = Request::isend((char *) rbuf + (i * increment), segment, datatype, to, (tag + i), comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_reduce_NTSL use default MPI_reduce."); Coll_reduce_default::reduce((char *)buf + (pipe_length * increment), (char *)rbuf + (pipe_length * increment), remainder, datatype, op, root, comm); } free(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-arrival-pattern-aware.cpp0000644000175000017500000002563613217757320025635 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "../colls_private.hpp" //#include int reduce_arrival_pattern_aware_segment_size_in_byte = 8192; #ifndef HEADER_SIZE #define HEADER_SIZE 1024 #endif #ifndef MAX_NODE #define MAX_NODE 1024 #endif namespace simgrid{ namespace smpi{ /* Non-topology-specific pipelined linear-reduce function */ int Coll_reduce_arrival_pattern_aware::reduce(void *buf, void *rbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int rank = comm->rank(); int tag = -COLL_TAG_REDUCE; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[MAX_NODE]; int size = comm->size(); int i; int sent_count; int header_index; int flag_array[MAX_NODE]; int already_received[MAX_NODE]; int header_buf[HEADER_SIZE]; char temp_buf[MAX_NODE]; MPI_Aint extent, lb; datatype->extent(&lb, &extent); /* source and destination */ int to, from; /* segment is segment size in number of elements (not bytes) */ int segment = reduce_arrival_pattern_aware_segment_size_in_byte / extent; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < MAX_NODE; i++) { already_received[i] = 0; } char *tmp_buf; tmp_buf = (char *) smpi_get_tmp_sendbuffer(count * extent); Request::sendrecv(buf, count, datatype, rank, tag, rbuf, count, datatype, rank, tag, comm, &status); /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { sent_count = 0; while (sent_count < (size - 1)) { for (i = 1; i < size; i++) { if (already_received[i] == 0) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], MPI_STATUSES_IGNORE); simcall_process_sleep(0.0001); } } header_index = 0; /* recv 1-byte message */ for (i = 0; i < size; i++) { if (i == rank) continue; /* 1-byte message arrive */ if ((flag_array[i] == 1) && (already_received[i] == 0)) { Request::recv(temp_buf, 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; //printf("root send to %d recv from %d : data = ",to,from); /* for (i=0;i<=header_index;i++) { printf("%d ",header_buf[i]); } printf("\n"); */ /* will receive in the next step */ already_received[i] = 1; } } /* send header followed by receive and reduce data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; from = header_buf[header_index - 1]; Request::send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); Request::recv(tmp_buf, count, datatype, from, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuf, &count, datatype); } } /* while loop */ } /* root */ /* non-root */ else { /* send 1-byte message to root */ Request::send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header and data, forward when required */ Request::recv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm, &status); // Request::recv(buf,count,datatype,MPI_ANY_SOURCE,tag,comm,&status); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* forward header */ if (header_buf[myordering + 1] != -1) { Request::send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); } //printf("node %d ordering %d\n",rank,myordering); /* receive, reduce, and forward data */ /* send only */ if (myordering == 0) { if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } Request::send(rbuf, count, datatype, to, tag, comm); } /* recv, reduce, send */ else { if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } from = header_buf[myordering - 1]; Request::recv(tmp_buf, count, datatype, from, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf, rbuf, &count, datatype); Request::send(rbuf, count, datatype, to, tag, comm); } } /* non-root */ } /* pipeline bcast */ else { // printf("node %d start\n",rank); send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); if (rank == 0) { sent_count = 0; int will_send[MAX_NODE]; for (i = 0; i < MAX_NODE; i++) will_send[i] = 0; /* loop until all data are received (sent) */ while (sent_count < (size - 1)) { int k; for (k = 0; k < 1; k++) { for (i = 1; i < size; i++) { //if (i == rank) //continue; if ((already_received[i] == 0) && (will_send[i] == 0)) { Request::iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; Request::recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); //printf("recv from %d\n",i); i = 1; } } } } /* end of probing */ header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { //if (i==rank) //continue; /* message arrived in this round (put in the header) */ if ((will_send[i] == 1) && (already_received[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_received[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; /* send header */ Request::send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); /* recv data - pipeline */ from = header_buf[header_index - 1]; for (i = 0; i < pipe_length; i++) { Request::recv(tmp_buf + (i * increment), segment, datatype, from, tag, comm, &status); if(op!=MPI_OP_NULL) op->apply( tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, datatype); } } } /* while loop (sent_count < size-1 ) */ } /* root */ /* none root */ else { /* send 1-byte message to root */ Request::send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header forward when required */ request=Request::irecv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm); Request::wait(&request, MPI_STATUS_IGNORE); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* send header when required */ if (header_buf[myordering + 1] != -1) { Request::send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); } /* (receive, reduce), and send data */ if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } /* send only */ if (myordering == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i]= Request::isend((char *)rbuf + (i * increment), segment, datatype, to, tag, comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } /* receive, reduce, and send */ else { from = header_buf[myordering - 1]; for (i = 0; i < pipe_length; i++) { recv_request_array[i]=Request::irecv(tmp_buf + (i * increment), segment, datatype, from, tag, comm); } for (i = 0; i < pipe_length; i++) { Request::wait(&recv_request_array[i], MPI_STATUS_IGNORE); if(op!=MPI_OP_NULL) op->apply( tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, datatype); send_request_array[i]=Request::isend((char *)rbuf + (i * increment), segment, datatype, to, tag, comm); } Request::waitall((pipe_length), send_request_array, send_status_array); } } /* non-root */ free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); //printf("node %d done\n",rank); } /* end pipeline */ /* if root is not zero send root after finished this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == 0) { Request::send(rbuf, count, datatype, root, tag, comm); } else if (rank == root) { Request::recv(rbuf, count, datatype, 0, tag, comm, &status); } } /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { Coll_reduce_default::reduce((char*)buf + (pipe_length * increment), (char*)rbuf + (pipe_length * increment), remainder, datatype, op, root, comm); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-mvapich-knomial.cpp0000644000175000017500000001570513217757320024500 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2012 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "../colls_private.hpp" #include extern int mv2_reduce_intra_knomial_factor; extern int mv2_reduce_inter_knomial_factor; #define SMPI_DEFAULT_KNOMIAL_FACTOR 4 // int mv2_reduce_knomial_factor = 2; static int MPIR_Reduce_knomial_trace(int root, int reduce_knomial_factor, MPI_Comm comm, int *dst, int *expected_send_count, int *expected_recv_count, int **src_array) { int mask=0x1, k, comm_size, src, rank, relative_rank, lroot=0; int orig_mask=0x1; int recv_iter=0, send_iter=0; int *knomial_reduce_src_array=NULL; comm_size = comm->size(); rank = comm->rank(); lroot = root; relative_rank = (rank - lroot + comm_size) % comm_size; /* First compute to whom we need to send data */ while (mask < comm_size) { if (relative_rank % (reduce_knomial_factor*mask)) { *dst = relative_rank/(reduce_knomial_factor*mask)* (reduce_knomial_factor*mask)+root; if (*dst >= comm_size) { *dst -= comm_size; } send_iter++; break; } mask *= reduce_knomial_factor; } mask /= reduce_knomial_factor; /* Now compute how many children we have in the knomial-tree */ orig_mask = mask; while (mask > 0) { for(k=1;k 0) { knomial_reduce_src_array = static_cast(smpi_get_tmp_sendbuffer(sizeof(int)*recv_iter)); } mask = orig_mask; recv_iter=0; while (mask > 0) { for(k=1;k= comm_size) { src -= comm_size; } knomial_reduce_src_array[recv_iter++] = src; } } mask /= reduce_knomial_factor; } *expected_recv_count = recv_iter; *expected_send_count = send_iter; *src_array = knomial_reduce_src_array; return 0; } namespace simgrid{ namespace smpi{ int Coll_reduce_mvapich2_knomial::reduce ( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int rank, is_commutative; int src, k; MPI_Request send_request; int index=0; MPI_Aint true_lb, true_extent, extent; MPI_Status status; int recv_iter=0, dst=-1, expected_send_count, expected_recv_count; int *src_array=NULL; void **tmp_buf=NULL; MPI_Request *requests=NULL; if (count == 0) return MPI_SUCCESS; rank = comm->rank(); /* Create a temporary buffer */ datatype->extent(&true_lb, &true_extent); extent = datatype->get_extent(); is_commutative = (op==MPI_OP_NULL || op->is_commutative()); if (rank != root) { recvbuf = (void*)smpi_get_tmp_recvbuffer(count * std::max(extent, true_extent)); recvbuf = (void *)((char*)recvbuf - true_lb); } if ((rank != root) || (sendbuf != MPI_IN_PLACE)) { mpi_errno = Datatype::copy(sendbuf, count, datatype, recvbuf, count, datatype); } if(mv2_reduce_intra_knomial_factor<0) { mv2_reduce_intra_knomial_factor = SMPI_DEFAULT_KNOMIAL_FACTOR; } if(mv2_reduce_inter_knomial_factor<0) { mv2_reduce_inter_knomial_factor = SMPI_DEFAULT_KNOMIAL_FACTOR; } MPIR_Reduce_knomial_trace(root, mv2_reduce_intra_knomial_factor, comm, &dst, &expected_send_count, &expected_recv_count, &src_array); if(expected_recv_count > 0 ) { tmp_buf = static_cast(xbt_malloc(sizeof(void *)*expected_recv_count)); requests = static_cast(xbt_malloc(sizeof(MPI_Request)*expected_recv_count)); for(k=0; k < expected_recv_count; k++ ) { tmp_buf[k] = smpi_get_tmp_sendbuffer(count * std::max(extent, true_extent)); tmp_buf[k] = (void *)((char*)tmp_buf[k] - true_lb); } while(recv_iter < expected_recv_count) { src = src_array[expected_recv_count - (recv_iter+1)]; requests[recv_iter]=Request::irecv (tmp_buf[recv_iter], count, datatype ,src, COLL_TAG_REDUCE, comm); recv_iter++; } recv_iter=0; while(recv_iter < expected_recv_count) { index=Request::waitany(expected_recv_count, requests, &status); recv_iter++; if (is_commutative) { if(op!=MPI_OP_NULL) op->apply( tmp_buf[index], recvbuf, &count, datatype); } } for(k=0; k < expected_recv_count; k++ ) { smpi_free_tmp_buffer(tmp_buf[k]); } xbt_free(tmp_buf); xbt_free(requests); } if(src_array != NULL) { xbt_free(src_array); } if(rank != root) { send_request=Request::isend(recvbuf,count, datatype, dst, COLL_TAG_REDUCE,comm); Request::waitall(1, &send_request, &status); smpi_free_tmp_buffer((void *)((char*)recvbuf + true_lb)); } /* --END ERROR HANDLING-- */ return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-mvapich-two-level.cpp0000644000175000017500000002622113217757320024757 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "../colls_private.hpp" #include #define MV2_INTRA_SHMEM_REDUCE_MSG 2048 #define mv2_g_shmem_coll_max_msg_size (1 << 17) #define SHMEM_COLL_BLOCK_SIZE (local_size * mv2_g_shmem_coll_max_msg_size) #define mv2_use_knomial_reduce 1 #define MPIR_Reduce_inter_knomial_wrapper_MV2 Coll_reduce_mvapich2_knomial::reduce #define MPIR_Reduce_intra_knomial_wrapper_MV2 Coll_reduce_mvapich2_knomial::reduce #define MPIR_Reduce_binomial_MV2 Coll_reduce_binomial::reduce #define MPIR_Reduce_redscat_gather_MV2 Coll_reduce_scatter_gather::reduce #define MPIR_Reduce_shmem_MV2 Coll_reduce_ompi_basic_linear::reduce extern int (*MV2_Reduce_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); extern int (*MV2_Reduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); /*Fn pointers for collectives */ static int (*reduce_fn)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm); namespace simgrid{ namespace smpi{ int Coll_reduce_mvapich2_two_level::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int my_rank, total_size, local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = 0; MPI_Comm shmem_comm, leader_comm; int leader_root, leader_of_root; void *in_buf = NULL, *out_buf = NULL, *tmp_buf = NULL; MPI_Aint true_lb, true_extent, extent; int is_commutative = 0, stride = 0; int intra_node_root=0; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Reduce_function==NULL) MV2_Reduce_function=Coll_reduce_mpich::reduce; if(MV2_Reduce_intra_function==NULL) MV2_Reduce_intra_function=Coll_reduce_mpich::reduce; if(comm->get_leaders_comm()==MPI_COMM_NULL){ comm->init_smp(); } my_rank = comm->rank(); total_size = comm->size(); shmem_comm = comm->get_intra_comm(); local_rank = shmem_comm->rank(); local_size = shmem_comm->size(); leader_comm = comm->get_leaders_comm(); int* leaders_map = comm->get_leaders_map(); leader_of_root = comm->group()->rank(leaders_map[root]); leader_root = leader_comm->group()->rank(leaders_map[root]); is_commutative= (op==MPI_OP_NULL || op->is_commutative()); datatype->extent(&true_lb, &true_extent); extent =datatype->get_extent(); stride = count * std::max(extent, true_extent); if (local_size == total_size) { /* First handle the case where there is only one node */ if (stride <= MV2_INTRA_SHMEM_REDUCE_MSG && is_commutative == 1) { if (local_rank == 0 ) { tmp_buf = (void*)smpi_get_tmp_sendbuffer(count * std::max(extent, true_extent)); tmp_buf = (void *) ((char *) tmp_buf - true_lb); } if (sendbuf != MPI_IN_PLACE) { in_buf = (void *)sendbuf; } else { in_buf = recvbuf; } if (local_rank == 0) { if( my_rank != root) { out_buf = tmp_buf; } else { out_buf = recvbuf; if(in_buf == out_buf) { in_buf = MPI_IN_PLACE; out_buf = recvbuf; } } } else { in_buf = (void *)sendbuf; out_buf = NULL; } if (count * (std::max(extent, true_extent)) < SHMEM_COLL_BLOCK_SIZE) { mpi_errno = MPIR_Reduce_shmem_MV2(in_buf, out_buf, count, datatype, op, 0, shmem_comm); } else { mpi_errno = MPIR_Reduce_intra_knomial_wrapper_MV2(in_buf, out_buf, count, datatype, op, 0, shmem_comm); } if (local_rank == 0 && root != my_rank) { Request::send(out_buf, count, datatype, root, COLL_TAG_REDUCE+1, comm); } if ((local_rank != 0) && (root == my_rank)) { Request::recv(recvbuf, count, datatype, leader_of_root, COLL_TAG_REDUCE+1, comm, MPI_STATUS_IGNORE); } } else { if(mv2_use_knomial_reduce == 1) { reduce_fn = &MPIR_Reduce_intra_knomial_wrapper_MV2; } else { reduce_fn = &MPIR_Reduce_binomial_MV2; } mpi_errno = reduce_fn(sendbuf, recvbuf, count, datatype, op, root, comm); } /* We are done */ if(tmp_buf!=NULL) smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); goto fn_exit; } if (local_rank == 0) { leader_comm = comm->get_leaders_comm(); if(leader_comm==MPI_COMM_NULL){ leader_comm = MPI_COMM_WORLD; } leader_comm_size = leader_comm->size(); leader_comm_rank = leader_comm->rank(); tmp_buf = (void*)smpi_get_tmp_sendbuffer(count * std::max(extent, true_extent)); tmp_buf = (void *) ((char *) tmp_buf - true_lb); } if (sendbuf != MPI_IN_PLACE) { in_buf = (void *)sendbuf; } else { in_buf = recvbuf; } if (local_rank == 0) { out_buf = tmp_buf; } else { out_buf = NULL; } if(local_size > 1) { /* Lets do the intra-node reduce operations, if we have more than one * process in the node */ /*Fix the input and outbuf buffers for the intra-node reduce. *Node leaders will have the reduced data in tmp_buf after *this step*/ if (MV2_Reduce_intra_function == & MPIR_Reduce_shmem_MV2) { if (is_commutative == 1 && (count * (std::max(extent, true_extent)) < SHMEM_COLL_BLOCK_SIZE)) { mpi_errno = MV2_Reduce_intra_function(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } else { mpi_errno = MPIR_Reduce_intra_knomial_wrapper_MV2(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } } else { mpi_errno = MV2_Reduce_intra_function(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } } else { smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); tmp_buf = in_buf; } /* Now work on the inter-leader phase. Data is in tmp_buf */ if (local_rank == 0 && leader_comm_size > 1) { /*The leader of root will have the global reduced data in tmp_buf or recv_buf at the end of the reduce */ if (leader_comm_rank == leader_root) { if (my_rank == root) { /* I am the root of the leader-comm, and the * root of the reduce op. So, I will write the * final result directly into my recvbuf */ if(tmp_buf != recvbuf) { in_buf = tmp_buf; out_buf = recvbuf; } else { in_buf = (char *)smpi_get_tmp_sendbuffer(count* datatype->get_extent()); Datatype::copy(tmp_buf, count, datatype, in_buf, count, datatype); //in_buf = MPI_IN_PLACE; out_buf = recvbuf; } } else { in_buf = (char *)smpi_get_tmp_sendbuffer(count* datatype->get_extent()); Datatype::copy(tmp_buf, count, datatype, in_buf, count, datatype); //in_buf = MPI_IN_PLACE; out_buf = tmp_buf; } } else { in_buf = tmp_buf; out_buf = NULL; } /* inter-leader communication */ mpi_errno = MV2_Reduce_function(in_buf, out_buf, count, datatype, op, leader_root, leader_comm); } if (local_size > 1) { /* Send the message to the root if the leader is not the * root of the reduce operation. The reduced data is in tmp_buf */ if ((local_rank == 0) && (root != my_rank) && (leader_root == leader_comm_rank)) { Request::send(tmp_buf, count, datatype, root, COLL_TAG_REDUCE+1, comm); } if ((local_rank != 0) && (root == my_rank)) { Request::recv(recvbuf, count, datatype, leader_of_root, COLL_TAG_REDUCE+1, comm, MPI_STATUS_IGNORE); } smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); if (leader_comm_rank == leader_root) { if (my_rank != root || (my_rank == root && tmp_buf == recvbuf)) { smpi_free_tmp_buffer(in_buf); } } } fn_exit: return mpi_errno; } } } SimGrid-3.18/src/smpi/colls/reduce/reduce-rab.cpp0000644000175000017500000011101513217757320022154 0ustar mquinsonmquinson/* extracted from mpig_myreduce.c with :3,$s/MPL/MPI/g and :%s/\\\\$/ \\/ */ /* Copyright: Rolf Rabenseifner, 1997 * Computing Center University of Stuttgart * rabenseifner@rus.uni-stuttgart.de * * The usage of this software is free, * but this header must not be removed. */ #include "../colls_private.hpp" #include #include #define REDUCE_NEW_ALWAYS 1 #ifdef CRAY # define SCR_LNG_OPTIM(bytelng) 128 + ((bytelng+127)/256) * 256; /* = 16 + multiple of 32 doubles*/ # define REDUCE_LIMITS /* values are lower limits for count arg. */ \ /* routine = reduce allreduce */ \ /* size = 2, 3,2**n,other 2, 3,2**n,other */ \ static int Lsh[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lin[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Llg[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lfp[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Ldb[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lby[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; #endif #ifdef REDUCE_NEW_ALWAYS # undef REDUCE_LIMITS # define REDUCE_LIMITS /* values are lower limits for count arg. */ \ /* routine = reduce allreduce */ \ /* size = 2, 3,2**n,other 2, 3,2**n,other */ \ static int Lsh[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lin[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Llg[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lfp[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Ldb[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lby[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; #endif /* Fast reduce and allreduce algorithm for longer buffers and predefined operations. This algorithm is explaned with the example of 13 nodes. The nodes are numbered 0, 1, 2, ... 12. The sendbuf content is a, b, c, ... m. The buffer array is notated with ABCDEFGH, this means that e.g. 'C' is the third 1/8 of the buffer. The algorithm computes { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } This is equivalent to the mostly used binary tree algorithm (e.g. used in mpich). size := number of nodes in the communicator. 2**n := the power of 2 that is next smaller or equal to the size. r := size - 2**n Exa.: size=13 ==> n=3, r=5 (i.e. size == 13 == 2**n+r == 2**3 + 5) The algoritm needs for the execution of one Colls::reduce - for r==0 exec_time = n*(L1+L2) + buf_lng * (1-1/2**n) * (T1 + T2 + O/d) - for r>0 exec_time = (n+1)*(L1+L2) + buf_lng * T1 + buf_lng*(1+1/2-1/2**n)* (T2 + O/d) with L1 = latency of a message transfer e.g. by send+recv L2 = latency of a message exchange e.g. by sendrecv T1 = additional time to transfer 1 byte (= 1/bandwidth) T2 = additional time to exchange 1 byte in both directions. O = time for one operation d = size of the datatype On a MPP system it is expected that T1==T2. In Comparison with the binary tree algorithm that needs: - for r==0 exec_time_bin_tree = n * L1 + buf_lng * n * (T1 + O/d) - for r>0 exec_time_bin_tree = (n+1)*L1 + buf_lng*(n+1)*(T1 + O/d) the new algorithm is faster if (assuming T1=T2) for n>2: for r==0: buf_lng > L2 * n / [ (n-2) * T1 + (n-1) * O/d ] for r>0: buf_lng > L2 * (n+1) / [ (n-1.5)*T1 + (n-0.5)*O/d ] for size = 5, 6, and 7: buf_lng > L2 / [0.25 * T1 + 0.58 * O/d ] for size = 4: buf_lng > L2 / [0.25 * T1 + 0.62 * O/d ] for size = 2 and 3: buf_lng > L2 / [ 0.5 * O/d ] and for O/d >> T1 we can summarize: The new algorithm is faster for about buf_lng > 2 * L2 * d / O Example L1 = L2 = 50 us, bandwidth=300MB/s, i.e. T1 = T2 = 3.3 ns/byte and 10 MFLOP/s, i.e. O = 100 ns/FLOP for double, i.e. d = 8 byte/FLOP ==> the new algorithm is faster for about buf_lng>8000 bytes i.e count > 1000 doubles ! Step 1) compute n and r Step 2) if myrank < 2*r split the buffer into ABCD and EFGH even myrank: send buffer EFGH to myrank+1 (buf_lng/2 exchange) receive buffer ABCD from myrank+1 compute op for ABCD (count/2 ops) receive result EFGH (buf_lng/2 transfer) odd myrank: send buffer ABCD to myrank+1 receive buffer EFGH from myrank+1 compute op for EFGH send result EFGH Result: node: 0 2 4 6 8 10 11 12 value: a+b c+d e+f g+h i+j k l m Step 3) The following algorithm uses only the nodes with (myrank is even && myrank < 2*r) || (myrank >= 2*r) Step 4) define NEWRANK(old) := (old < 2*r ? old/2 : old-r) define OLDRANK(new) := (new < r ? new*2 : new+r) Result: old ranks: 0 2 4 6 8 10 11 12 new ranks: 0 1 2 3 4 5 6 7 value: a+b c+d e+f g+h i+j k l m Step 5.1) Split the buffer (ABCDEFGH) in the middle, the lower half (ABCD) is computed on even (new) ranks, the opper half (EFGH) is computed on odd (new) ranks. exchange: ABCD from 1 to 0, from 3 to 2, from 5 to 4 and from 7 to 6 EFGH from 0 to 1, from 2 to 3, from 4 to 5 and from 6 to 7 (i.e. buf_lng/2 exchange) compute op in each node on its half: (i.e. count/2 ops) Result: node 0: (a+b)+(c+d) for ABCD node 1: (a+b)+(c+d) for EFGH node 2: (e+f)+(g+h) for ABCD node 3: (e+f)+(g+h) for EFGH node 4: (i+j)+ k for ABCD node 5: (i+j)+ k for EFGH node 6: l + m for ABCD node 7: l + m for EFGH Step 5.2) Same with double distance and oncemore the half of the buffer. (i.e. buf_lng/4 exchange) (i.e. count/4 ops) Result: node 0: [(a+b)+(c+d)] + [(e+f)+(g+h)] for AB node 1: [(a+b)+(c+d)] + [(e+f)+(g+h)] for EF node 2: [(a+b)+(c+d)] + [(e+f)+(g+h)] for CD node 3: [(a+b)+(c+d)] + [(e+f)+(g+h)] for GH node 4: [(i+j)+ k ] + [ l + m ] for AB node 5: [(i+j)+ k ] + [ l + m ] for EF node 6: [(i+j)+ k ] + [ l + m ] for CD node 7: [(i+j)+ k ] + [ l + m ] for GH ... Step 5.n) Same with double distance and oncemore the half of the buffer. (i.e. buf_lng/2**n exchange) (i.e. count/2**n ops) Result: 0: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for A 1: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for E 2: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for C 3: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for G 4: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for B 5: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for F 6: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for D 7: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for H For Colls::allreduce: ------------------ Step 6.1) Exchange on the last distance (2*n) the last result (i.e. buf_lng/2**n exchange) Step 6.2) Same with half the distance and double of the result (i.e. buf_lng/2**(n-1) exchange) ... Step 6.n) Same with distance 1 and double of the result (== half of the original buffer) (i.e. buf_lng/2 exchange) Result after 6.1 6.2 6.n on node 0: AB ABCD ABCDEFGH on node 1: EF EFGH ABCDEFGH on node 2: CD ABCD ABCDEFGH on node 3: GH EFGH ABCDEFGH on node 4: AB ABCD ABCDEFGH on node 5: EF EFGH ABCDEFGH on node 6: CD ABCD ABCDEFGH on node 7: GH EFGH ABCDEFGH Step 7) If r > 0 transfer the result from the even nodes with old rank < 2*r to the odd nodes with old rank < 2*r (i.e. buf_lng transfer) Result: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for ABCDEFGH on all nodes 0..12 For Colls::reduce: --------------- Step 6.0) If root node not in the list of the nodes with newranks (see steps 3+4) then send last result from node 0 to the root (buf_lng/2**n transfer) and replace the role of node 0 by the root node. Step 6.1) Send on the last distance (2**(n-1)) the last result from node with bit '2**(n-1)' in the 'new rank' unequal to that of root's new rank to the node with same '2**(n-1)' bit. (i.e. buf_lng/2**n transfer) Step 6.2) Same with half the distance and double of the result and bit '2**(n-2)' (i.e. buf_lng/2**(n-1) transfer) ... Step 6.n) Same with distance 1 and double of the result (== half of the original buffer) and bit '2**0' (i.e. buf_lng/2 transfer) Example: roots old rank: 10 roots new rank: 5 Results: 6.1 6.2 6.n action result action result action result on node 0: send A on node 1: send E on node 2: send C on node 3: send G on node 4: recv A => AB recv CD => ABCD send ABCD on node 5: recv E => EF recv GH => EFGH recv ABCD => ABCDEFGH on node 6: recv C => CD send CD on node 7: recv G => GH send GH Benchmark results on CRAY T3E ----------------------------- uname -a: (sn6307 hwwt3e 1.6.1.51 unicosmk CRAY T3E) MPI: /opt/ctl/mpt/1.1.0.3 datatype: MPI_DOUBLE Ldb[][] = {{ 896,1728, 576, 736},{ 448,1280, 512, 512}} env: export MPI_BUFFER_MAX=4099 compiled with: cc -c -O3 -h restrict=f old = binary tree protocol of the vendor new = the new protocol and its implementation mixed = 'new' is used if count > limit(datatype, communicator size) REDUCE: communicator size measurement count prot. unit 2 3 4 6 8 12 16 24 -------------------------------------------------------------------- latency 1 mixed us 20.7 35.1 35.6 49.1 49.2 61.8 62.4 74.8 old us 19.0 32.9 33.8 47.1 47.2 61.7 62.1 73.2 -------------------------------------------------------------------- bandwidth 128k mixed MB/s 75.4 34.9 49.1 27.3 41.1 24.6 38.0 23.7 (=buf_lng/time) old MB/s 28.8 16.2 16.3 11.6 11.6 8.8 8.8 7.2 ration = mixed/old 2.6 2.1 3.0 2.4 3.5 2.8 4.3 3.3 -------------------------------------------------------------------- limit doubles 896 1536 576 736 576 736 576 736 bandwidth limit mixed MB/s 35.9 20.5 18.6 12.3 13.5 9.2 10.0 8.6 old MB/s 35.9 20.3 17.8 13.0 12.5 9.6 9.2 7.8 ratio = mixed/old 1.00 1.01 1.04 0.95 1.08 0.96 1.09 1.10 communicator size measurement count prot. unit 32 48 64 96 128 192 256 --------------------------------------------------------------- latency 1 mixed us 77.8 88.5 90.6 102 1) old us 78.6 87.2 90.1 99.7 108 119 120 --------------------------------------------------------------- bandwidth 128k mixed MB/s 35.1 23.3 34.1 22.8 34.4 22.4 33.9 (=buf_lng/time) old MB/s 6.0 6.0 6.0 5.2 5.2 4.6 4.6 ration = mixed/old 5.8 3.9 5.7 4.4 6.6 4.8 7.4 5) --------------------------------------------------------------- limit doubles 576 736 576 736 576 736 576 2) bandwidth limit mixed MB/s 9.7 7.5 8.4 6.5 6.9 5.5 5.1 3) old MB/s 7.7 6.4 6.4 5.7 5.5 4.9 4.7 3) ratio = mixed/old 1.26 1.17 1.31 1.14 1.26 1.12 1.08 4) ALLREDUCE: communicator size measurement count prot. unit 2 3 4 6 8 12 16 24 -------------------------------------------------------------------- latency 1 mixed us 28.2 51.0 44.5 74.4 59.9 102 74.2 133 old us 26.9 48.3 42.4 69.8 57.6 96.0 75.7 126 -------------------------------------------------------------------- bandwidth 128k mixed MB/s 74.0 29.4 42.4 23.3 35.0 20.9 32.8 19.7 (=buf_lng/time) old MB/s 20.9 14.4 13.2 9.7 9.6 7.3 7.4 5.8 ration = mixed/old 3.5 2.0 3.2 2.4 3.6 2.9 4.4 3.4 -------------------------------------------------------------------- limit doubles 448 1280 512 512 512 512 512 512 bandwidth limit mixed MB/s 26.4 15.1 16.2 8.2 12.4 7.2 10.8 5.7 old MB/s 26.1 14.9 15.0 9.1 10.5 6.7 7.3 5.3 ratio = mixed/old 1.01 1.01 1.08 0.90 1.18 1.07 1.48 1.08 communicator size measurement count prot. unit 32 48 64 96 128 192 256 --------------------------------------------------------------- latency 1 mixed us 90.3 162 109 190 old us 92.7 152 104 179 122 225 135 --------------------------------------------------------------- bandwidth 128k mixed MB/s 31.1 19.7 30.1 19.2 29.8 18.7 29.0 (=buf_lng/time) old MB/s 5.9 4.8 5.0 4.1 4.4 3.4 3.8 ration = mixed/old 5.3 4.1 6.0 4.7 6.8 5.5 7.7 --------------------------------------------------------------- limit doubles 512 512 512 512 512 512 512 bandwidth limit mixed MB/s 6.6 5.6 5.7 3.5 4.4 3.2 3.8 old MB/s 6.3 4.2 5.4 3.6 4.4 3.1 ratio = mixed/old 1.05 1.33 1.06 0.97 1.00 1.03 Footnotes: 1) This line shows that the overhead to decide which protocol should be used can be ignored. 2) This line shows the limit for the count argument. If count < limit then the vendor protocol is used, otherwise the new protocol is used (see variable Ldb). 3) These lines show the bandwidth (=bufer length / execution time) for both protocols. 4) This line shows that the limit is choosen well if the ratio is between 0.95 (loosing 5% for buffer length near and >=limit) and 1.10 (not gaining 10% for buffer length near and b2[i]?b1[i]:b2[i]); break; \ case MPIM_MIN : \ for(i=0;ib2[i]?b1[i]:b2[i]); break; \ case MPIM_MIN : \ for(i=0;i 1) /*otherwise no balancing_protocol*/ { int ss; if (size==2) ss=0; else if (size==3) ss=1; else { int s = size; while (!(s & 1)) s = s >> 1; if (s==1) /* size == power of 2 */ ss = 2; else /* size != power of 2 */ ss = 3; } switch(op) { case MPIM_MAX: case MPIM_MIN: case MPIM_SUM: case MPIM_PROD: case MPIM_LAND: case MPIM_LOR: case MPIM_LXOR: case MPIM_BAND: case MPIM_BOR: case MPIM_BXOR: switch(datatype) { case MPIM_SHORT: case MPIM_UNSIGNED_SHORT: new_prot = count >= Lsh[is_all][ss]; break; case MPIM_INT: case MPIM_UNSIGNED: new_prot = count >= Lin[is_all][ss]; break; case MPIM_LONG: case MPIM_UNSIGNED_LONG: case MPIM_UNSIGNED_LONG_LONG: new_prot = count >= Llg[is_all][ss]; break; default: break; } default: break;} switch(op) { case MPIM_MAX: case MPIM_MIN: case MPIM_SUM: case MPIM_PROD: switch(datatype) { case MPIM_FLOAT: new_prot = count >= Lfp[is_all][ss]; break; case MPIM_DOUBLE: new_prot = count >= Ldb[is_all][ss]; break; default: break; } default: break;} switch(op) { case MPIM_BAND: case MPIM_BOR: case MPIM_BXOR: switch(datatype) { case MPIM_BYTE: new_prot = count >= Lby[is_all][ss]; break; default: break; } default: break;} # ifdef DEBUG { char *ss_str[]={"two","three","power of 2","no power of 2"}; printf("MPI_(All)Reduce: is_all=%1d, size=%1d=%s, new_prot=%1d\n", is_all, size, ss_str[ss], new_prot); fflush(stdout); } # endif } if (new_prot) { sendbuf = (char*) Sendbuf; recvbuf = (char*) Recvbuf; MPI_Comm_rank(comm, &myrank); MPI_Type_extent(mpi_datatype, &typelng); scrlng = typelng * count; #ifdef NO_CACHE_OPTIMIZATION scr1buf = new char[scrlng]; scr2buf = new char[scrlng]; scr3buf = new char[scrlng]; #else # ifdef SCR_LNG_OPTIM scrlng = SCR_LNG_OPTIM(scrlng); # endif scr2buf = new char[3 * scrlng]; /* To test cache problems. */ scr1buf = scr2buf + 1*scrlng; /* scr1buf and scr3buf must not*/ scr3buf = scr2buf + 2*scrlng; /* be used for malloc because */ /* they are interchanged below.*/ #endif computed = 0; if (is_all) root = myrank; /* for correct recvbuf handling */ /*...step 1 */ # ifdef DEBUG printf("[%2d] step 1 begin\n",myrank); fflush(stdout); # endif n = 0; x_size = 1; while (2*x_size <= size) { n++; x_size = x_size * 2; } /* x_sixe == 2**n */ r = size - x_size; /*...step 2 */ # ifdef DEBUG printf("[%2d] step 2 begin n=%d, r=%d\n",myrank,n,r);fflush(stdout); # endif if (myrank < 2*r) { if ((myrank % 2) == 0 /*even*/) { MPI_I_Sendrecv(sendbuf + (count/2)*typelng, count - count/2, mpi_datatype, myrank+1, 1220, scr2buf, count/2,mpi_datatype, myrank+1, 1221, comm, &status); MPI_I_do_op(sendbuf, scr2buf, scr1buf, count/2, datatype, op); Request::recv(scr1buf + (count/2)*typelng, count - count/2, mpi_datatype, myrank+1, 1223, comm, &status); computed = 1; # ifdef DEBUG { int i; printf("[%2d] after step 2: val=", myrank); for (i=0; i= 2*r) || ((myrank%2 == 0) && (myrank < 2*r))) mynewrank = (myrank < 2*r ? myrank/2 : myrank-r); else mynewrank = -1; if (mynewrank >= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new) < r ? (new)*2 : (new)+r) /*...step 5 */ x_start = 0; x_count = count; for (idx=0, x_base=1; idx= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new) < r ? (new)*2 : (new)+r) for(idx=n-1, x_base=x_size/2; idx>=0; idx--, x_base=x_base/2) { # ifdef DEBUG printf("[%2d](%2d) step 6.%d begin\n",myrank,mynewrank,n-idx); fflush(stdout); # endif if (((mynewrank/x_base) % 2) == 0 /*even*/) { MPI_I_Sendrecv(recvbuf + start_even[idx]*typelng, count_even[idx], mpi_datatype, OLDRANK(mynewrank+x_base),1241, recvbuf + start_odd[idx]*typelng, count_odd[idx], mpi_datatype, OLDRANK(mynewrank+x_base),1242, comm, &status); # ifdef DEBUG x_start = start_odd[idx]; x_count = count_odd[idx]; # endif } else /*odd*/ { MPI_I_Sendrecv(recvbuf + start_odd[idx]*typelng, count_odd[idx], mpi_datatype, OLDRANK(mynewrank-x_base),1242, recvbuf + start_even[idx]*typelng, count_even[idx], mpi_datatype, OLDRANK(mynewrank-x_base),1241, comm, &status); # ifdef DEBUG x_start = start_even[idx]; x_count = count_even[idx]; # endif } # ifdef DEBUG { int i; printf("[%2d](%2d) after step 6.%d end: start=%2d count=%2d val=", myrank,mynewrank,n-idx,x_start,x_count); for (i=0; i= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new)==newroot ? root \ : ((new)=0; idx--, x_base=x_base/2) { # ifdef DEBUG printf("[%2d](%2d) step 6.%d begin\n",myrank,mynewrank,n-idx); fflush(stdout); # endif if ((mynewrank & x_base) != (newroot & x_base)) { if (((mynewrank/x_base) % 2) == 0 /*even*/) { x_start = start_even[idx]; x_count = count_even[idx]; partner = mynewrank+x_base; } else { x_start = start_odd[idx]; x_count = count_odd[idx]; partner = mynewrank-x_base; } Request::send(scr1buf + x_start*typelng, x_count, mpi_datatype, OLDRANK(partner), 1244, comm); } else /*odd*/ { if (((mynewrank/x_base) % 2) == 0 /*even*/) { x_start = start_odd[idx]; x_count = count_odd[idx]; partner = mynewrank+x_base; } else { x_start = start_even[idx]; x_count = count_even[idx]; partner = mynewrank-x_base; } Request::recv((myrank==root ? recvbuf : scr1buf) + x_start*typelng, x_count, mpi_datatype, OLDRANK(partner), 1244, comm, &status); # ifdef DEBUG { int i; printf("[%2d](%2d) after step 6.%d end: start=%2d count=%2d val=", myrank,mynewrank,n-idx,x_start,x_count); for (i=0; i static inline int MPIU_Mirror_permutation(unsigned int x, int bits) { /* a mask for the high order bits that should be copied as-is */ int high_mask = ~((0x1 << bits) - 1); int retval = x & high_mask; int i; for (i = 0; i < bits; ++i) { unsigned int bitval = (x & (0x1 << i)) >> i; /* 0x1 or 0x0 */ retval |= bitval << ((bits - i) - 1); } return retval; } namespace simgrid{ namespace smpi{ int Coll_reduce_scatter_mpich_pair::reduce_scatter(void *sendbuf, void *recvbuf, int recvcounts[], MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank, comm_size, i; MPI_Aint extent, true_extent, true_lb; int *disps; void *tmp_recvbuf; int mpi_errno = MPI_SUCCESS; int total_count, dst, src; int is_commutative; comm_size = comm->size(); rank = comm->rank(); extent =datatype->get_extent(); datatype->extent(&true_lb, &true_extent); if (op->is_commutative()) { is_commutative = 1; } disps = (int*)xbt_malloc( comm_size * sizeof(int)); total_count = 0; for (i=0; iapply(tmp_recvbuf, recvbuf, &recvcounts[rank], datatype); } else { if (op != MPI_OP_NULL) op->apply(tmp_recvbuf, ((char*)recvbuf + disps[rank] * extent), &recvcounts[rank], datatype); /* we can't store the result at the beginning of recvbuf right here because there is useful data there that other process/processes need. at the end, we will copy back the result to the beginning of recvbuf. */ } } else { if (sendbuf != MPI_IN_PLACE) { if (op != MPI_OP_NULL) op->apply(recvbuf, tmp_recvbuf, &recvcounts[rank], datatype); /* copy result back into recvbuf */ mpi_errno = Datatype::copy(tmp_recvbuf, recvcounts[rank], datatype, recvbuf, recvcounts[rank], datatype); if (mpi_errno) return (mpi_errno); } else { if (op != MPI_OP_NULL) op->apply(((char*)recvbuf + disps[rank] * extent), tmp_recvbuf, &recvcounts[rank], datatype); /* copy result back into recvbuf */ mpi_errno = Datatype::copy(tmp_recvbuf, recvcounts[rank], datatype, ((char*)recvbuf + disps[rank] * extent), recvcounts[rank], datatype); if (mpi_errno) return (mpi_errno); } } } /* if MPI_IN_PLACE, move output data to the beginning of recvbuf. already done for rank 0. */ if ((sendbuf == MPI_IN_PLACE) && (rank != 0)) { mpi_errno = Datatype::copy(((char *)recvbuf + disps[rank]*extent), recvcounts[rank], datatype, recvbuf, recvcounts[rank], datatype ); if (mpi_errno) return(mpi_errno); } xbt_free(disps); smpi_free_tmp_buffer(tmp_recvbuf); return MPI_SUCCESS; } int Coll_reduce_scatter_mpich_noncomm::reduce_scatter(void *sendbuf, void *recvbuf, int recvcounts[], MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int comm_size = comm->size() ; int rank = comm->rank(); int pof2; int log2_comm_size; int i, k; int recv_offset, send_offset; int block_size, total_count, size; MPI_Aint true_extent, true_lb; int buf0_was_inout; void *tmp_buf0; void *tmp_buf1; void *result_ptr; datatype->extent(&true_lb, &true_extent); pof2 = 1; log2_comm_size = 0; while (pof2 < comm_size) { pof2 <<= 1; ++log2_comm_size; } /* begin error checking */ xbt_assert(pof2 == comm_size); /* FIXME this version only works for power of 2 procs */ for (i = 0; i < (comm_size - 1); ++i) { xbt_assert(recvcounts[i] == recvcounts[i+1]); } /* end error checking */ /* size of a block (count of datatype per block, NOT bytes per block) */ block_size = recvcounts[0]; total_count = block_size * comm_size; tmp_buf0=( void *)smpi_get_tmp_sendbuffer( true_extent * total_count); tmp_buf1=( void *)smpi_get_tmp_recvbuffer( true_extent * total_count); void *tmp_buf0_save=tmp_buf0; void *tmp_buf1_save=tmp_buf1; /* adjust for potential negative lower bound in datatype */ tmp_buf0 = (void *)((char*)tmp_buf0 - true_lb); tmp_buf1 = (void *)((char*)tmp_buf1 - true_lb); /* Copy our send data to tmp_buf0. We do this one block at a time and permute the blocks as we go according to the mirror permutation. */ for (i = 0; i < comm_size; ++i) { mpi_errno = Datatype::copy((char *)(sendbuf == MPI_IN_PLACE ? recvbuf : sendbuf) + (i * true_extent * block_size), block_size, datatype, (char *)tmp_buf0 + (MPIU_Mirror_permutation(i, log2_comm_size) * true_extent * block_size), block_size, datatype); if (mpi_errno) return(mpi_errno); } buf0_was_inout = 1; send_offset = 0; recv_offset = 0; size = total_count; for (k = 0; k < log2_comm_size; ++k) { /* use a double-buffering scheme to avoid local copies */ char *incoming_data = static_cast(buf0_was_inout ? tmp_buf1 : tmp_buf0); char *outgoing_data = static_cast(buf0_was_inout ? tmp_buf0 : tmp_buf1); int peer = rank ^ (0x1 << k); size /= 2; if (rank > peer) { /* we have the higher rank: send top half, recv bottom half */ recv_offset += size; } else { /* we have the lower rank: recv top half, send bottom half */ send_offset += size; } Request::sendrecv(outgoing_data + send_offset*true_extent, size, datatype, peer, COLL_TAG_SCATTER, incoming_data + recv_offset*true_extent, size, datatype, peer, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); /* always perform the reduction at recv_offset, the data at send_offset is now our peer's responsibility */ if (rank > peer) { /* higher ranked value so need to call op(received_data, my_data) */ if(op!=MPI_OP_NULL) op->apply( incoming_data + recv_offset*true_extent, outgoing_data + recv_offset*true_extent, &size, datatype ); /* buf0_was_inout = buf0_was_inout; */ } else { /* lower ranked value so need to call op(my_data, received_data) */ if (op != MPI_OP_NULL) op->apply(outgoing_data + recv_offset * true_extent, incoming_data + recv_offset * true_extent, &size, datatype); buf0_was_inout = not buf0_was_inout; } /* the next round of send/recv needs to happen within the block (of size "size") that we just received and reduced */ send_offset = recv_offset; } xbt_assert(size == recvcounts[rank]); /* copy the reduced data to the recvbuf */ result_ptr = (char *)(buf0_was_inout ? tmp_buf0 : tmp_buf1) + recv_offset * true_extent; mpi_errno = Datatype::copy(result_ptr, size, datatype, recvbuf, size, datatype); smpi_free_tmp_buffer(tmp_buf0_save); smpi_free_tmp_buffer(tmp_buf1_save); if (mpi_errno) return(mpi_errno); return MPI_SUCCESS; } int Coll_reduce_scatter_mpich_rdb::reduce_scatter(void *sendbuf, void *recvbuf, int recvcounts[], MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank, comm_size, i; MPI_Aint extent, true_extent, true_lb; int *disps; void *tmp_recvbuf, *tmp_results; int mpi_errno = MPI_SUCCESS; int dis[2], blklens[2], total_count, dst; int mask, dst_tree_root, my_tree_root, j, k; int received; MPI_Datatype sendtype, recvtype; int nprocs_completed, tmp_mask, tree_root, is_commutative=0; comm_size = comm->size(); rank = comm->rank(); extent =datatype->get_extent(); datatype->extent(&true_lb, &true_extent); if ((op==MPI_OP_NULL) || op->is_commutative()) { is_commutative = 1; } disps = (int*)xbt_malloc( comm_size * sizeof(int)); total_count = 0; for (i=0; i> i; dst_tree_root <<= i; my_tree_root = rank >> i; my_tree_root <<= i; /* At step 1, processes exchange (n-n/p) amount of data; at step 2, (n-2n/p) amount of data; at step 3, (n-4n/p) amount of data, and so forth. We use derived datatypes for this. At each step, a process does not need to send data indexed from my_tree_root to my_tree_root+mask-1. Similarly, a process won't receive data indexed from dst_tree_root to dst_tree_root+mask-1. */ /* calculate sendtype */ blklens[0] = blklens[1] = 0; for (j=0; jcommit(); /* calculate recvtype */ blklens[0] = blklens[1] = 0; for (j=0; jcommit(); received = 0; if (dst < comm_size) { /* tmp_results contains data to be sent in each step. Data is received in tmp_recvbuf and then accumulated into tmp_results. accumulation is done later below. */ Request::sendrecv(tmp_results, 1, sendtype, dst, COLL_TAG_SCATTER, tmp_recvbuf, 1, recvtype, dst, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); received = 1; } /* if some processes in this process's subtree in this step did not have any destination process to communicate with because of non-power-of-two, we need to send them the result. We use a logarithmic recursive-halfing algorithm for this. */ if (dst_tree_root + mask > comm_size) { nprocs_completed = comm_size - my_tree_root - mask; /* nprocs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + nprocs_completed) && (dst >= tree_root + nprocs_completed)) { /* send the current result */ Request::send(tmp_recvbuf, 1, recvtype, dst, COLL_TAG_SCATTER, comm); } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + nprocs_completed) && (rank >= tree_root + nprocs_completed)) { Request::recv(tmp_recvbuf, 1, recvtype, dst, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); received = 1; } tmp_mask >>= 1; k--; } } /* The following reduction is done here instead of after the MPIC_Sendrecv_ft or MPIC_Recv_ft above. This is because to do it above, in the noncommutative case, we would need an extra temp buffer so as not to overwrite temp_recvbuf, because temp_recvbuf may have to be communicated to other processes in the non-power-of-two case. To avoid that extra allocation, we do the reduce here. */ if (received) { if (is_commutative || (dst_tree_root < my_tree_root)) { { if (op != MPI_OP_NULL) op->apply(tmp_recvbuf, tmp_results, &blklens[0], datatype); if (op != MPI_OP_NULL) op->apply(((char*)tmp_recvbuf + dis[1] * extent), ((char*)tmp_results + dis[1] * extent), &blklens[1], datatype); } } else { { if (op != MPI_OP_NULL) op->apply(tmp_results, tmp_recvbuf, &blklens[0], datatype); if (op != MPI_OP_NULL) op->apply(((char*)tmp_results + dis[1] * extent), ((char*)tmp_recvbuf + dis[1] * extent), &blklens[1], datatype); } /* copy result back into tmp_results */ mpi_errno = Datatype::copy(tmp_recvbuf, 1, recvtype, tmp_results, 1, recvtype); if (mpi_errno) return(mpi_errno); } } Datatype::unref(sendtype); Datatype::unref(recvtype); mask <<= 1; i++; } /* now copy final results from tmp_results to recvbuf */ mpi_errno = Datatype::copy(((char *)tmp_results+disps[rank]*extent), recvcounts[rank], datatype, recvbuf, recvcounts[rank], datatype); if (mpi_errno) return(mpi_errno); xbt_free(disps); smpi_free_tmp_buffer(tmp_recvbuf); smpi_free_tmp_buffer(tmp_results); return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/colls/reduce_scatter/reduce_scatter-ompi.cpp0000644000175000017500000005104613217757320025635 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2012 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ #include "../coll_tuned_topo.hpp" #include "../colls_private.hpp" /* * Recursive-halving function is (*mostly*) copied from the BASIC coll module. * I have removed the part which handles "large" message sizes * (non-overlapping version of reduce_Scatter). */ /* copied function (with appropriate renaming) starts here */ /* * reduce_scatter_ompi_basic_recursivehalving * * Function: - reduce scatter implementation using recursive-halving * algorithm * Accepts: - same as MPI_Reduce_scatter() * Returns: - MPI_SUCCESS or error code * Limitation: - Works only for commutative operations. */ namespace simgrid{ namespace smpi{ int Coll_reduce_scatter_ompi_basic_recursivehalving::reduce_scatter(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int i, rank, size, count, err = MPI_SUCCESS; int tmp_size=1, remain = 0, tmp_rank, *disps = NULL; ptrdiff_t true_lb, true_extent, lb, extent, buf_size; char *recv_buf = NULL, *recv_buf_free = NULL; char *result_buf = NULL, *result_buf_free = NULL; /* Initialize */ rank = comm->rank(); size = comm->size(); XBT_DEBUG("coll:tuned:reduce_scatter_ompi_basic_recursivehalving, rank %d", rank); if ((op != MPI_OP_NULL && not op->is_commutative())) THROWF(arg_error,0, " reduce_scatter ompi_basic_recursivehalving can only be used for commutative operations! "); /* Find displacements and the like */ disps = (int*) xbt_malloc(sizeof(int) * size); if (NULL == disps) return MPI_ERR_OTHER; disps[0] = 0; for (i = 0; i < (size - 1); ++i) { disps[i + 1] = disps[i] + rcounts[i]; } count = disps[size - 1] + rcounts[size - 1]; /* short cut the trivial case */ if (0 == count) { xbt_free(disps); return MPI_SUCCESS; } /* get datatype information */ dtype->extent(&lb, &extent); dtype->extent(&true_lb, &true_extent); buf_size = true_extent + (ptrdiff_t)(count - 1) * extent; /* Handle MPI_IN_PLACE */ if (MPI_IN_PLACE == sbuf) { sbuf = rbuf; } /* Allocate temporary receive buffer. */ recv_buf_free = (char*) smpi_get_tmp_recvbuffer(buf_size); recv_buf = recv_buf_free - lb; if (NULL == recv_buf_free) { err = MPI_ERR_OTHER; goto cleanup; } /* allocate temporary buffer for results */ result_buf_free = (char*) smpi_get_tmp_sendbuffer(buf_size); result_buf = result_buf_free - lb; /* copy local buffer into the temporary results */ err =Datatype::copy(sbuf, count, dtype, result_buf, count, dtype); if (MPI_SUCCESS != err) goto cleanup; /* figure out power of two mapping: grow until larger than comm size, then go back one, to get the largest power of two less than comm size */ while (tmp_size <= size) tmp_size <<= 1; tmp_size >>= 1; remain = size - tmp_size; /* If comm size is not a power of two, have the first "remain" procs with an even rank send to rank + 1, leaving a power of two procs to do the rest of the algorithm */ if (rank < 2 * remain) { if ((rank & 1) == 0) { Request::send(result_buf, count, dtype, rank + 1, COLL_TAG_REDUCE_SCATTER, comm); /* we don't participate from here on out */ tmp_rank = -1; } else { Request::recv(recv_buf, count, dtype, rank - 1, COLL_TAG_REDUCE_SCATTER, comm, MPI_STATUS_IGNORE); /* integrate their results into our temp results */ if(op!=MPI_OP_NULL) op->apply( recv_buf, result_buf, &count, dtype); /* adjust rank to be the bottom "remain" ranks */ tmp_rank = rank / 2; } } else { /* just need to adjust rank to show that the bottom "even remain" ranks dropped out */ tmp_rank = rank - remain; } /* For ranks not kicked out by the above code, perform the recursive halving */ if (tmp_rank >= 0) { int *tmp_disps = NULL, *tmp_rcounts = NULL; int mask, send_index, recv_index, last_index; /* recalculate disps and rcounts to account for the special "remainder" processes that are no longer doing anything */ tmp_rcounts = (int*) xbt_malloc(tmp_size * sizeof(int)); if (NULL == tmp_rcounts) { err = MPI_ERR_OTHER; goto cleanup; } tmp_disps = (int*) xbt_malloc(tmp_size * sizeof(int)); if (NULL == tmp_disps) { xbt_free(tmp_rcounts); err = MPI_ERR_OTHER; goto cleanup; } for (i = 0 ; i < tmp_size ; ++i) { if (i < remain) { /* need to include old neighbor as well */ tmp_rcounts[i] = rcounts[i * 2 + 1] + rcounts[i * 2]; } else { tmp_rcounts[i] = rcounts[i + remain]; } } tmp_disps[0] = 0; for (i = 0; i < tmp_size - 1; ++i) { tmp_disps[i + 1] = tmp_disps[i] + tmp_rcounts[i]; } /* do the recursive halving communication. Don't use the dimension information on the communicator because I think the information is invalidated by our "shrinking" of the communicator */ mask = tmp_size >> 1; send_index = recv_index = 0; last_index = tmp_size; while (mask > 0) { int tmp_peer, peer, send_count, recv_count; MPI_Request request; tmp_peer = tmp_rank ^ mask; peer = (tmp_peer < remain) ? tmp_peer * 2 + 1 : tmp_peer + remain; /* figure out if we're sending, receiving, or both */ send_count = recv_count = 0; if (tmp_rank < tmp_peer) { send_index = recv_index + mask; for (i = send_index ; i < last_index ; ++i) { send_count += tmp_rcounts[i]; } for (i = recv_index ; i < send_index ; ++i) { recv_count += tmp_rcounts[i]; } } else { recv_index = send_index + mask; for (i = send_index ; i < recv_index ; ++i) { send_count += tmp_rcounts[i]; } for (i = recv_index ; i < last_index ; ++i) { recv_count += tmp_rcounts[i]; } } /* actual data transfer. Send from result_buf, receive into recv_buf */ if (send_count > 0 && recv_count != 0) { request=Request::irecv(recv_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, recv_count, dtype, peer, COLL_TAG_REDUCE_SCATTER, comm); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } if (recv_count > 0 && send_count != 0) { Request::send(result_buf + (ptrdiff_t)tmp_disps[send_index] * extent, send_count, dtype, peer, COLL_TAG_REDUCE_SCATTER, comm); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } if (send_count > 0 && recv_count != 0) { Request::wait(&request, MPI_STATUS_IGNORE); } /* if we received something on this step, push it into the results buffer */ if (recv_count > 0) { if(op!=MPI_OP_NULL) op->apply( recv_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, result_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, &recv_count, dtype); } /* update for next iteration */ send_index = recv_index; last_index = recv_index + mask; mask >>= 1; } /* copy local results from results buffer into real receive buffer */ if (0 != rcounts[rank]) { err = Datatype::copy(result_buf + disps[rank] * extent, rcounts[rank], dtype, rbuf, rcounts[rank], dtype); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } xbt_free(tmp_rcounts); xbt_free(tmp_disps); } /* Now fix up the non-power of two case, by having the odd procs send the even procs the proper results */ if (rank < (2 * remain)) { if ((rank & 1) == 0) { if (rcounts[rank]) { Request::recv(rbuf, rcounts[rank], dtype, rank + 1, COLL_TAG_REDUCE_SCATTER, comm, MPI_STATUS_IGNORE); } } else { if (rcounts[rank - 1]) { Request::send(result_buf + disps[rank - 1] * extent, rcounts[rank - 1], dtype, rank - 1, COLL_TAG_REDUCE_SCATTER, comm); } } } cleanup: if (NULL != disps) xbt_free(disps); if (NULL != recv_buf_free) smpi_free_tmp_buffer(recv_buf_free); if (NULL != result_buf_free) smpi_free_tmp_buffer(result_buf_free); return err; } /* copied function (with appropriate renaming) ends here */ /* * Coll_reduce_scatter_ompi_ring::reduce_scatter * * Function: Ring algorithm for reduce_scatter operation * Accepts: Same as MPI_Reduce_scatter() * Returns: MPI_SUCCESS or error code * * Description: Implements ring algorithm for reduce_scatter: * the block sizes defined in rcounts are exchanged and 8 updated until they reach proper destination. * Algorithm requires 2 * max(rcounts) extra buffering * * Limitations: The algorithm DOES NOT preserve order of operations so it * can be used only for commutative operations. * Example on 5 nodes: * Initial state * # 0 1 2 3 4 * [00] [10] -> [20] [30] [40] * [01] [11] [21] -> [31] [41] * [02] [12] [22] [32] -> [42] * -> [03] [13] [23] [33] [43] --> .. * [04] -> [14] [24] [34] [44] * * COMPUTATION PHASE * Step 0: rank r sends block (r-1) to rank (r+1) and * receives block (r+1) from rank (r-1) [with wraparound]. * # 0 1 2 3 4 * [00] [10] [10+20] -> [30] [40] * [01] [11] [21] [21+31] -> [41] * -> [02] [12] [22] [32] [32+42] -->.. * [43+03] -> [13] [23] [33] [43] * [04] [04+14] -> [24] [34] [44] * * Step 1: * # 0 1 2 3 4 * [00] [10] [10+20] [10+20+30] -> [40] * -> [01] [11] [21] [21+31] [21+31+41] -> * [32+42+02] -> [12] [22] [32] [32+42] * [03] [43+03+13] -> [23] [33] [43] * [04] [04+14] [04+14+24] -> [34] [44] * * Step 2: * # 0 1 2 3 4 * -> [00] [10] [10+20] [10+20+30] [10+20+30+40] -> * [21+31+41+01]-> [11] [21] [21+31] [21+31+41] * [32+42+02] [32+42+02+12]-> [22] [32] [32+42] * [03] [43+03+13] [43+03+13+23]-> [33] [43] * [04] [04+14] [04+14+24] [04+14+24+34] -> [44] * * Step 3: * # 0 1 2 3 4 * [10+20+30+40+00] [10] [10+20] [10+20+30] [10+20+30+40] * [21+31+41+01] [21+31+41+01+11] [21] [21+31] [21+31+41] * [32+42+02] [32+42+02+12] [32+42+02+12+22] [32] [32+42] * [03] [43+03+13] [43+03+13+23] [43+03+13+23+33] [43] * [04] [04+14] [04+14+24] [04+14+24+34] [04+14+24+34+44] * DONE :) * */ int Coll_reduce_scatter_ompi_ring::reduce_scatter(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int ret, line, rank, size, i, k, recv_from, send_to, total_count, max_block_count; int inbi, *displs = NULL; char *tmpsend = NULL, *tmprecv = NULL, *accumbuf = NULL, *accumbuf_free = NULL; char *inbuf_free[2] = {NULL, NULL}, *inbuf[2] = {NULL, NULL}; ptrdiff_t true_lb, true_extent, lb, extent, max_real_segsize; MPI_Request reqs[2] = {NULL, NULL}; size = comm->size(); rank = comm->rank(); XBT_DEBUG( "coll:tuned:reduce_scatter_ompi_ring rank %d, size %d", rank, size); /* Determine the maximum number of elements per node, corresponding block size, and displacements array. */ displs = (int*) xbt_malloc(size * sizeof(int)); if (NULL == displs) { ret = -1; line = __LINE__; goto error_hndl; } displs[0] = 0; total_count = rcounts[0]; max_block_count = rcounts[0]; for (i = 1; i < size; i++) { displs[i] = total_count; total_count += rcounts[i]; if (max_block_count < rcounts[i]) max_block_count = rcounts[i]; } /* Special case for size == 1 */ if (1 == size) { if (MPI_IN_PLACE != sbuf) { ret = Datatype::copy((char*)sbuf, total_count, dtype, (char*)rbuf, total_count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } xbt_free(displs); return MPI_SUCCESS; } /* Allocate and initialize temporary buffers, we need: - a temporary buffer to perform reduction (size total_count) since rbuf can be of rcounts[rank] size. - up to two temporary buffers used for communication/computation overlap. */ dtype->extent(&lb, &extent); dtype->extent(&true_lb, &true_extent); max_real_segsize = true_extent + (ptrdiff_t)(max_block_count - 1) * extent; accumbuf_free = (char*)smpi_get_tmp_recvbuffer(true_extent + (ptrdiff_t)(total_count - 1) * extent); if (NULL == accumbuf_free) { ret = -1; line = __LINE__; goto error_hndl; } accumbuf = accumbuf_free - lb; inbuf_free[0] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf_free[0]) { ret = -1; line = __LINE__; goto error_hndl; } inbuf[0] = inbuf_free[0] - lb; if (size > 2) { inbuf_free[1] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf_free[1]) { ret = -1; line = __LINE__; goto error_hndl; } inbuf[1] = inbuf_free[1] - lb; } /* Handle MPI_IN_PLACE for size > 1 */ if (MPI_IN_PLACE == sbuf) { sbuf = rbuf; } ret = Datatype::copy((char*)sbuf, total_count, dtype, accumbuf, total_count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } /* Computation loop */ /* For each of the remote nodes: - post irecv for block (r-2) from (r-1) with wrap around - send block (r-1) to (r+1) - in loop for every step k = 2 .. n - post irecv for block (r - 1 + n - k) % n - wait on block (r + n - k) % n to arrive - compute on block (r + n - k ) % n - send block (r + n - k) % n - wait on block (r) - compute on block (r) - copy block (r) to rbuf Note that we must be careful when computing the begining of buffers and for send operations and computation we must compute the exact block size. */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; inbi = 0; /* Initialize first receive from the neighbor on the left */ reqs[inbi]=Request::irecv(inbuf[inbi], max_block_count, dtype, recv_from, COLL_TAG_REDUCE_SCATTER, comm ); tmpsend = accumbuf + (ptrdiff_t)displs[recv_from] * extent; Request::send(tmpsend, rcounts[recv_from], dtype, send_to, COLL_TAG_REDUCE_SCATTER, comm); for (k = 2; k < size; k++) { const int prevblock = (rank + size - k) % size; inbi = inbi ^ 0x1; /* Post irecv for the current block */ reqs[inbi]=Request::irecv(inbuf[inbi], max_block_count, dtype, recv_from, COLL_TAG_REDUCE_SCATTER, comm ); /* Wait on previous block to arrive */ Request::wait(&reqs[inbi ^ 0x1], MPI_STATUS_IGNORE); /* Apply operation on previous block: result goes to rbuf rbuf[prevblock] = inbuf[inbi ^ 0x1] (op) rbuf[prevblock] */ tmprecv = accumbuf + (ptrdiff_t)displs[prevblock] * extent; if(op!=MPI_OP_NULL) op->apply( inbuf[inbi ^ 0x1], tmprecv, &(rcounts[prevblock]), dtype); /* send previous block to send_to */ Request::send(tmprecv, rcounts[prevblock], dtype, send_to, COLL_TAG_REDUCE_SCATTER, comm); } /* Wait on the last block to arrive */ Request::wait(&reqs[inbi], MPI_STATUS_IGNORE); /* Apply operation on the last block (my block) rbuf[rank] = inbuf[inbi] (op) rbuf[rank] */ tmprecv = accumbuf + (ptrdiff_t)displs[rank] * extent; if(op!=MPI_OP_NULL) op->apply( inbuf[inbi], tmprecv, &(rcounts[rank]), dtype); /* Copy result from tmprecv to rbuf */ ret = Datatype::copy(tmprecv, rcounts[rank], dtype, (char*)rbuf, rcounts[rank], dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } if (NULL != displs) xbt_free(displs); if (NULL != accumbuf_free) smpi_free_tmp_buffer(accumbuf_free); if (NULL != inbuf_free[0]) smpi_free_tmp_buffer(inbuf_free[0]); if (NULL != inbuf_free[1]) smpi_free_tmp_buffer(inbuf_free[1]); return MPI_SUCCESS; error_hndl: XBT_DEBUG( "%s:%4d\tRank %d Error occurred %d\n", __FILE__, line, rank, ret); if (NULL != displs) xbt_free(displs); if (NULL != accumbuf_free) smpi_free_tmp_buffer(accumbuf_free); if (NULL != inbuf_free[0]) smpi_free_tmp_buffer(inbuf_free[0]); if (NULL != inbuf_free[1]) smpi_free_tmp_buffer(inbuf_free[1]); return ret; } } } SimGrid-3.18/src/smpi/colls/smpi_mpich_selector.cpp0000644000175000017500000007014113217757320022730 0ustar mquinsonmquinson/* selector for collective algorithms based on mpich decision logic */ /* Copyright (c) 2009-2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.hpp" /* This is the default implementation of allreduce. The algorithm is: Algorithm: MPI_Allreduce For the heterogeneous case, we call MPI_Reduce followed by MPI_Bcast in order to meet the requirement that all processes must have the same result. For the homogeneous case, we use the following algorithms. For long messages and for builtin ops and if count >= pof2 (where pof2 is the nearest power-of-two less than or equal to the number of processes), we use Rabenseifner's algorithm (see http://www.hlrs.de/mpi/myreduce.html). This algorithm implements the allreduce in two steps: first a reduce-scatter, followed by an allgather. A recursive-halving algorithm (beginning with processes that are distance 1 apart) is used for the reduce-scatter, and a recursive doubling algorithm is used for the allgather. The non-power-of-two case is handled by dropping to the nearest lower power-of-two: the first few even-numbered processes send their data to their right neighbors (rank+1), and the reduce-scatter and allgather happen among the remaining power-of-two processes. At the end, the first few even-numbered processes get the result from their right neighbors. For the power-of-two case, the cost for the reduce-scatter is lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the allgather lgp.alpha + n.((p-1)/p).beta. Therefore, the total cost is: Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma For the non-power-of-two case, Cost = (2.floor(lgp)+2).alpha + (2.((p-1)/p) + 2).n.beta + n.(1+(p-1)/p).gamma For short messages, for user-defined ops, and for count < pof2 we use a recursive doubling algorithm (similar to the one in MPI_Allgather). We use this algorithm in the case of user-defined ops because in this case derived datatypes are allowed, and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky. Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma Possible improvements: End Algorithm: MPI_Allreduce */ namespace simgrid{ namespace smpi{ int Coll_allreduce_mpich::allreduce(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { size_t dsize, block_dsize; int comm_size = comm->size(); const size_t large_message = 2048; //MPIR_PARAM_ALLREDUCE_SHORT_MSG_SIZE dsize = dtype->size(); block_dsize = dsize * count; /* find nearest power-of-two less than or equal to comm_size */ int pof2 = 1; while (pof2 <= comm_size) pof2 <<= 1; pof2 >>=1; if (block_dsize > large_message && count >= pof2 && (op==MPI_OP_NULL || op->is_commutative())) { //for long messages return (Coll_allreduce_rab_rdb::allreduce (sbuf, rbuf, count, dtype, op, comm)); }else { //for short ones and count < pof2 return (Coll_allreduce_rdb::allreduce (sbuf, rbuf, count, dtype, op, comm)); } } /* This is the default implementation of alltoall. The algorithm is: Algorithm: MPI_Alltoall We use four algorithms for alltoall. For short messages and (comm_size >= 8), we use the algorithm by Jehoshua Bruck et al, IEEE TPDS, Nov. 1997. It is a store-and-forward algorithm that takes lgp steps. Because of the extra communication, the bandwidth requirement is (n/2).lgp.beta. Cost = lgp.alpha + (n/2).lgp.beta where n is the total amount of data a process needs to send to all other processes. For medium size messages and (short messages for comm_size < 8), we use an algorithm that posts all irecvs and isends and then does a waitall. We scatter the order of sources and destinations among the processes, so that all processes don't try to send/recv to/from the same process at the same time. *** Modification: We post only a small number of isends and irecvs at a time and wait on them as suggested by Tony Ladd. *** *** See comments below about an additional modification that we may want to consider *** For long messages and power-of-two number of processes, we use a pairwise exchange algorithm, which takes p-1 steps. We calculate the pairs by using an exclusive-or algorithm: for (i=1; isize(); unsigned int short_size=256; unsigned int medium_size=32768; //short size and comm_size >=8 -> bruck // medium size messages and (short messages for comm_size < 8), we // use an algorithm that posts all irecvs and isends and then does a // waitall. // For long messages and power-of-two number of processes, we use a // pairwise exchange algorithm // For a non-power-of-two number of processes, we use an // algorithm in which, in step i, each process receives from (rank-i) // and sends to (rank+i). dsize = sdtype->size(); block_dsize = dsize * scount; if ((block_dsize < short_size) && (communicator_size >= 8)) { return Coll_alltoall_bruck::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (block_dsize < medium_size) { return Coll_alltoall_basic_linear::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); }else if (communicator_size%2){ return Coll_alltoall_ring::alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return Coll_alltoall_ring::alltoall (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } int Coll_alltoallv_mpich::alltoallv(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { /* For starters, just keep the original algorithm. */ return Coll_alltoallv_bruck::alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int Coll_barrier_mpich::barrier(MPI_Comm comm) { return Coll_barrier_ompi_bruck::barrier(comm); } /* This is the default implementation of broadcast. The algorithm is: Algorithm: MPI_Bcast For short messages, we use a binomial tree algorithm. Cost = lgp.alpha + n.lgp.beta For long messages, we do a scatter followed by an allgather. We first scatter the buffer using a binomial tree algorithm. This costs lgp.alpha + n.((p-1)/p).beta If the datatype is contiguous and the communicator is homogeneous, we treat the data as bytes and divide (scatter) it among processes by using ceiling division. For the noncontiguous or heterogeneous cases, we first pack the data into a temporary buffer by using MPI_Pack, scatter it as bytes, and unpack it after the allgather. For the allgather, we use a recursive doubling algorithm for medium-size messages and power-of-two number of processes. This takes lgp steps. In each step pairs of processes exchange all the data they have (we take care of non-power-of-two situations). This costs approximately lgp.alpha + n.((p-1)/p).beta. (Approximately because it may be slightly more in the non-power-of-two case, but it's still a logarithmic algorithm.) Therefore, for long messages Total Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta Note that this algorithm has twice the latency as the tree algorithm we use for short messages, but requires lower bandwidth: 2.n.beta versus n.lgp.beta. Therefore, for long messages and when lgp > 2, this algorithm will perform better. For long messages and for medium-size messages and non-power-of-two processes, we use a ring algorithm for the allgather, which takes p-1 steps, because it performs better than recursive doubling. Total Cost = (lgp+p-1).alpha + 2.n.((p-1)/p).beta Possible improvements: For clusters of SMPs, we may want to do something differently to take advantage of shared memory on each node. End Algorithm: MPI_Bcast */ int Coll_bcast_mpich::bcast(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm ) { /* Decision function based on MX results for messages up to 36MB and communicator sizes up to 64 nodes */ const size_t small_message_size = 12288; const size_t intermediate_message_size = 524288; int communicator_size; //int segsize = 0; size_t message_size, dsize; communicator_size = comm->size(); /* else we need data size for decision function */ dsize = datatype->size(); message_size = dsize * (unsigned long)count; /* needed for decision */ /* Handle messages of small and intermediate size, and single-element broadcasts */ if ((message_size < small_message_size) || (communicator_size <= 8)) { /* Binomial without segmentation */ return Coll_bcast_binomial_tree::bcast (buff, count, datatype, root, comm); } else if (message_size < intermediate_message_size && !(communicator_size%2)) { // SplittedBinary with 1KB segments return Coll_bcast_scatter_rdb_allgather::bcast(buff, count, datatype, root, comm); } //Handle large message sizes return Coll_bcast_scatter_LR_allgather::bcast (buff, count, datatype, root, comm); } /* This is the default implementation of reduce. The algorithm is: Algorithm: MPI_Reduce For long messages and for builtin ops and if count >= pof2 (where pof2 is the nearest power-of-two less than or equal to the number of processes), we use Rabenseifner's algorithm (see http://www.hlrs.de/organization/par/services/models/mpi/myreduce.html ). This algorithm implements the reduce in two steps: first a reduce-scatter, followed by a gather to the root. A recursive-halving algorithm (beginning with processes that are distance 1 apart) is used for the reduce-scatter, and a binomial tree algorithm is used for the gather. The non-power-of-two case is handled by dropping to the nearest lower power-of-two: the first few odd-numbered processes send their data to their left neighbors (rank-1), and the reduce-scatter happens among the remaining power-of-two processes. If the root is one of the excluded processes, then after the reduce-scatter, rank 0 sends its result to the root and exits; the root now acts as rank 0 in the binomial tree algorithm for gather. For the power-of-two case, the cost for the reduce-scatter is lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the gather to root is lgp.alpha + n.((p-1)/p).beta. Therefore, the total cost is: Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma For the non-power-of-two case, assuming the root is not one of the odd-numbered processes that get excluded in the reduce-scatter, Cost = (2.floor(lgp)+1).alpha + (2.((p-1)/p) + 1).n.beta + n.(1+(p-1)/p).gamma For short messages, user-defined ops, and count < pof2, we use a binomial tree algorithm for both short and long messages. Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma We use the binomial tree algorithm in the case of user-defined ops because in this case derived datatypes are allowed, and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky. FIXME: Per the MPI-2.1 standard this case is not possible. We should be able to use the reduce-scatter/gather approach as long as count >= pof2. [goodell@ 2009-01-21] Possible improvements: End Algorithm: MPI_Reduce */ int Coll_reduce_mpich::reduce( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { int communicator_size=0; //int segsize = 0; size_t message_size, dsize; communicator_size = comm->size(); /* need data size for decision function */ dsize=datatype->size(); message_size = dsize * count; /* needed for decision */ int pof2 = 1; while (pof2 <= communicator_size) pof2 <<= 1; pof2 >>= 1; if ((count < pof2) || (message_size < 2048) || (op != MPI_OP_NULL && not op->is_commutative())) { return Coll_reduce_binomial::reduce(sendbuf, recvbuf, count, datatype, op, root, comm); } return Coll_reduce_scatter_gather::reduce(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } /* This is the default implementation of reduce_scatter. The algorithm is: Algorithm: MPI_Reduce_scatter If the operation is commutative, for short and medium-size messages, we use a recursive-halving algorithm in which the first p/2 processes send the second n/2 data to their counterparts in the other half and receive the first n/2 data from them. This procedure continues recursively, halving the data communicated at each step, for a total of lgp steps. If the number of processes is not a power-of-two, we convert it to the nearest lower power-of-two by having the first few even-numbered processes send their data to the neighboring odd-numbered process at (rank+1). Those odd-numbered processes compute the result for their left neighbor as well in the recursive halving algorithm, and then at the end send the result back to the processes that didn't participate. Therefore, if p is a power-of-two, Cost = lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma If p is not a power-of-two, Cost = (floor(lgp)+2).alpha + n.(1+(p-1+n)/p).beta + n.(1+(p-1)/p).gamma The above cost in the non power-of-two case is approximate because there is some imbalance in the amount of work each process does because some processes do the work of their neighbors as well. For commutative operations and very long messages we use we use a pairwise exchange algorithm similar to the one used in MPI_Alltoall. At step i, each process sends n/p amount of data to (rank+i) and receives n/p amount of data from (rank-i). Cost = (p-1).alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma If the operation is not commutative, we do the following: We use a recursive doubling algorithm, which takes lgp steps. At step 1, processes exchange (n-n/p) amount of data; at step 2, (n-2n/p) amount of data; at step 3, (n-4n/p) amount of data, and so forth. Cost = lgp.alpha + n.(lgp-(p-1)/p).beta + n.(lgp-(p-1)/p).gamma Possible improvements: End Algorithm: MPI_Reduce_scatter */ int Coll_reduce_scatter_mpich::reduce_scatter( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int comm_size, i; size_t total_message_size; if(sbuf==rbuf)sbuf=MPI_IN_PLACE; //restore MPI_IN_PLACE as these algorithms handle it XBT_DEBUG("Coll_reduce_scatter_mpich::reduce"); comm_size = comm->size(); // We need data size for decision function total_message_size = 0; for (i = 0; i < comm_size; i++) { total_message_size += rcounts[i]; } if( (op==MPI_OP_NULL || op->is_commutative()) && total_message_size > 524288) { return Coll_reduce_scatter_mpich_pair::reduce_scatter (sbuf, rbuf, rcounts, dtype, op, comm); } else if ((op != MPI_OP_NULL && not op->is_commutative())) { int is_block_regular = 1; for (i = 0; i < (comm_size - 1); ++i) { if (rcounts[i] != rcounts[i + 1]) { is_block_regular = 0; break; } } /* slightly retask pof2 to mean pof2 equal or greater, not always greater as it is above */ int pof2 = 1; while (pof2 < comm_size) pof2 <<= 1; if (pof2 == comm_size && is_block_regular) { /* noncommutative, pof2 size, and block regular */ return Coll_reduce_scatter_mpich_noncomm::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); } return Coll_reduce_scatter_mpich_rdb::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); }else{ return Coll_reduce_scatter_mpich_rdb::reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm); } } /* This is the default implementation of allgather. The algorithm is: Algorithm: MPI_Allgather For short messages and non-power-of-two no. of processes, we use the algorithm from the Jehoshua Bruck et al IEEE TPDS Nov 97 paper. It is a variant of the disemmination algorithm for barrier. It takes ceiling(lg p) steps. Cost = lgp.alpha + n.((p-1)/p).beta where n is total size of data gathered on each process. For short or medium-size messages and power-of-two no. of processes, we use the recursive doubling algorithm. Cost = lgp.alpha + n.((p-1)/p).beta TODO: On TCP, we may want to use recursive doubling instead of the Bruck algorithm in all cases because of the pairwise-exchange property of recursive doubling (see Benson et al paper in Euro PVM/MPI 2003). It is interesting to note that either of the above algorithms for MPI_Allgather has the same cost as the tree algorithm for MPI_Gather! For long messages or medium-size messages and non-power-of-two no. of processes, we use a ring algorithm. In the first step, each process i sends its contribution to process i+1 and receives the contribution from process i-1 (with wrap-around). From the second step onwards, each process i forwards to process i+1 the data it received from process i-1 in the previous step. This takes a total of p-1 steps. Cost = (p-1).alpha + n.((p-1)/p).beta We use this algorithm instead of recursive doubling for long messages because we find that this communication pattern (nearest neighbor) performs twice as fast as recursive doubling for long messages (on Myrinet and IBM SP). Possible improvements: End Algorithm: MPI_Allgather */ int Coll_allgather_mpich::allgather(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size; size_t dsize, total_dsize; communicator_size = comm->size(); /* Determine complete data size */ dsize=sdtype->size(); total_dsize = dsize * scount * communicator_size; for (pow2_size = 1; pow2_size < communicator_size; pow2_size <<=1); /* Decision as in MPICH-2 presented in Thakur et.al. "Optimization of Collective Communication Operations in MPICH", International Journal of High Performance Computing Applications, Vol. 19, No. 1, 49-66 (2005) - for power-of-two processes and small and medium size messages (up to 512KB) use recursive doubling - for non-power-of-two processes and small messages (80KB) use bruck, - for everything else use ring. */ if ((pow2_size == communicator_size) && (total_dsize < 524288)) { return Coll_allgather_rdb::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (total_dsize <= 81920) { return Coll_allgather_bruck::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return Coll_allgather_ring::allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } /* This is the default implementation of allgatherv. The algorithm is: Algorithm: MPI_Allgatherv For short messages and non-power-of-two no. of processes, we use the algorithm from the Jehoshua Bruck et al IEEE TPDS Nov 97 paper. It is a variant of the disemmination algorithm for barrier. It takes ceiling(lg p) steps. Cost = lgp.alpha + n.((p-1)/p).beta where n is total size of data gathered on each process. For short or medium-size messages and power-of-two no. of processes, we use the recursive doubling algorithm. Cost = lgp.alpha + n.((p-1)/p).beta TODO: On TCP, we may want to use recursive doubling instead of the Bruck algorithm in all cases because of the pairwise-exchange property of recursive doubling (see Benson et al paper in Euro PVM/MPI 2003). For long messages or medium-size messages and non-power-of-two no. of processes, we use a ring algorithm. In the first step, each process i sends its contribution to process i+1 and receives the contribution from process i-1 (with wrap-around). From the second step onwards, each process i forwards to process i+1 the data it received from process i-1 in the previous step. This takes a total of p-1 steps. Cost = (p-1).alpha + n.((p-1)/p).beta Possible improvements: End Algorithm: MPI_Allgatherv */ int Coll_allgatherv_mpich::allgatherv(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size,i; size_t total_dsize; communicator_size = comm->size(); /* Determine complete data size */ total_dsize = 0; for (i=0; irank()!=root){ sbuf=xbt_malloc(rcount*rdtype->get_extent()); scount=rcount; sdtype=rdtype; } int ret= Coll_scatter_ompi_binomial::scatter (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); if(comm->rank()!=root){ xbt_free(sbuf); } return ret; } } } SimGrid-3.18/src/smpi/colls/smpi_automatic_selector.cpp0000644000175000017500000001714213217757320023620 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "colls_private.hpp" #include "smpi_process.hpp" //attempt to do a quick autotuning version of the collective, #define TRACE_AUTO_COLL(cat) \ if (TRACE_is_enabled()) { \ simgrid::instr::EventType* type = simgrid::instr::Container::getRoot()->type_->getOrCreateEventType(#cat); \ \ std::string cont_name = std::string("rank-" + std::to_string(smpi_process()->index())); \ type->addEntityValue(Colls::mpi_coll_##cat##_description[i].name, "1.0 1.0 1.0"); \ new simgrid::instr::NewEvent(SIMIX_get_clock(), simgrid::instr::Container::byName(cont_name), type, \ type->getEntityValue(Colls::mpi_coll_##cat##_description[i].name)); \ } #define AUTOMATIC_COLL_BENCH(cat, ret, args, args2) \ ret Coll_##cat##_automatic::cat(COLL_UNPAREN args) \ { \ double time1, time2, time_min = DBL_MAX; \ int min_coll = -1, global_coll = -1; \ int i; \ double buf_in, buf_out, max_min = DBL_MAX; \ for (i = 0; Colls::mpi_coll_##cat##_description[i].name; i++) { \ if (not strcmp(Colls::mpi_coll_##cat##_description[i].name, "automatic")) \ continue; \ if (not strcmp(Colls::mpi_coll_##cat##_description[i].name, "default")) \ continue; \ Coll_barrier_default::barrier(comm); \ TRACE_AUTO_COLL(cat) \ time1 = SIMIX_get_clock(); \ try { \ ((int(*) args)Colls::mpi_coll_##cat##_description[i].coll) args2; \ } catch (std::exception & ex) { \ continue; \ } \ time2 = SIMIX_get_clock(); \ buf_out = time2 - time1; \ Coll_reduce_default::reduce((void*)&buf_out, (void*)&buf_in, 1, MPI_DOUBLE, MPI_MAX, 0, comm); \ if (time2 - time1 < time_min) { \ min_coll = i; \ time_min = time2 - time1; \ } \ if (comm->rank() == 0) { \ if (buf_in < max_min) { \ max_min = buf_in; \ global_coll = i; \ } \ } \ } \ if (comm->rank() == 0) { \ XBT_WARN("For rank 0, the quickest was %s : %f , but global was %s : %f at max", \ Colls::mpi_coll_##cat##_description[min_coll].name, time_min, \ Colls::mpi_coll_##cat##_description[global_coll].name, max_min); \ } else \ XBT_WARN("The quickest %s was %s on rank %d and took %f", #cat, \ Colls::mpi_coll_##cat##_description[min_coll].name, comm->rank(), time_min); \ return (min_coll != -1) ? MPI_SUCCESS : MPI_ERR_INTERN; \ } namespace simgrid{ namespace smpi{ COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_ALLGATHERV_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_disps, recv_type, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_ALLREDUCE_SIG, (sbuf, rbuf, rcount, dtype, op, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_GATHER_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_type, root, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_ALLGATHER_SIG, (send_buff,send_count,send_type,recv_buff,recv_count,recv_type,comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_ALLTOALL_SIG,(send_buff, send_count, send_type, recv_buff, recv_count, recv_type,comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_ALLTOALLV_SIG, (send_buff, send_counts, send_disps, send_type, recv_buff, recv_counts, recv_disps, recv_type, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_BCAST_SIG , (buf, count, datatype, root, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_REDUCE_SIG,(buf,rbuf, count, datatype, op, root, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_REDUCE_SCATTER_SIG ,(sbuf,rbuf, rcounts,dtype,op,comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_SCATTER_SIG ,(sendbuf, sendcount, sendtype,recvbuf, recvcount, recvtype,root, comm)); COLL_APPLY(AUTOMATIC_COLL_BENCH, COLL_BARRIER_SIG,(comm)); } } SimGrid-3.18/src/smpi/smpitools.sh0000644000175000017500000000405713217757325017455 0ustar mquinsonmquinson#!/bin/sh # Copyright (c) 2013-2017. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SAVEIFS="$IFS" LISTSEP="$(printf '\b')" # Create a temporary file, with its name of the form $1_XXX$2, where XXX is replaced by an unique string. # $1: prefix, $2: suffix mymktemp () { tmp=$(mktemp --suffix="$2" "$1_XXXXXXXXXX" 2> /dev/null) if [ -z "$tmp" ]; then # mktemp failed (unsupported --suffix ?), try unsafe mode tmp=$(mktemp -u "$1_XXXXXXXXXX" 2> /dev/null) if [ -z "$tmp" ]; then # mktemp failed again (doesn't exist ?), try very unsafe mode if [ -z "${mymktemp_seq}" ]; then mymktemp_seq=$(date +%d%H%M%S) fi tmp="$1_$$x${mymktemp_seq}" mymktemp_seq=$((mymktemp_seq + 1)) fi tmp="${tmp}$2" # create temp file, and exit if it existed before sh -C -c "true > \"${tmp}\"" || exit 1 fi echo "${tmp}" } # Add a word to the end of a list (words separated by LISTSEP) # $1: list, $2...: words to add list_add () { local list content newcontent list="$1" shift if [ $# -gt 0 ]; then eval content=\"\${$list}\" IFS="$LISTSEP" newcontent="$*" IFS="$SAVEIFS" if [ -z "$content" ]; then content="$newcontent" else content="$content${LISTSEP}$newcontent" fi eval $list=\"\${content}\" fi } # Like list_add, but only if first word to add ($2) is not empty list_add_not_empty () { if [ -n "$2" ]; then list_add "$@" fi } # Set contents of a list (words separated by LISTSEP) # $1: list, $2...: words to set list_set () { eval $1="" list_add "$@" } # Get the content of a list: positional parameters ($1, $2, ...) are set to the content of the list # $1: list # usage: eval $(list_get list) list_get () { printf 'IFS="'\$'LISTSEP"; eval set -- \$%s; IFS="'\$'SAVEIFS"\n' "$1" } SimGrid-3.18/src/smpi/bindings/0000755000175000017500000000000013217757317016660 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/bindings/smpi_mpi.cpp0000644000175000017500000013514113217757317021206 0ustar mquinsonmquinson/* Copyright ,(c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license ,(GNU LGPL) which comes with this package. */ #include "private.hpp" #include "simgrid/sg_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_mpi, smpi, "Logging specific to SMPI ,(mpi)"); #define NOT_YET_IMPLEMENTED \ { \ XBT_WARN("Not yet implemented : %s. Please contact the SimGrid team if support is needed", __FUNCTION__); \ return MPI_SUCCESS; \ } #define WRAPPED_PMPI_CALL(type,name,args,args2) \ type name args { \ XBT_VERB("SMPI - Entering %s", __FUNCTION__);\ return P##name args2 ; \ }\ #define UNIMPLEMENTED_WRAPPED_PMPI_CALL(type,name,args,args2) \ type P##name args { \ NOT_YET_IMPLEMENTED \ }\ type name args { \ return P##name args2 ; \ }\ /* MPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage WRAPPED_PMPI_CALL(double, MPI_Wtick,(void),()) WRAPPED_PMPI_CALL(double, MPI_Wtime,(void),()) WRAPPED_PMPI_CALL(int,MPI_Abort,(MPI_Comm comm, int errorcode),(comm, errorcode)) WRAPPED_PMPI_CALL(int,MPI_Accumulate,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank,MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win),( origin_addr,origin_count, origin_datatype,target_rank,target_disp, target_count,target_datatype,op, win)) WRAPPED_PMPI_CALL(int,MPI_Address,(void *location, MPI_Aint * address),(location, address)) WRAPPED_PMPI_CALL(int,MPI_Allgather,(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)) WRAPPED_PMPI_CALL(int,MPI_Allgatherv,(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs,MPI_Datatype recvtype, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm)) WRAPPED_PMPI_CALL(int,MPI_Alloc_mem,(MPI_Aint size, MPI_Info info, void *baseptr),(size, info, baseptr)) WRAPPED_PMPI_CALL(int,MPI_Allreduce,(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm),(sendbuf, recvbuf, count, datatype, op, comm)) WRAPPED_PMPI_CALL(int,MPI_Alltoall,(void *sendbuf, int sendcount, MPI_Datatype sendtype,void *recvbuf, int recvcount,MPI_Datatype recvtype, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm)) WRAPPED_PMPI_CALL(int,MPI_Alltoallv,(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm),(sendbuf, sendcounts, senddisps, sendtype, recvbuf, recvcounts, recvdisps, recvtype, comm)) WRAPPED_PMPI_CALL(int,MPI_Attr_delete,(MPI_Comm comm, int keyval) ,(comm, keyval)) WRAPPED_PMPI_CALL(int,MPI_Attr_get,(MPI_Comm comm, int keyval, void* attr_value, int* flag) ,(comm, keyval, attr_value, flag)) WRAPPED_PMPI_CALL(int,MPI_Attr_put,(MPI_Comm comm, int keyval, void* attr_value) ,(comm, keyval, attr_value)) WRAPPED_PMPI_CALL(int,MPI_Barrier,(MPI_Comm comm),(comm)) WRAPPED_PMPI_CALL(int,MPI_Bcast,(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm),(buf, count, datatype, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Cart_coords,(MPI_Comm comm, int rank, int maxdims, int* coords) ,(comm, rank, maxdims, coords)) WRAPPED_PMPI_CALL(int,MPI_Cart_create,(MPI_Comm comm_old, int ndims, int* dims, int* periods, int reorder, MPI_Comm* comm_cart) ,(comm_old, ndims, dims, periods, reorder, comm_cart)) WRAPPED_PMPI_CALL(int,MPI_Cartdim_get,(MPI_Comm comm, int* ndims) ,(comm, ndims)) WRAPPED_PMPI_CALL(int,MPI_Cart_get,(MPI_Comm comm, int maxdims, int* dims, int* periods, int* coords) ,(comm, maxdims, dims, periods, coords)) WRAPPED_PMPI_CALL(int,MPI_Cart_rank,(MPI_Comm comm, int* coords, int* rank) ,(comm, coords, rank)) WRAPPED_PMPI_CALL(int,MPI_Cart_shift,(MPI_Comm comm, int direction, int displ, int* source, int* dest) ,(comm, direction, displ, source, dest)) WRAPPED_PMPI_CALL(int,MPI_Cart_sub,(MPI_Comm comm, int* remain_dims, MPI_Comm* comm_new) ,(comm, remain_dims, comm_new)) WRAPPED_PMPI_CALL(int,MPI_Comm_compare,(MPI_Comm comm1, MPI_Comm comm2, int *result),(comm1, comm2, result)) WRAPPED_PMPI_CALL(int,MPI_Comm_create_keyval,(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state),(copy_fn,delete_fn,keyval,extra_state)) WRAPPED_PMPI_CALL(int,MPI_Comm_create,(MPI_Comm comm, MPI_Group group, MPI_Comm * newcomm),(comm, group, newcomm)) WRAPPED_PMPI_CALL(int,MPI_Comm_delete_attr ,(MPI_Comm comm, int comm_keyval),(comm,comm_keyval)) WRAPPED_PMPI_CALL(int,MPI_Comm_disconnect,(MPI_Comm * comm),(comm)) WRAPPED_PMPI_CALL(int,MPI_Comm_dup,(MPI_Comm comm, MPI_Comm * newcomm),(comm, newcomm)) WRAPPED_PMPI_CALL(int,MPI_Comm_free_keyval,(int* keyval) ,( keyval)) WRAPPED_PMPI_CALL(int,MPI_Comm_free,(MPI_Comm * comm),(comm)) WRAPPED_PMPI_CALL(int,MPI_Comm_get_attr ,(MPI_Comm comm, int comm_keyval, void *attribute_val, int *flag),(comm, comm_keyval, attribute_val, flag)) WRAPPED_PMPI_CALL(int,MPI_Comm_get_name ,(MPI_Comm comm, char* name, int* len),(comm, name, len)) WRAPPED_PMPI_CALL(int,MPI_Comm_group,(MPI_Comm comm, MPI_Group * group),(comm, group)) WRAPPED_PMPI_CALL(int,MPI_Comm_rank,(MPI_Comm comm, int *rank),(comm, rank)) WRAPPED_PMPI_CALL(int,MPI_Comm_set_attr ,(MPI_Comm comm, int comm_keyval, void *attribute_val),( comm, comm_keyval, attribute_val)) WRAPPED_PMPI_CALL(int,MPI_Comm_size,(MPI_Comm comm, int *size),(comm, size)) WRAPPED_PMPI_CALL(int,MPI_Comm_split,(MPI_Comm comm, int color, int key, MPI_Comm* comm_out),(comm, color, key, comm_out)) WRAPPED_PMPI_CALL(int,MPI_Comm_create_group,(MPI_Comm comm, MPI_Group group, int tag, MPI_Comm* comm_out),(comm, group, tag, comm_out)) WRAPPED_PMPI_CALL(int,MPI_Compare_and_swap,(void *origin_addr, void *compare_addr, void *result_addr, MPI_Datatype datatype, int target_rank, MPI_Aint target_disp, MPI_Win win), (origin_addr, compare_addr, result_addr, datatype, target_rank, target_disp, win)) WRAPPED_PMPI_CALL(int,MPI_Dims_create,(int nnodes, int ndims, int* dims) ,(nnodes, ndims, dims)) WRAPPED_PMPI_CALL(int,MPI_Error_class,(int errorcode, int* errorclass) ,(errorcode, errorclass)) WRAPPED_PMPI_CALL(int,MPI_Exscan,(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm),(sendbuf, recvbuf, count, datatype, op, comm)) WRAPPED_PMPI_CALL(int,MPI_Finalized,(int * flag),(flag)) WRAPPED_PMPI_CALL(int,MPI_Finalize,(void),()) WRAPPED_PMPI_CALL(int,MPI_Free_mem,(void *baseptr),(baseptr)) WRAPPED_PMPI_CALL(int,MPI_Gather,(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Gatherv,(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs,MPI_Datatype recvtype, int root, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Get_address,(void *location, MPI_Aint * address),(location, address)) WRAPPED_PMPI_CALL(int,MPI_Get_count,(MPI_Status * status, MPI_Datatype datatype, int *count),(status, datatype, count)) WRAPPED_PMPI_CALL(int,MPI_Get_library_version ,(char *version,int *len),(version,len)) WRAPPED_PMPI_CALL(int,MPI_Get_processor_name,(char *name, int *resultlen),(name, resultlen)) WRAPPED_PMPI_CALL(int,MPI_Get_version ,(int *version,int *subversion),(version,subversion)) WRAPPED_PMPI_CALL(int,MPI_Get,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank,MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win),(origin_addr,origin_count, origin_datatype,target_rank, target_disp, target_count,target_datatype,win)) WRAPPED_PMPI_CALL(int,MPI_Get_accumulate, (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win),(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op, win)) WRAPPED_PMPI_CALL(int,MPI_Fetch_and_op, (void *origin_addr, void *result_addr, MPI_Datatype datatype, int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win),(origin_addr, result_addr, datatype, target_rank, target_disp, op, win)) WRAPPED_PMPI_CALL(int,MPI_Group_compare,(MPI_Group group1, MPI_Group group2, int *result),(group1, group2, result)) WRAPPED_PMPI_CALL(int,MPI_Group_difference,(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup),(group1, group2, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_excl,(MPI_Group group, int n, int *ranks, MPI_Group * newgroup),(group, n, ranks, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_free,(MPI_Group * group),(group)) WRAPPED_PMPI_CALL(int,MPI_Group_incl,(MPI_Group group, int n, int *ranks, MPI_Group * newgroup),(group, n, ranks, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_intersection,(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup),(group1, group2, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_range_excl,(MPI_Group group, int n, int ranges[][3], MPI_Group * newgroup),(group, n, ranges, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_range_incl,(MPI_Group group, int n, int ranges[][3], MPI_Group * newgroup),(group, n, ranges, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Group_rank,(MPI_Group group, int *rank),(group, rank)) WRAPPED_PMPI_CALL(int,MPI_Group_size,(MPI_Group group, int *size),(group, size)) WRAPPED_PMPI_CALL(int,MPI_Group_translate_ranks,(MPI_Group group1, int n, int *ranks1, MPI_Group group2, int *ranks2),(group1, n, ranks1, group2, ranks2)) WRAPPED_PMPI_CALL(int,MPI_Group_union,(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup),(group1, group2, newgroup)) WRAPPED_PMPI_CALL(int,MPI_Info_create,( MPI_Info *info),( info)) WRAPPED_PMPI_CALL(int,MPI_Info_delete,(MPI_Info info, char *key),(info, key)) WRAPPED_PMPI_CALL(int,MPI_Info_dup,(MPI_Info info, MPI_Info *newinfo),(info, newinfo)) WRAPPED_PMPI_CALL(int,MPI_Info_free,( MPI_Info *info),( info)) WRAPPED_PMPI_CALL(int,MPI_Info_get,(MPI_Info info,char *key,int valuelen, char *value, int *flag),(info,key,valuelen, value, flag)) WRAPPED_PMPI_CALL(int,MPI_Info_get_nkeys,( MPI_Info info, int *nkeys),(info, nkeys)) WRAPPED_PMPI_CALL(int,MPI_Info_get_nthkey,( MPI_Info info, int n, char *key),( info, n, key)) WRAPPED_PMPI_CALL(int,MPI_Info_get_valuelen,( MPI_Info info, char *key, int *valuelen, int *flag),( info, key, valuelen, flag)) WRAPPED_PMPI_CALL(int,MPI_Info_set,( MPI_Info info, char *key, char *value),( info, key, value)) WRAPPED_PMPI_CALL(int,MPI_Initialized,(int* flag) ,(flag)) WRAPPED_PMPI_CALL(int,MPI_Init,(int *argc, char ***argv),(argc, argv)) WRAPPED_PMPI_CALL(int,MPI_Init_thread,(int *argc, char ***argv, int required, int *provided),(argc, argv, required, provided)) WRAPPED_PMPI_CALL(int,MPI_Iprobe,(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status) ,(source, tag, comm, flag, status)) WRAPPED_PMPI_CALL(int,MPI_Irecv,(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request),(buf, count, datatype, src, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Isend,(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request),(buf, count, datatype, dst, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Issend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request) ,(buf, count, datatype, dest, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Is_thread_main,(int *flag),(flag)) WRAPPED_PMPI_CALL(int,MPI_Keyval_create,(MPI_Copy_function* copy_fn, MPI_Delete_function* delete_fn, int* keyval, void* extra_state) ,(copy_fn, delete_fn, keyval, extra_state)) WRAPPED_PMPI_CALL(int,MPI_Keyval_free,(int* keyval) ,(keyval)) WRAPPED_PMPI_CALL(int,MPI_Op_create,(MPI_User_function * function, int commute, MPI_Op * op),(function, commute, op)) WRAPPED_PMPI_CALL(int,MPI_Op_free,(MPI_Op * op),(op)) WRAPPED_PMPI_CALL(int,MPI_Op_commutative,(MPI_Op op, int *commute), (op, commute)) WRAPPED_PMPI_CALL(int,MPI_Pack_size,(int incount, MPI_Datatype datatype, MPI_Comm comm, int* size) ,(incount, datatype, comm, size)) WRAPPED_PMPI_CALL(int,MPI_Pack,(void* inbuf, int incount, MPI_Datatype type, void* outbuf, int outcount, int* position, MPI_Comm comm) ,(inbuf, incount, type, outbuf, outcount, position, comm)) WRAPPED_PMPI_CALL(int,MPI_Probe,(int source, int tag, MPI_Comm comm, MPI_Status* status) ,(source, tag, comm, status)) WRAPPED_PMPI_CALL(int,MPI_Put,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win),(origin_addr,origin_count, origin_datatype,target_rank,target_disp, target_count,target_datatype, win)) WRAPPED_PMPI_CALL(int,MPI_Query_thread,(int *provided),(provided)) WRAPPED_PMPI_CALL(int,MPI_Raccumulate,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank,MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request* request),( origin_addr,origin_count, origin_datatype,target_rank,target_disp, target_count,target_datatype,op, win, request)) WRAPPED_PMPI_CALL(int,MPI_Recv_init,(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request),(buf, count, datatype, src, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Recv,(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status),(buf, count, datatype, src, tag, comm, status)) WRAPPED_PMPI_CALL(int,MPI_Reduce_local,(void *inbuf, void *inoutbuf, int count, MPI_Datatype datatype, MPI_Op op),(inbuf, inoutbuf, count, datatype, op)) WRAPPED_PMPI_CALL(int,MPI_Reduce_scatter_block,(void *sendbuf, void *recvbuf, int recvcount, MPI_Datatype datatype, MPI_Op op,MPI_Comm comm),(sendbuf, recvbuf, recvcount, datatype, op, comm)) WRAPPED_PMPI_CALL(int,MPI_Reduce_scatter,(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm),(sendbuf, recvbuf, recvcounts, datatype, op, comm)) WRAPPED_PMPI_CALL(int,MPI_Reduce,(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm),(sendbuf, recvbuf, count, datatype, op, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Request_free,(MPI_Request * request),(request)) WRAPPED_PMPI_CALL(int,MPI_Rget,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank,MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win, MPI_Request* request),(origin_addr,origin_count, origin_datatype,target_rank, target_disp, target_count,target_datatype,win, request)) WRAPPED_PMPI_CALL(int,MPI_Rget_accumulate, (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request* request),(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op, win, request)) WRAPPED_PMPI_CALL(int,MPI_Rput,( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win, MPI_Request* request),(origin_addr,origin_count, origin_datatype,target_rank,target_disp, target_count,target_datatype, win, request)) WRAPPED_PMPI_CALL(int,MPI_Scan,(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm),(sendbuf, recvbuf, count, datatype, op, comm)) WRAPPED_PMPI_CALL(int,MPI_Scatter,(void *sendbuf, int sendcount, MPI_Datatype sendtype,void *recvbuf, int recvcount, MPI_Datatype recvtype,int root, MPI_Comm comm),(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Scatterv,(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount,MPI_Datatype recvtype, int root, MPI_Comm comm),(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm)) WRAPPED_PMPI_CALL(int,MPI_Send_init,(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request),(buf, count, datatype, dst, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Sendrecv_replace,(void *buf, int count, MPI_Datatype datatype, int dst, int sendtag, int src, int recvtag,MPI_Comm comm, MPI_Status * status),(buf, count, datatype, dst, sendtag, src, recvtag, comm, status)) WRAPPED_PMPI_CALL(int,MPI_Sendrecv,(void *sendbuf, int sendcount, MPI_Datatype sendtype,int dst, int sendtag, void *recvbuf, int recvcount,MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status),(sendbuf, sendcount, sendtype, dst, sendtag, recvbuf, recvcount, recvtype, src, recvtag,comm, status)) WRAPPED_PMPI_CALL(int,MPI_Send,(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm),(buf, count, datatype, dst, tag, comm)) WRAPPED_PMPI_CALL(int,MPI_Ssend_init,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request),(buf, count, datatype, dest, tag, comm, request)) WRAPPED_PMPI_CALL(int,MPI_Ssend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) ,(buf, count, datatype, dest, tag, comm)) WRAPPED_PMPI_CALL(int,MPI_Startall,(int count, MPI_Request * requests),(count, requests)) WRAPPED_PMPI_CALL(int,MPI_Start,(MPI_Request * request),(request)) WRAPPED_PMPI_CALL(int,MPI_Testall,(int count, MPI_Request* requests, int* flag, MPI_Status* statuses) ,(count, requests, flag, statuses)) WRAPPED_PMPI_CALL(int,MPI_Testany,(int count, MPI_Request requests[], int *index, int *flag, MPI_Status * status),(count, requests, index, flag, status)) WRAPPED_PMPI_CALL(int,MPI_Test,(MPI_Request * request, int *flag, MPI_Status * status),(request, flag, status)) WRAPPED_PMPI_CALL(int,MPI_Testsome,(int incount, MPI_Request* requests, int* outcount, int* indices, MPI_Status* statuses) ,(incount, requests, outcount, indices, statuses)) WRAPPED_PMPI_CALL(int,MPI_Type_commit,(MPI_Datatype* datatype) ,(datatype)) WRAPPED_PMPI_CALL(int,MPI_Type_contiguous,(int count, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_create_hindexed_block,(int count, int blocklength, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklength, indices, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_create_hindexed,(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type,MPI_Datatype* new_type) ,(count, blocklens,indices,old_type,new_type)) WRAPPED_PMPI_CALL(int,MPI_Type_create_hvector,(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) ,(count, blocklen, stride, old_type, new_type)) WRAPPED_PMPI_CALL(int,MPI_Type_create_indexed_block,(int count, int blocklength, int* indices,MPI_Datatype old_type,MPI_Datatype *newtype),(count, blocklength, indices, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_create_indexed,(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklens, indices, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_create_keyval,(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval,void* extra_state),(copy_fn,delete_fn,keyval,extra_state)) WRAPPED_PMPI_CALL(int,MPI_Type_create_resized,(MPI_Datatype oldtype,MPI_Aint lb, MPI_Aint extent, MPI_Datatype *newtype),(oldtype,lb, extent, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_create_struct,(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types,MPI_Datatype* newtype) ,(count, blocklens, indices, old_types, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_delete_attr ,(MPI_Datatype type, int type_keyval),(type,type_keyval)) WRAPPED_PMPI_CALL(int,MPI_Type_dup,(MPI_Datatype datatype, MPI_Datatype * newdatatype),(datatype, newdatatype)) WRAPPED_PMPI_CALL(int,MPI_Type_extent,(MPI_Datatype datatype, MPI_Aint * extent),(datatype, extent)) WRAPPED_PMPI_CALL(int,MPI_Type_free_keyval,(int* keyval) ,(keyval)) WRAPPED_PMPI_CALL(int,MPI_Type_free,(MPI_Datatype * datatype),(datatype)) WRAPPED_PMPI_CALL(int,MPI_Type_get_attr ,(MPI_Datatype type, int type_keyval, void *attribute_val, int* flag),( type, type_keyval, attribute_val, flag)) WRAPPED_PMPI_CALL(int,MPI_Type_get_extent,(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent),(datatype, lb, extent)) WRAPPED_PMPI_CALL(int,MPI_Type_get_name,(MPI_Datatype datatype, char * name, int* len),(datatype,name,len)) WRAPPED_PMPI_CALL(int,MPI_Type_get_true_extent,(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent),(datatype, lb, extent)) WRAPPED_PMPI_CALL(int,MPI_Type_hindexed,(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklens, indices, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_hvector,(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklen, stride, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_indexed,(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklens, indices, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_lb,(MPI_Datatype datatype, MPI_Aint * disp),(datatype, disp)) WRAPPED_PMPI_CALL(int,MPI_Type_set_attr ,(MPI_Datatype type, int type_keyval, void *attribute_val),( type, type_keyval, attribute_val)) WRAPPED_PMPI_CALL(int,MPI_Type_set_name,(MPI_Datatype datatype, char * name),(datatype, name)) WRAPPED_PMPI_CALL(int,MPI_Type_size,(MPI_Datatype datatype, int *size),(datatype, size)) WRAPPED_PMPI_CALL(int,MPI_Type_size_x,(MPI_Datatype datatype, MPI_Count *size),(datatype, size)) WRAPPED_PMPI_CALL(int,MPI_Type_struct,(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* newtype) ,(count, blocklens, indices, old_types, newtype)) WRAPPED_PMPI_CALL(int,MPI_Type_ub,(MPI_Datatype datatype, MPI_Aint * disp),(datatype, disp)) WRAPPED_PMPI_CALL(int,MPI_Type_vector,(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* newtype) ,(count, blocklen, stride, old_type, newtype)) WRAPPED_PMPI_CALL(int,MPI_Unpack,(void* inbuf, int insize, int* position, void* outbuf, int outcount, MPI_Datatype type, MPI_Comm comm) ,(inbuf, insize, position, outbuf, outcount, type, comm)) WRAPPED_PMPI_CALL(int,MPI_Waitall,(int count, MPI_Request requests[], MPI_Status status[]),(count, requests, status)) WRAPPED_PMPI_CALL(int,MPI_Waitany,(int count, MPI_Request requests[], int *index, MPI_Status * status),(count, requests, index, status)) WRAPPED_PMPI_CALL(int,MPI_Wait,(MPI_Request * request, MPI_Status * status),(request, status)) WRAPPED_PMPI_CALL(int,MPI_Waitsome,(int incount, MPI_Request requests[], int *outcount, int *indices, MPI_Status status[]),(incount, requests, outcount, indices, status)) WRAPPED_PMPI_CALL(int,MPI_Win_complete,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_create,( void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, MPI_Win *win),( base, size, disp_unit, info, comm,win)) WRAPPED_PMPI_CALL(int,MPI_Win_allocate,(MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, void *base, MPI_Win *win),(size, disp_unit, info, comm, base, win)) WRAPPED_PMPI_CALL(int,MPI_Win_attach,(MPI_Win win, void *base, MPI_Aint size),(win, base, size)) WRAPPED_PMPI_CALL(int,MPI_Win_detach,(MPI_Win win, void *base),(win, base)) WRAPPED_PMPI_CALL(int,MPI_Win_create_dynamic,( MPI_Info info, MPI_Comm comm, MPI_Win *win),(info, comm,win)) WRAPPED_PMPI_CALL(int,MPI_Win_fence,( int assert,MPI_Win win),( assert, win)) WRAPPED_PMPI_CALL(int,MPI_Win_free,( MPI_Win* win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_get_group,(MPI_Win win, MPI_Group * group),(win, group)) WRAPPED_PMPI_CALL(int,MPI_Win_get_name,(MPI_Win win, char * name, int* len),(win,name,len)) WRAPPED_PMPI_CALL(int,MPI_Win_get_info,(MPI_Win win, MPI_Info * info),(win,info)) WRAPPED_PMPI_CALL(int,MPI_Win_post,(MPI_Group group, int assert, MPI_Win win),(group, assert, win)) WRAPPED_PMPI_CALL(int,MPI_Win_set_name,(MPI_Win win, char * name),(win, name)) WRAPPED_PMPI_CALL(int,MPI_Win_set_info,(MPI_Win win, MPI_Info info),(win,info)) WRAPPED_PMPI_CALL(int,MPI_Win_start,(MPI_Group group, int assert, MPI_Win win),(group, assert, win)) WRAPPED_PMPI_CALL(int,MPI_Win_wait,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_lock,(int lock_type, int rank, int assert, MPI_Win win) ,(lock_type, rank, assert, win)) WRAPPED_PMPI_CALL(int,MPI_Win_unlock,(int rank, MPI_Win win),(rank, win)) WRAPPED_PMPI_CALL(int,MPI_Win_lock_all,(int assert, MPI_Win win) ,(assert, win)) WRAPPED_PMPI_CALL(int,MPI_Win_unlock_all,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_flush,(int rank, MPI_Win win),(rank, win)) WRAPPED_PMPI_CALL(int,MPI_Win_flush_local,(int rank, MPI_Win win),(rank, win)) WRAPPED_PMPI_CALL(int,MPI_Win_flush_all,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_flush_local_all,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(int,MPI_Win_get_attr, (MPI_Win type, int type_keyval, void *attribute_val, int* flag), (type, type_keyval, attribute_val, flag)) WRAPPED_PMPI_CALL(int,MPI_Win_set_attr, (MPI_Win type, int type_keyval, void *att), (type, type_keyval, att)) WRAPPED_PMPI_CALL(int,MPI_Win_delete_attr, (MPI_Win type, int comm_keyval), (type, comm_keyval)) WRAPPED_PMPI_CALL(int,MPI_Win_create_keyval,(MPI_Win_copy_attr_function* copy_fn, MPI_Win_delete_attr_function* delete_fn, int* keyval, void* extra_state), (copy_fn, delete_fn, keyval, extra_state)) WRAPPED_PMPI_CALL(int,MPI_Win_free_keyval,(int* keyval), (keyval)) WRAPPED_PMPI_CALL(MPI_Comm, MPI_Comm_f2c,(MPI_Fint comm),(comm)) WRAPPED_PMPI_CALL(MPI_Datatype, MPI_Type_f2c,(MPI_Fint datatype),(datatype)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Comm_c2f,(MPI_Comm comm),(comm)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Group_c2f,(MPI_Group group),(group)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Info_c2f,(MPI_Info info),(info)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Op_c2f,(MPI_Op op),(op)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Request_c2f,(MPI_Request request) ,(request)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Type_c2f,(MPI_Datatype datatype),( datatype)) WRAPPED_PMPI_CALL(MPI_Fint, MPI_Win_c2f,(MPI_Win win),(win)) WRAPPED_PMPI_CALL(MPI_Group, MPI_Group_f2c,(MPI_Fint group),( group)) WRAPPED_PMPI_CALL(MPI_Info, MPI_Info_f2c,(MPI_Fint info),(info)) WRAPPED_PMPI_CALL(MPI_Op, MPI_Op_f2c,(MPI_Fint op),(op)) WRAPPED_PMPI_CALL(MPI_Request, MPI_Request_f2c,(MPI_Fint request),(request)) WRAPPED_PMPI_CALL(MPI_Win, MPI_Win_f2c,(MPI_Fint win),(win)) /* Unimplemented Calls - both PMPI and MPI calls are generated. When implementing, please move ahead, swap UNIMPLEMENTED_WRAPPED_PMPI_CALL for WRAPPED_PMPI_CALL, and implement PMPI version of the function in smpi_pmpi.cpp file */ UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Add_error_class,( int *errorclass),( errorclass)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Add_error_code,(int errorclass, int *errorcode),(errorclass, errorcode)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Add_error_string,( int errorcode, char *string),(errorcode, string)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Alltoallw,( void *sendbuf, int *sendcnts, int *sdispls, MPI_Datatype *sendtypes, void *recvbuf, int *recvcnts, int *rdispls, MPI_Datatype *recvtypes, MPI_Comm comm),( sendbuf, sendcnts, sdispls, sendtypes, recvbuf, recvcnts, rdispls, recvtypes, comm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Bsend_init,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request),(buf, count, datatype, dest, tag, comm, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Bsend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) ,(buf, count, datatype, dest, tag, comm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Buffer_attach,(void* buffer, int size) ,(buffer, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Buffer_detach,(void* buffer, int* size) ,(buffer, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Cancel,(MPI_Request* request) ,(request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Cart_map,(MPI_Comm comm_old, int ndims, int* dims, int* periods, int* newrank) ,(comm_old, ndims, dims, periods, newrank)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Close_port,( char *port_name),( port_name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_accept,( char *port_name, MPI_Info info, int root, MPI_Comm comm, MPI_Comm *newcomm),( port_name, info, root, comm, newcomm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_call_errhandler,(MPI_Comm comm,int errorcode),(comm, errorcode)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_connect,( char *port_name, MPI_Info info, int root, MPI_Comm comm, MPI_Comm *newcomm),( port_name, info, root, comm, newcomm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_create_errhandler,( MPI_Comm_errhandler_fn *function, MPI_Errhandler *errhandler),( function, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_dup_with_info,(MPI_Comm comm, MPI_Info info, MPI_Comm * newcomm),(comm,info,newcomm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_get_errhandler,(MPI_Comm comm, MPI_Errhandler* errhandler) ,(comm, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_get_info ,(MPI_Comm comm, MPI_Info* info),(comm, info)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_get_parent,( MPI_Comm *parent),( parent)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_join,( int fd, MPI_Comm *intercomm),( fd, intercomm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_remote_group,(MPI_Comm comm, MPI_Group* group) ,(comm, group)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_remote_size,(MPI_Comm comm, int* size) ,(comm, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_set_errhandler,(MPI_Comm comm, MPI_Errhandler errhandler) ,(comm, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_set_info ,(MPI_Comm comm, MPI_Info info),(comm, info)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_set_name ,(MPI_Comm comm, char* name),(comm, name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_spawn,( char *command, char **argv, int maxprocs, MPI_Info info, int root, MPI_Comm comm, MPI_Comm *intercomm, int* array_of_errcodes),( command, argv, maxprocs, info, root, comm, intercomm, array_of_errcodes)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_spawn_multiple,(int count, char **array_of_commands, char*** array_of_argv, int* array_of_maxprocs, MPI_Info* array_of_info, int root, MPI_Comm comm, MPI_Comm *intercomm, int* array_of_errcodes), (count, array_of_commands, array_of_argv, array_of_maxprocs, array_of_info, root, comm, intercomm, array_of_errcodes)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_split_type,(MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm),(comm, split_type, key, info, newcomm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Comm_test_inter,(MPI_Comm comm, int* flag) ,(comm, flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Errhandler_create,(MPI_Handler_function* function, MPI_Errhandler* errhandler) ,(function, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Errhandler_free,(MPI_Errhandler* errhandler) ,(errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Errhandler_get,(MPI_Comm comm, MPI_Errhandler* errhandler) ,(comm, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Errhandler_set,(MPI_Comm comm, MPI_Errhandler errhandler) ,(comm, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Error_string,(int errorcode, char* string, int* resultlen) ,(errorcode, string, resultlen)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Register_datarep, (char *datarep, MPI_Datarep_conversion_function *read_conversion_fn, MPI_Datarep_conversion_function *write_conversion_fn, MPI_Datarep_extent_function *dtype_file_extent_fn, void *extra_state) ,(datarep, read_conversion_fn, write_conversion_fn, dtype_file_extent_fn, extra_state)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(MPI_Fint, MPI_File_c2f,(MPI_File file), (file)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(MPI_File, MPI_File_f2c,(MPI_Fint file), (file)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_call_errhandler,(MPI_File fh, int errorcode), (fh, errorcode)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_create_errhandler,(MPI_File_errhandler_function *function, MPI_Errhandler *errhandler),(function, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_set_errhandler,( MPI_File file, MPI_Errhandler errhandler), (file, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_errhandler,( MPI_File file, MPI_Errhandler *errhandler), (file, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_open,(MPI_Comm comm, char *filename, int amode, MPI_Info info, MPI_File *fh),(comm, filename, amode, info, fh)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_close,(MPI_File *fh), (fh)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_delete,(char *filename, MPI_Info info), (filename, info)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_set_size,(MPI_File fh, MPI_Offset size), (fh, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_preallocate,(MPI_File fh, MPI_Offset size), (fh, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_size,(MPI_File fh, MPI_Offset *size), (fh, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_group,(MPI_File fh, MPI_Group *group), (fh, group)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_amode,(MPI_File fh, int *amode), (fh, amode)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_set_info,(MPI_File fh, MPI_Info info), (fh, info)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_info,(MPI_File fh, MPI_Info *info_used), (fh, info_used)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_set_view,(MPI_File fh, MPI_Offset disp, MPI_Datatype etype, MPI_Datatype filetype, char *datarep, MPI_Info info), (fh, disp, etype, filetype, datarep, info)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_view,(MPI_File fh, MPI_Offset *disp, MPI_Datatype *etype, MPI_Datatype *filetype, char *datarep), (fh, disp, etype, filetype, datarep)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_at,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, offset, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_at_all,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, offset, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_at,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, offset, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_at_all,(MPI_File fh, MPI_Offset offset, void *buf,int count, MPI_Datatype datatype, MPI_Status *status), (fh, offset, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iread_at,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, offset, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iwrite_at,(MPI_File fh, MPI_Offset offset, void *buf,int count, MPI_Datatype datatype, MPI_Request *request), (fh, offset, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iread_at_all,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, offset, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iwrite_at_all,(MPI_File fh, MPI_Offset offset, void *buf,int count, MPI_Datatype datatype, MPI_Request *request), (fh, offset, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read,(MPI_File fh, void *buf, int count,MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_all,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_all,(MPI_File fh, void *buf, int count,MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iread,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iwrite,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iread_all,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iwrite_all,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_seek,(MPI_File fh, MPI_Offset offset, int whenace), (fh, offset, whenace)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_position,(MPI_File fh, MPI_Offset *offset), (fh, offset)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_byte_offset,(MPI_File fh, MPI_Offset offset, MPI_Offset *disp), (fh, offset, disp)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_shared,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_shared,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iread_shared,(MPI_File fh, void *buf, int count,MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_iwrite_shared,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Request *request), (fh, buf, count, datatype, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_ordered,(MPI_File fh, void *buf, int count,MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_ordered,(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status), (fh, buf, count, datatype, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_seek_shared,(MPI_File fh, MPI_Offset offset, int whence), (fh, offset, whence)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_position_shared,(MPI_File fh, MPI_Offset *offset), (fh, offset)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_at_all_begin,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype), (fh, offset, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_at_all_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_at_all_begin,(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype), (fh, offset, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_at_all_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_all_begin,(MPI_File fh, void *buf, int count, MPI_Datatype datatype), (fh, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_all_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_all_begin,(MPI_File fh, void *buf, int count, MPI_Datatype datatype), (fh, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_all_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_ordered_begin,(MPI_File fh, void *buf, int count, MPI_Datatype datatype), (fh, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_read_ordered_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_ordered_begin,(MPI_File fh, void *buf, int count, MPI_Datatype datatype), (fh, buf, count, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_write_ordered_end,(MPI_File fh, void *buf, MPI_Status *status), (fh, buf, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_type_extent,(MPI_File fh, MPI_Datatype datatype, MPI_Aint *extent), (fh, datatype, extent)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_set_atomicity,(MPI_File fh, int flag), (fh, flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_get_atomicity,(MPI_File fh, int *flag), (fh, flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_File_sync,(MPI_File fh), (fh)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Get_elements,(MPI_Status* status, MPI_Datatype datatype, int* elements) ,(status, datatype, elements)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graph_create,(MPI_Comm comm_old, int nnodes, int* index, int* edges, int reorder, MPI_Comm* comm_graph) ,(comm_old, nnodes, index, edges, reorder, comm_graph)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graphdims_get,(MPI_Comm comm, int* nnodes, int* nedges) ,(comm, nnodes, nedges)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graph_get,(MPI_Comm comm, int maxindex, int maxedges, int* index, int* edges) ,(comm, maxindex, maxedges, index, edges)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graph_map,(MPI_Comm comm_old, int nnodes, int* index, int* edges, int* newrank) ,(comm_old, nnodes, index, edges, newrank)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graph_neighbors_count,(MPI_Comm comm, int rank, int* nneighbors) ,(comm, rank, nneighbors)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Graph_neighbors,(MPI_Comm comm, int rank, int maxneighbors, int* neighbors) ,(comm, rank, maxneighbors, neighbors)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Grequest_complete,( MPI_Request request),( request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Grequest_start,(MPI_Grequest_query_function *query_fn, MPI_Grequest_free_function *free_fn,MPI_Grequest_cancel_function *cancel_fn, void *extra_state, MPI_Request *request),( query_fn, free_fn, cancel_fn, extra_state, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Ibsend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request) ,(buf, count, datatype, dest, tag, comm, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Intercomm_create,(MPI_Comm local_comm, int local_leader, MPI_Comm peer_comm, int remote_leader, int tag,MPI_Comm* comm_out) ,(local_comm, local_leader, peer_comm, remote_leader, tag, comm_out)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Intercomm_merge,(MPI_Comm comm, int high, MPI_Comm* comm_out) ,(comm, high, comm_out)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Irsend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request) ,(buf, count, datatype, dest, tag, comm, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Lookup_name,( char *service_name, MPI_Info info, char *port_name),( service_name, info, port_name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Open_port,( MPI_Info info, char *port_name),( info,port_name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Pack_external,(char *datarep, void *inbuf, int incount, MPI_Datatype datatype, void *outbuf, MPI_Aint outcount, MPI_Aint *position),(datarep, inbuf, incount, datatype, outbuf, outcount, position)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Pack_external_size,(char *datarep, int incount, MPI_Datatype datatype, MPI_Aint *size),(datarep, incount, datatype, size)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Pcontrol,(const int level, ... ),(level)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Publish_name,( char *service_name, MPI_Info info, char *port_name),( service_name, info, port_name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Request_get_status,( MPI_Request request, int *flag, MPI_Status *status),( request, flag, status)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Rsend_init,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request* request),(buf, count, datatype, dest, tag, comm, request)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Rsend,(void* buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) ,(buf, count, datatype, dest, tag, comm)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Status_set_cancelled,(MPI_Status *status,int flag),(status,flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Status_set_elements,( MPI_Status *status, MPI_Datatype datatype, int count),( status, datatype, count)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int, MPI_Test_cancelled,(MPI_Status* status, int* flag) ,(status, flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Topo_test,(MPI_Comm comm, int* top_type) ,(comm, top_type)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Type_create_darray,(int size, int rank, int ndims, int* array_of_gsizes, int* array_of_distribs, int* array_of_dargs, int* array_of_psizes,int order, MPI_Datatype oldtype, MPI_Datatype *newtype) ,(size, rank, ndims, array_of_gsizes,array_of_distribs, array_of_dargs, array_of_psizes,order,oldtype, newtype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Type_create_subarray,(int ndims,int *array_of_sizes, int *array_of_subsizes, int *array_of_starts, int order,MPI_Datatype oldtype, MPI_Datatype *newtype),(ndims,array_of_sizes, array_of_subsizes, array_of_starts, order, oldtype, newtype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Type_get_contents,(MPI_Datatype datatype, int max_integers, int max_addresses, int max_datatypes, int* array_of_integers, MPI_Aint* array_of_addresses, MPI_Datatype *array_of_datatypes),(datatype, max_integers, max_addresses,max_datatypes, array_of_integers, array_of_addresses, array_of_datatypes)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Type_get_envelope,( MPI_Datatype datatype, int *num_integers, int *num_addresses, int *num_datatypes, int *combiner),(datatype, num_integers, num_addresses, num_datatypes, combiner)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Type_match_size,(int typeclass,int size,MPI_Datatype *datatype),(typeclass,size,datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Unpack_external,(char *datarep, void *inbuf, MPI_Aint insize, MPI_Aint *position, void *outbuf, int outcount, MPI_Datatype datatype),( datarep, inbuf, insize, position, outbuf, outcount, datatype)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Unpublish_name,( char *service_name, MPI_Info info, char *port_name),( service_name, info, port_name)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Win_set_errhandler,(MPI_Win win, MPI_Errhandler errhandler) ,(win, errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(int,MPI_Win_test,(MPI_Win win, int *flag),(win, flag)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(MPI_Errhandler, MPI_Errhandler_f2c,(MPI_Fint errhandler),(errhandler)) UNIMPLEMENTED_WRAPPED_PMPI_CALL(MPI_Fint, MPI_Errhandler_c2f,(MPI_Errhandler errhandler),(errhandler)) } // extern "C" SimGrid-3.18/src/smpi/bindings/smpi_pmpi_comm.cpp0000644000175000017500000001513613217757317022402 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_process.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Comm_rank(MPI_Comm comm, int *rank) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (rank == nullptr) { return MPI_ERR_ARG; } else { *rank = comm->rank(); return MPI_SUCCESS; } } int PMPI_Comm_size(MPI_Comm comm, int *size) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (size == nullptr) { return MPI_ERR_ARG; } else { *size = comm->size(); return MPI_SUCCESS; } } int PMPI_Comm_get_name (MPI_Comm comm, char* name, int* len) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (name == nullptr || len == nullptr) { return MPI_ERR_ARG; } else { comm->get_name(name, len); return MPI_SUCCESS; } } int PMPI_Comm_group(MPI_Comm comm, MPI_Group * group) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (group == nullptr) { return MPI_ERR_ARG; } else { *group = comm->group(); if (*group != MPI_COMM_WORLD->group() && *group != MPI_GROUP_NULL && *group != MPI_GROUP_EMPTY) (*group)->ref(); return MPI_SUCCESS; } } int PMPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result) { if (comm1 == MPI_COMM_NULL || comm2 == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (result == nullptr) { return MPI_ERR_ARG; } else { if (comm1 == comm2) { /* Same communicators means same groups */ *result = MPI_IDENT; } else { *result = comm1->group()->compare(comm2->group()); if (*result == MPI_IDENT) { *result = MPI_CONGRUENT; } } return MPI_SUCCESS; } } int PMPI_Comm_dup(MPI_Comm comm, MPI_Comm * newcomm) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (newcomm == nullptr) { return MPI_ERR_ARG; } else { return comm->dup(newcomm); } } int PMPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm * newcomm) { if (comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newcomm == nullptr) { return MPI_ERR_ARG; } else if(group->rank(smpi_process()->index())==MPI_UNDEFINED){ *newcomm= MPI_COMM_NULL; return MPI_SUCCESS; }else{ group->ref(); *newcomm = new simgrid::smpi::Comm(group, nullptr); return MPI_SUCCESS; } } int PMPI_Comm_free(MPI_Comm * comm) { if (comm == nullptr) { return MPI_ERR_ARG; } else if (*comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else { simgrid::smpi::Comm::destroy(*comm); *comm = MPI_COMM_NULL; return MPI_SUCCESS; } } int PMPI_Comm_disconnect(MPI_Comm * comm) { /* TODO: wait until all communication in comm are done */ if (comm == nullptr) { return MPI_ERR_ARG; } else if (*comm == MPI_COMM_NULL) { return MPI_ERR_COMM; } else { simgrid::smpi::Comm::destroy(*comm); *comm = MPI_COMM_NULL; return MPI_SUCCESS; } } int PMPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm* comm_out) { int retval = 0; smpi_bench_end(); if (comm_out == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { *comm_out = comm->split(color, key); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Comm_create_group(MPI_Comm comm, MPI_Group group, int, MPI_Comm* comm_out) { int retval = 0; smpi_bench_end(); if (comm_out == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { retval = MPI_Comm_create(comm, group, comm_out); } smpi_bench_begin(); return retval; } MPI_Comm PMPI_Comm_f2c(MPI_Fint comm){ return static_cast(simgrid::smpi::Comm::f2c(comm)); } MPI_Fint PMPI_Comm_c2f(MPI_Comm comm){ return comm->c2f(); } int PMPI_Comm_get_attr (MPI_Comm comm, int comm_keyval, void *attribute_val, int *flag) { return PMPI_Attr_get(comm, comm_keyval, attribute_val,flag); } int PMPI_Comm_set_attr (MPI_Comm comm, int comm_keyval, void *attribute_val) { return PMPI_Attr_put(comm, comm_keyval, attribute_val); } int PMPI_Comm_delete_attr (MPI_Comm comm, int comm_keyval) { return PMPI_Attr_delete(comm, comm_keyval); } int PMPI_Comm_create_keyval(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state) { return PMPI_Keyval_create(copy_fn, delete_fn, keyval, extra_state); } int PMPI_Comm_free_keyval(int* keyval) { return PMPI_Keyval_free(keyval); } int PMPI_Attr_delete(MPI_Comm comm, int keyval) { if(keyval == MPI_TAG_UB||keyval == MPI_HOST||keyval == MPI_IO ||keyval == MPI_WTIME_IS_GLOBAL||keyval == MPI_APPNUM ||keyval == MPI_UNIVERSE_SIZE||keyval == MPI_LASTUSEDCODE) return MPI_ERR_ARG; else if (comm==MPI_COMM_NULL) return MPI_ERR_COMM; else return comm->attr_delete(keyval); } int PMPI_Attr_get(MPI_Comm comm, int keyval, void* attr_value, int* flag) { static int one = 1; static int zero = 0; static int tag_ub = INT_MAX; static int last_used_code = MPI_ERR_LASTCODE; if (comm==MPI_COMM_NULL){ *flag = 0; return MPI_ERR_COMM; } switch (keyval) { case MPI_HOST: case MPI_IO: case MPI_APPNUM: *flag = 1; *static_cast(attr_value) = &zero; return MPI_SUCCESS; case MPI_UNIVERSE_SIZE: *flag = 1; *static_cast(attr_value) = &smpi_universe_size; return MPI_SUCCESS; case MPI_LASTUSEDCODE: *flag = 1; *static_cast(attr_value) = &last_used_code; return MPI_SUCCESS; case MPI_TAG_UB: *flag=1; *static_cast(attr_value) = &tag_ub; return MPI_SUCCESS; case MPI_WTIME_IS_GLOBAL: *flag = 1; *static_cast(attr_value) = &one; return MPI_SUCCESS; default: return comm->attr_get(keyval, attr_value, flag); } } int PMPI_Attr_put(MPI_Comm comm, int keyval, void* attr_value) { if(keyval == MPI_TAG_UB||keyval == MPI_HOST||keyval == MPI_IO ||keyval == MPI_WTIME_IS_GLOBAL||keyval == MPI_APPNUM ||keyval == MPI_UNIVERSE_SIZE||keyval == MPI_LASTUSEDCODE) return MPI_ERR_ARG; else if (comm==MPI_COMM_NULL) return MPI_ERR_COMM; else return comm->attr_put(keyval, attr_value); } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_win.cpp0000644000175000017500000006600313217757317022243 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype_derived.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" #include "smpi_win.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Win_create( void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, MPI_Win *win){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval= MPI_ERR_COMM; }else if ((base == nullptr && size != 0) || disp_unit <= 0 || size < 0 ){ retval= MPI_ERR_OTHER; }else{ *win = new simgrid::smpi::Win( base, size, disp_unit, info, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Win_allocate( MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, void *base, MPI_Win *win){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval= MPI_ERR_COMM; }else if (disp_unit <= 0 || size < 0 ){ retval= MPI_ERR_OTHER; }else{ void* ptr = xbt_malloc(size); if(ptr==nullptr) return MPI_ERR_NO_MEM; *static_cast(base) = ptr; *win = new simgrid::smpi::Win( ptr, size, disp_unit, info, comm,1); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Win_create_dynamic( MPI_Info info, MPI_Comm comm, MPI_Win *win){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval= MPI_ERR_COMM; }else{ *win = new simgrid::smpi::Win(info, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Win_attach(MPI_Win win, void *base, MPI_Aint size){ int retval = 0; smpi_bench_end(); if(win == MPI_WIN_NULL){ retval = MPI_ERR_WIN; } else if ((base == nullptr && size != 0) || size < 0 ){ retval= MPI_ERR_OTHER; }else{ retval = win->attach(base, size); } smpi_bench_begin(); return retval; } int PMPI_Win_detach(MPI_Win win, void* base) { int retval = 0; smpi_bench_end(); if(win == MPI_WIN_NULL){ retval = MPI_ERR_WIN; } else if (base == nullptr){ retval= MPI_ERR_OTHER; }else{ retval = win->detach(base); } smpi_bench_begin(); return retval; } int PMPI_Win_free( MPI_Win* win){ int retval = 0; smpi_bench_end(); if (win == nullptr || *win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; }else{ delete *win; retval=MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Win_set_name(MPI_Win win, char * name) { if (win == MPI_WIN_NULL) { return MPI_ERR_TYPE; } else if (name == nullptr) { return MPI_ERR_ARG; } else { win->set_name(name); return MPI_SUCCESS; } } int PMPI_Win_get_name(MPI_Win win, char * name, int* len) { if (win == MPI_WIN_NULL) { return MPI_ERR_WIN; } else if (name == nullptr) { return MPI_ERR_ARG; } else { win->get_name(name, len); return MPI_SUCCESS; } } int PMPI_Win_get_info(MPI_Win win, MPI_Info* info) { if (win == MPI_WIN_NULL) { return MPI_ERR_WIN; } else { *info = win->info(); return MPI_SUCCESS; } } int PMPI_Win_set_info(MPI_Win win, MPI_Info info) { if (win == MPI_WIN_NULL) { return MPI_ERR_TYPE; } else { win->set_info(info); return MPI_SUCCESS; } } int PMPI_Win_get_group(MPI_Win win, MPI_Group * group){ if (win == MPI_WIN_NULL) { return MPI_ERR_WIN; }else { win->get_group(group); (*group)->ref(); return MPI_SUCCESS; } } int PMPI_Win_fence( int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_fence")); retval = win->fence(assert); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Get", target_rank, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); retval = win->get( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Rget( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win, MPI_Request* request){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if(request == nullptr){ retval = MPI_ERR_REQUEST; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Rget", target_rank, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); retval = win->get( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, request); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); int dst_traced = group->index(target_rank); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Put", dst_traced, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); TRACE_smpi_send(rank, rank, dst_traced, SMPI_RMA_TAG, origin_count*origin_datatype->size()); retval = win->put( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Rput( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win, MPI_Request* request){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if(request == nullptr){ retval = MPI_ERR_REQUEST; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); int dst_traced = group->index(target_rank); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Rput", dst_traced, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); TRACE_smpi_send(rank, rank, dst_traced, SMPI_RMA_TAG, origin_count*origin_datatype->size()); retval = win->put( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, request); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Accumulate", target_rank, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); retval = win->accumulate( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Raccumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request* request){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if (((origin_datatype == MPI_DATATYPE_NULL) || (target_datatype == MPI_DATATYPE_NULL)) || ((not origin_datatype->is_valid()) || (not target_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if(request == nullptr){ retval = MPI_ERR_REQUEST; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Raccumulate", target_rank, origin_datatype->is_replayable() ? origin_count : origin_count * origin_datatype->size(), encode_datatype(origin_datatype))); retval = win->accumulate( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, request); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Get_accumulate(void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0 || result_count <0) || (origin_addr==nullptr && origin_count > 0 && op != MPI_NO_OP) || (result_addr==nullptr && result_count > 0)){ retval = MPI_ERR_COUNT; } else if (((target_datatype == MPI_DATATYPE_NULL) || (result_datatype == MPI_DATATYPE_NULL)) || (((origin_datatype != MPI_DATATYPE_NULL) && (not origin_datatype->is_valid())) || (not target_datatype->is_valid()) || (not result_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Get_accumulate", target_rank, target_datatype->is_replayable() ? target_count : target_count * target_datatype->size(), encode_datatype(target_datatype))); retval = win->get_accumulate( origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Rget_accumulate(void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win, MPI_Request* request){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0 || result_count <0) || (origin_addr==nullptr && origin_count > 0 && op != MPI_NO_OP) || (result_addr==nullptr && result_count > 0)){ retval = MPI_ERR_COUNT; } else if (((target_datatype == MPI_DATATYPE_NULL) || (result_datatype == MPI_DATATYPE_NULL)) || (((origin_datatype != MPI_DATATYPE_NULL) && (not origin_datatype->is_valid())) || (not target_datatype->is_valid()) || (not result_datatype->is_valid()))) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if(request == nullptr){ retval = MPI_ERR_REQUEST; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Rget_accumulate", target_rank, target_datatype->is_replayable() ? target_count : target_count * target_datatype->size(), encode_datatype(target_datatype))); retval = win->get_accumulate( origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op, request); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Fetch_and_op(void *origin_addr, void *result_addr, MPI_Datatype dtype, int target_rank, MPI_Aint target_disp, MPI_Op op, MPI_Win win){ return PMPI_Get_accumulate(origin_addr, origin_addr==nullptr?0:1, dtype, result_addr, 1, dtype, target_rank, target_disp, 1, dtype, op, win); } int PMPI_Compare_and_swap(void* origin_addr, void* compare_addr, void* result_addr, MPI_Datatype datatype, int target_rank, MPI_Aint target_disp, MPI_Win win) { int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (win->dynamic()==0 && target_disp <0){ //in case of dynamic window, target_disp can be mistakenly seen as negative, as it is an address retval = MPI_ERR_ARG; } else if (origin_addr==nullptr || result_addr==nullptr || compare_addr==nullptr){ retval = MPI_ERR_COUNT; } else if ((datatype == MPI_DATATYPE_NULL) || (not datatype->is_valid())) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process()->index(); MPI_Group group; win->get_group(&group); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Compare_and_swap", target_rank, datatype->is_replayable() ? 1 : datatype->size(), encode_datatype(datatype))); retval = win->compare_and_swap(origin_addr, compare_addr, result_addr, datatype, target_rank, target_disp); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Win_post(MPI_Group group, int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (group==MPI_GROUP_NULL){ retval = MPI_ERR_GROUP; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_post")); retval = win->post(group,assert); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Win_start(MPI_Group group, int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (group==MPI_GROUP_NULL){ retval = MPI_ERR_GROUP; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_start")); retval = win->start(group,assert); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Win_complete(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_complete")); retval = win->complete(); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Win_wait(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_wait")); retval = win->wait(); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Win_lock(int lock_type, int rank, int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (lock_type != MPI_LOCK_EXCLUSIVE && lock_type != MPI_LOCK_SHARED) { retval = MPI_ERR_LOCKTYPE; } else if (rank == MPI_PROC_NULL){ retval = MPI_SUCCESS; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __func__, new simgrid::instr::NoOpTIData("Win_lock")); retval = win->lock(lock_type,rank,assert); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_unlock(int rank, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (rank == MPI_PROC_NULL){ retval = MPI_SUCCESS; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_unlock")); retval = win->unlock(rank); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_lock_all(int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_lock_all")); retval = win->lock_all(assert); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_unlock_all(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_unlock_all")); retval = win->unlock_all(); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_flush(int rank, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (rank == MPI_PROC_NULL){ retval = MPI_SUCCESS; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_flush")); retval = win->flush(rank); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_flush_local(int rank, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (rank == MPI_PROC_NULL){ retval = MPI_SUCCESS; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_flush_local")); retval = win->flush_local(rank); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_flush_all(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_flush_all")); retval = win->flush_all(); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_flush_local_all(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int myrank = smpi_process()->index(); TRACE_smpi_comm_in(myrank, __FUNCTION__, new simgrid::instr::NoOpTIData("Win_flush_local_all")); retval = win->flush_local_all(); TRACE_smpi_comm_out(myrank); } smpi_bench_begin(); return retval; } int PMPI_Win_get_attr (MPI_Win win, int keyval, void *attribute_val, int* flag) { static MPI_Aint size; static int disp_unit; if (win==MPI_WIN_NULL) return MPI_ERR_TYPE; else{ switch (keyval) { case MPI_WIN_BASE : *static_cast(attribute_val) = win->base(); *flag = 1; return MPI_SUCCESS; case MPI_WIN_SIZE : size = win->size(); *static_cast(attribute_val) = &size; *flag = 1; return MPI_SUCCESS; case MPI_WIN_DISP_UNIT : disp_unit=win->disp_unit(); *static_cast(attribute_val) = &disp_unit; *flag = 1; return MPI_SUCCESS; default: return win->attr_get(keyval, attribute_val, flag); } } } int PMPI_Win_set_attr (MPI_Win win, int type_keyval, void *attribute_val) { if (win==MPI_WIN_NULL) return MPI_ERR_TYPE; else return win->attr_put(type_keyval, attribute_val); } int PMPI_Win_delete_attr (MPI_Win win, int type_keyval) { if (win==MPI_WIN_NULL) return MPI_ERR_TYPE; else return win->attr_delete(type_keyval); } int PMPI_Win_create_keyval(MPI_Win_copy_attr_function* copy_fn, MPI_Win_delete_attr_function* delete_fn, int* keyval, void* extra_state) { smpi_copy_fn _copy_fn={nullptr, nullptr, copy_fn}; smpi_delete_fn _delete_fn={nullptr, nullptr, delete_fn}; return simgrid::smpi::Keyval::keyval_create(_copy_fn, _delete_fn, keyval, extra_state); } int PMPI_Win_free_keyval(int* keyval) { return simgrid::smpi::Keyval::keyval_free(keyval); } MPI_Win PMPI_Win_f2c(MPI_Fint win){ return static_cast(simgrid::smpi::Win::f2c(win)); } MPI_Fint PMPI_Win_c2f(MPI_Win win){ return win->c2f(); } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_request.cpp0000644000175000017500000005402013217757317023132 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_process.hpp" #include "smpi_request.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = simgrid::smpi::Request::send_init(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (src == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = simgrid::smpi::Request::recv_init(buf, count, datatype, src, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Ssend_init(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request* request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = simgrid::smpi::Request::ssend_init(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Start(MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr || *request == MPI_REQUEST_NULL) { retval = MPI_ERR_REQUEST; } else { (*request)->start(); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Startall(int count, MPI_Request * requests) { int retval; smpi_bench_end(); if (requests == nullptr) { retval = MPI_ERR_ARG; } else { retval = MPI_SUCCESS; for (int i = 0; i < count; i++) { if(requests[i] == MPI_REQUEST_NULL) { retval = MPI_ERR_REQUEST; } } if(retval != MPI_ERR_REQUEST) { simgrid::smpi::Request::startall(count, requests); } } smpi_bench_begin(); return retval; } int PMPI_Request_free(MPI_Request * request) { int retval = 0; smpi_bench_end(); if (*request == MPI_REQUEST_NULL) { retval = MPI_ERR_ARG; } else { simgrid::smpi::Request::unref(request); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (src == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (src!=MPI_ANY_SOURCE && (src >= comm->group()->size() || src <0)){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Irecv", comm->group()->index(src), datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); *request = simgrid::smpi::Request::irecv(buf, count, datatype, src, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (dst >= comm->group()->size() || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int trace_dst = comm->group()->index(dst); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Isend", trace_dst, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); TRACE_smpi_send(rank, rank, trace_dst, tag, count * datatype->size()); *request = simgrid::smpi::Request::isend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); if (retval != MPI_SUCCESS && request!=nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Issend(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request* request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (dst >= comm->group()->size() || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0)|| (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int trace_dst = comm->group()->index(dst); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("ISsend", trace_dst, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); TRACE_smpi_send(rank, rank, trace_dst, tag, count * datatype->size()); *request = simgrid::smpi::Request::issend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); if (retval != MPI_SUCCESS && request!=nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Recv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (src == MPI_PROC_NULL) { simgrid::smpi::Status::empty(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else if (src!=MPI_ANY_SOURCE && (src >= comm->group()->size() || src <0)){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int src_traced = comm->group()->index(src); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("recv", src_traced, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); simgrid::smpi::Request::recv(buf, count, datatype, src, tag, comm, status); retval = MPI_SUCCESS; // the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) if (status != MPI_STATUS_IGNORE) { src_traced = comm->group()->index(status->MPI_SOURCE); if (not TRACE_smpi_view_internals()) { TRACE_smpi_recv(src_traced, rank, tag); } } TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (dst >= comm->group()->size() || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf == nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag < 0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int dst_traced = comm->group()->index(dst); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("send", dst_traced, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); if (not TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, dst_traced, tag,count*datatype->size()); } simgrid::smpi::Request::send(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Ssend(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (dst >= comm->group()->size() || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int dst_traced = comm->group()->index(dst); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Ssend", dst_traced, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); TRACE_smpi_send(rank, rank, dst_traced, tag, count * datatype->size()); simgrid::smpi::Request::ssend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Sendrecv(void* sendbuf, int sendcount, MPI_Datatype sendtype, int dst, int sendtag, void* recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status* status) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not sendtype->is_valid() || not recvtype->is_valid()) { retval = MPI_ERR_TYPE; } else if (src == MPI_PROC_NULL || dst == MPI_PROC_NULL) { simgrid::smpi::Status::empty(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; }else if (dst >= comm->group()->size() || dst <0 || (src!=MPI_ANY_SOURCE && (src >= comm->group()->size() || src <0))){ retval = MPI_ERR_RANK; } else if ((sendcount < 0 || recvcount<0) || (sendbuf==nullptr && sendcount > 0) || (recvbuf==nullptr && recvcount>0)) { retval = MPI_ERR_COUNT; } else if((sendtag<0 && sendtag != MPI_ANY_TAG)||(recvtag<0 && recvtag != MPI_ANY_TAG)){ retval = MPI_ERR_TAG; } else { int rank = smpi_process()->index(); int dst_traced = comm->group()->index(dst); int src_traced = comm->group()->index(src); // FIXME: Hack the way to trace this one std::vector* dst_hack = new std::vector; std::vector* src_hack = new std::vector; dst_hack->push_back(dst_traced); src_hack->push_back(src_traced); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "sendRecv", -1, sendtype->is_replayable() ? sendcount : sendcount * sendtype->size(), dst_hack, recvtype->is_replayable() ? recvcount : recvcount * recvtype->size(), src_hack, encode_datatype(sendtype), encode_datatype(recvtype))); TRACE_smpi_send(rank, rank, dst_traced, sendtag, sendcount * sendtype->size()); simgrid::smpi::Request::sendrecv(sendbuf, sendcount, sendtype, dst, sendtag, recvbuf, recvcount, recvtype, src, recvtag, comm, status); retval = MPI_SUCCESS; TRACE_smpi_recv(src_traced, rank, recvtag); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Sendrecv_replace(void* buf, int count, MPI_Datatype datatype, int dst, int sendtag, int src, int recvtag, MPI_Comm comm, MPI_Status* status) { int retval = 0; if (not datatype->is_valid()) { return MPI_ERR_TYPE; } else if (count < 0) { return MPI_ERR_COUNT; } else { int size = datatype->get_extent() * count; void* recvbuf = xbt_new0(char, size); retval = MPI_Sendrecv(buf, count, datatype, dst, sendtag, recvbuf, count, datatype, src, recvtag, comm, status); if(retval==MPI_SUCCESS){ simgrid::smpi::Datatype::copy(recvbuf, count, datatype, buf, count, datatype); } xbt_free(recvbuf); } return retval; } int PMPI_Test(MPI_Request * request, int *flag, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (request == nullptr || flag == nullptr) { retval = MPI_ERR_ARG; } else if (*request == MPI_REQUEST_NULL) { *flag= true; simgrid::smpi::Status::empty(status); retval = MPI_SUCCESS; } else { int rank = ((*request)->comm() != MPI_COMM_NULL) ? smpi_process()->index() : -1; TRACE_smpi_testing_in(rank); *flag = simgrid::smpi::Request::test(request,status); TRACE_smpi_testing_out(rank); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testany(int count, MPI_Request requests[], int *index, int *flag, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (index == nullptr || flag == nullptr) { retval = MPI_ERR_ARG; } else { *flag = simgrid::smpi::Request::testany(count, requests, index, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testall(int count, MPI_Request* requests, int* flag, MPI_Status* statuses) { int retval = 0; smpi_bench_end(); if (flag == nullptr) { retval = MPI_ERR_ARG; } else { *flag = simgrid::smpi::Request::testall(count, requests, statuses); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status* status) { int retval = 0; smpi_bench_end(); if (status == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (source == MPI_PROC_NULL) { simgrid::smpi::Status::empty(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else { simgrid::smpi::Request::probe(source, tag, comm, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status) { int retval = 0; smpi_bench_end(); if (flag == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (source == MPI_PROC_NULL) { *flag=true; simgrid::smpi::Status::empty(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else { simgrid::smpi::Request::iprobe(source, tag, comm, flag, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Wait(MPI_Request * request, MPI_Status * status) { int retval = 0; smpi_bench_end(); simgrid::smpi::Status::empty(status); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (*request == MPI_REQUEST_NULL) { retval = MPI_SUCCESS; } else { int rank = (*request)->comm() != MPI_COMM_NULL ? smpi_process()->index() : -1; int src_traced = (*request)->src(); int dst_traced = (*request)->dst(); int tag_traced= (*request)->tag(); MPI_Comm comm = (*request)->comm(); int is_wait_for_receive = ((*request)->flags() & RECV); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("wait")); simgrid::smpi::Request::wait(request, status); retval = MPI_SUCCESS; //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) TRACE_smpi_comm_out(rank); if (is_wait_for_receive) { if(src_traced==MPI_ANY_SOURCE) src_traced = (status != MPI_STATUS_IGNORE) ? comm->group()->rank(status->MPI_SOURCE) : src_traced; TRACE_smpi_recv(src_traced, dst_traced, tag_traced); } } smpi_bench_begin(); return retval; } int PMPI_Waitany(int count, MPI_Request requests[], int *index, MPI_Status * status) { if (index == nullptr) return MPI_ERR_ARG; if (count <= 0) return MPI_SUCCESS; smpi_bench_end(); //save requests information for tracing struct savedvalstype { int src; int dst; int recv; int tag; MPI_Comm comm; }; savedvalstype* savedvals = xbt_new0(savedvalstype, count); for (int i = 0; i < count; i++) { MPI_Request req = requests[i]; //already received requests are no longer valid if (req) { savedvals[i]=(savedvalstype){req->src(), req->dst(), (req->flags() & RECV), req->tag(), req->comm()}; } } int rank_traced = smpi_process()->index(); TRACE_smpi_comm_in(rank_traced, __FUNCTION__, new simgrid::instr::CpuTIData("waitAny", static_cast(count))); *index = simgrid::smpi::Request::waitany(count, requests, status); if(*index!=MPI_UNDEFINED){ int src_traced = savedvals[*index].src; //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) int dst_traced = savedvals[*index].dst; int is_wait_for_receive = savedvals[*index].recv; if (is_wait_for_receive) { if(savedvals[*index].src==MPI_ANY_SOURCE) src_traced = (status != MPI_STATUSES_IGNORE) ? savedvals[*index].comm->group()->rank(status->MPI_SOURCE) : savedvals[*index].src; TRACE_smpi_recv(src_traced, dst_traced, savedvals[*index].tag); } TRACE_smpi_comm_out(rank_traced); } xbt_free(savedvals); smpi_bench_begin(); return MPI_SUCCESS; } int PMPI_Waitall(int count, MPI_Request requests[], MPI_Status status[]) { smpi_bench_end(); //save information from requests struct savedvalstype { int src; int dst; int recv; int tag; int valid; MPI_Comm comm; }; savedvalstype* savedvals=xbt_new0(savedvalstype, count); for (int i = 0; i < count; i++) { MPI_Request req = requests[i]; if(req!=MPI_REQUEST_NULL){ savedvals[i]=(savedvalstype){req->src(), req->dst(), (req->flags() & RECV), req->tag(), 1, req->comm()}; }else{ savedvals[i].valid=0; } } int rank_traced = smpi_process()->index(); TRACE_smpi_comm_in(rank_traced, __FUNCTION__, new simgrid::instr::CpuTIData("waitAll", static_cast(count))); int retval = simgrid::smpi::Request::waitall(count, requests, status); for (int i = 0; i < count; i++) { if(savedvals[i].valid){ // the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) int src_traced = savedvals[i].src; int dst_traced = savedvals[i].dst; int is_wait_for_receive = savedvals[i].recv; if (is_wait_for_receive) { if(src_traced==MPI_ANY_SOURCE) src_traced = (status != MPI_STATUSES_IGNORE) ? savedvals[i].comm->group()->rank(status[i].MPI_SOURCE) : savedvals[i].src; TRACE_smpi_recv(src_traced, dst_traced,savedvals[i].tag); } } } TRACE_smpi_comm_out(rank_traced); xbt_free(savedvals); smpi_bench_begin(); return retval; } int PMPI_Waitsome(int incount, MPI_Request requests[], int *outcount, int *indices, MPI_Status status[]) { int retval = 0; smpi_bench_end(); if (outcount == nullptr) { retval = MPI_ERR_ARG; } else { *outcount = simgrid::smpi::Request::waitsome(incount, requests, indices, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testsome(int incount, MPI_Request requests[], int* outcount, int* indices, MPI_Status status[]) { int retval = 0; smpi_bench_end(); if (outcount == nullptr) { retval = MPI_ERR_ARG; } else { *outcount = simgrid::smpi::Request::testsome(incount, requests, indices, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } MPI_Request PMPI_Request_f2c(MPI_Fint request){ return static_cast(simgrid::smpi::Request::f2c(request)); } MPI_Fint PMPI_Request_c2f(MPI_Request request) { return request->c2f(); } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi.cpp0000644000175000017500000001347013217757317021366 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "simgrid/s4u/Engine.hpp" #include "simgrid/s4u/Host.hpp" #include "smpi_comm.hpp" #include "smpi_datatype_derived.hpp" #include "smpi_process.hpp" #include "smpi_status.hpp" #include "src/simix/ActorImpl.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_pmpi, smpi, "Logging specific to SMPI (pmpi)"); //this function need to be here because of the calls to smpi_bench void TRACE_smpi_set_category(const char *category) { //need to end bench otherwise categories for execution tasks are wrong smpi_bench_end(); TRACE_internal_smpi_set_category (category); //begin bench after changing process's category smpi_bench_begin(); } /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Init(int *argc, char ***argv) { xbt_assert(simgrid::s4u::Engine::isInitialized(), "Your MPI program was not properly initialized. The easiest is to use smpirun to start it."); // PMPI_Init is called only once per SMPI process int already_init; MPI_Initialized(&already_init); if(already_init == 0){ simgrid::smpi::Process::init(argc, argv); smpi_process()->mark_as_initialized(); int rank = smpi_process()->index(); TRACE_smpi_init(rank); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("init")); TRACE_smpi_comm_out(rank); TRACE_smpi_computing_init(rank); smpi_bench_begin(); } smpi_mpi_init(); return MPI_SUCCESS; } int PMPI_Finalize() { smpi_bench_end(); int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("finalize")); smpi_process()->finalize(); TRACE_smpi_comm_out(rank); TRACE_smpi_finalize(rank); return MPI_SUCCESS; } int PMPI_Finalized(int* flag) { *flag=smpi_process()!=nullptr ? smpi_process()->finalized() : 0; return MPI_SUCCESS; } int PMPI_Get_version (int *version,int *subversion){ *version = MPI_VERSION; *subversion= MPI_SUBVERSION; return MPI_SUCCESS; } int PMPI_Get_library_version (char *version,int *len){ smpi_bench_end(); snprintf(version, MPI_MAX_LIBRARY_VERSION_STRING, "SMPI Version %d.%d. Copyright The Simgrid Team 2007-2017", SIMGRID_VERSION_MAJOR, SIMGRID_VERSION_MINOR); *len = strlen(version) > MPI_MAX_LIBRARY_VERSION_STRING ? MPI_MAX_LIBRARY_VERSION_STRING : strlen(version); smpi_bench_begin(); return MPI_SUCCESS; } int PMPI_Init_thread(int* argc, char*** argv, int /*required*/, int* provided) { if (provided != nullptr) { *provided = MPI_THREAD_SINGLE; } return MPI_Init(argc, argv); } int PMPI_Query_thread(int *provided) { if (provided == nullptr) { return MPI_ERR_ARG; } else { *provided = MPI_THREAD_SINGLE; return MPI_SUCCESS; } } int PMPI_Is_thread_main(int *flag) { if (flag == nullptr) { return MPI_ERR_ARG; } else { *flag = smpi_process()->index() == 0; return MPI_SUCCESS; } } int PMPI_Abort(MPI_Comm /*comm*/, int /*errorcode*/) { smpi_bench_end(); // FIXME: should kill all processes in comm instead smx_actor_t process = SIMIX_process_self(); simgrid::simix::kernelImmediate([process] { SIMIX_process_kill(process, process); }); return MPI_SUCCESS; } double PMPI_Wtime() { return smpi_mpi_wtime(); } extern double sg_maxmin_precision; double PMPI_Wtick() { return sg_maxmin_precision; } int PMPI_Address(void *location, MPI_Aint * address) { if (address==nullptr) { return MPI_ERR_ARG; } else { *address = reinterpret_cast(location); return MPI_SUCCESS; } } int PMPI_Get_address(void *location, MPI_Aint * address) { return PMPI_Address(location, address); } int PMPI_Get_processor_name(char *name, int *resultlen) { strncpy(name, sg_host_self()->getCname(), strlen(sg_host_self()->getCname()) < MPI_MAX_PROCESSOR_NAME - 1 ? strlen(sg_host_self()->getCname()) + 1 : MPI_MAX_PROCESSOR_NAME - 1); *resultlen = strlen(name) > MPI_MAX_PROCESSOR_NAME ? MPI_MAX_PROCESSOR_NAME : strlen(name); return MPI_SUCCESS; } int PMPI_Get_count(MPI_Status * status, MPI_Datatype datatype, int *count) { if (status == nullptr || count == nullptr) { return MPI_ERR_ARG; } else if (not datatype->is_valid()) { return MPI_ERR_TYPE; } else { size_t size = datatype->size(); if (size == 0) { *count = 0; return MPI_SUCCESS; } else if (status->count % size != 0) { return MPI_UNDEFINED; } else { *count = simgrid::smpi::Status::get_count(status, datatype); return MPI_SUCCESS; } } } int PMPI_Initialized(int* flag) { *flag=(smpi_process()!=nullptr && smpi_process()->initialized()); return MPI_SUCCESS; } int PMPI_Alloc_mem(MPI_Aint size, MPI_Info /*info*/, void* baseptr) { void *ptr = xbt_malloc(size); if(ptr==nullptr) return MPI_ERR_NO_MEM; else { *static_cast(baseptr) = ptr; return MPI_SUCCESS; } } int PMPI_Free_mem(void *baseptr){ xbt_free(baseptr); return MPI_SUCCESS; } int PMPI_Error_class(int errorcode, int* errorclass) { // assume smpi uses only standard mpi error codes *errorclass=errorcode; return MPI_SUCCESS; } int PMPI_Keyval_create(MPI_Copy_function* copy_fn, MPI_Delete_function* delete_fn, int* keyval, void* extra_state) { smpi_copy_fn _copy_fn={copy_fn,nullptr,nullptr}; smpi_delete_fn _delete_fn={delete_fn,nullptr,nullptr}; return simgrid::smpi::Keyval::keyval_create(_copy_fn, _delete_fn, keyval, extra_state); } int PMPI_Keyval_free(int* keyval) { return simgrid::smpi::Keyval::keyval_free(keyval); } } // extern "C" SimGrid-3.18/src/smpi/bindings/smpi_pmpi_group.cpp0000644000175000017500000001135513217757317022602 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype_derived.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Group_free(MPI_Group * group) { if (group == nullptr) { return MPI_ERR_ARG; } else { if(*group != MPI_COMM_WORLD->group() && *group != MPI_GROUP_EMPTY) simgrid::smpi::Group::unref(*group); *group = MPI_GROUP_NULL; return MPI_SUCCESS; } } int PMPI_Group_size(MPI_Group group, int *size) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (size == nullptr) { return MPI_ERR_ARG; } else { *size = group->size(); return MPI_SUCCESS; } } int PMPI_Group_rank(MPI_Group group, int *rank) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (rank == nullptr) { return MPI_ERR_ARG; } else { *rank = group->rank(smpi_process()->index()); return MPI_SUCCESS; } } int PMPI_Group_translate_ranks(MPI_Group group1, int n, int *ranks1, MPI_Group group2, int *ranks2) { if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else { for (int i = 0; i < n; i++) { if(ranks1[i]==MPI_PROC_NULL){ ranks2[i]=MPI_PROC_NULL; }else{ int index = group1->index(ranks1[i]); ranks2[i] = group2->rank(index); } } return MPI_SUCCESS; } } int PMPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result) { if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (result == nullptr) { return MPI_ERR_ARG; } else { *result = group1->compare(group2); return MPI_SUCCESS; } } int PMPI_Group_union(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { return group1->group_union(group2, newgroup); } } int PMPI_Group_intersection(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { return group1->intersection(group2,newgroup); } } int PMPI_Group_difference(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { return group1->difference(group2,newgroup); } } int PMPI_Group_incl(MPI_Group group, int n, int *ranks, MPI_Group * newgroup) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { return group->incl(n, ranks, newgroup); } } int PMPI_Group_excl(MPI_Group group, int n, int *ranks, MPI_Group * newgroup) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { if (n == 0) { *newgroup = group; if (group != MPI_COMM_WORLD->group() && group != MPI_COMM_SELF->group() && group != MPI_GROUP_EMPTY) group->ref(); return MPI_SUCCESS; } else if (n == group->size()) { *newgroup = MPI_GROUP_EMPTY; return MPI_SUCCESS; } else { return group->excl(n,ranks,newgroup); } } } int PMPI_Group_range_incl(MPI_Group group, int n, int ranges[][3], MPI_Group * newgroup) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { if (n == 0) { *newgroup = MPI_GROUP_EMPTY; return MPI_SUCCESS; } else { return group->range_incl(n,ranges,newgroup); } } } int PMPI_Group_range_excl(MPI_Group group, int n, int ranges[][3], MPI_Group * newgroup) { if (group == MPI_GROUP_NULL) { return MPI_ERR_GROUP; } else if (newgroup == nullptr) { return MPI_ERR_ARG; } else { if (n == 0) { *newgroup = group; if (group != MPI_COMM_WORLD->group() && group != MPI_COMM_SELF->group() && group != MPI_GROUP_EMPTY) group->ref(); return MPI_SUCCESS; } else { return group->range_excl(n,ranges,newgroup); } } } MPI_Group PMPI_Group_f2c(MPI_Fint group){ return simgrid::smpi::Group::f2c(group); } MPI_Fint PMPI_Group_c2f(MPI_Group group){ return group->c2f(); } } SimGrid-3.18/src/smpi/bindings/smpi_f77.cpp0000644000175000017500000007471113217757317021031 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" #include "smpi_request.hpp" #include "smpi_win.hpp" static int running_processes = 0; static void smpi_init_fortran_types(){ if(simgrid::smpi::F2C::lookup() == nullptr){ MPI_COMM_WORLD->add_f(); MPI_BYTE->add_f();//MPI_BYTE MPI_CHAR->add_f();//MPI_CHARACTER #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) MPI_C_BOOL->add_f();//MPI_LOGICAL MPI_INT->add_f();//MPI_INTEGER #else MPI_C_BOOL->add_f();//MPI_LOGICAL MPI_LONG->add_f();//MPI_INTEGER #endif MPI_INT8_T->add_f();//MPI_INTEGER1 MPI_INT16_T->add_f();//MPI_INTEGER2 MPI_INT32_T->add_f();//MPI_INTEGER4 MPI_INT64_T->add_f();//MPI_INTEGER8 MPI_REAL->add_f();//MPI_REAL MPI_REAL4->add_f();//MPI_REAL4 MPI_REAL8->add_f();//MPI_REAL8 MPI_DOUBLE->add_f();//MPI_DOUBLE_PRECISION MPI_C_FLOAT_COMPLEX->add_f();//MPI_COMPLEX MPI_C_DOUBLE_COMPLEX->add_f();//MPI_DOUBLE_COMPLEX #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) MPI_2INT->add_f();//MPI_2INTEGER #else MPI_2LONG->add_f();//MPI_2INTEGER #endif MPI_UINT8_T->add_f();//MPI_LOGICAL1 MPI_UINT16_T->add_f();//MPI_LOGICAL2 MPI_UINT32_T->add_f();//MPI_LOGICAL4 MPI_UINT64_T->add_f();//MPI_LOGICAL8 MPI_2FLOAT->add_f();//MPI_2REAL MPI_2DOUBLE->add_f();//MPI_2DOUBLE_PRECISION MPI_PTR->add_f();//MPI_AINT MPI_OFFSET->add_f();//MPI_OFFSET MPI_AINT->add_f();//MPI_COUNT MPI_REAL16->add_f();//MPI_REAL16 MPI_PACKED->add_f();//MPI_PACKED MPI_MAX->add_f(); MPI_MIN->add_f(); MPI_MAXLOC->add_f(); MPI_MINLOC->add_f(); MPI_SUM->add_f(); MPI_PROD->add_f(); MPI_LAND->add_f(); MPI_LOR->add_f(); MPI_LXOR->add_f(); MPI_BAND->add_f(); MPI_BOR->add_f(); MPI_BXOR->add_f(); } } extern "C" { // This should really use the C linkage to be usable from Fortran void mpi_init_(int* ierr) { smpi_init_fortran_types(); *ierr = MPI_Init(nullptr, nullptr); running_processes++; } void mpi_finalize_(int* ierr) { *ierr = MPI_Finalize(); running_processes--; if(running_processes==0){ simgrid::smpi::F2C::delete_lookup(); } } void mpi_abort_(int* comm, int* errorcode, int* ierr) { *ierr = MPI_Abort(simgrid::smpi::Comm::f2c(*comm), *errorcode); } double mpi_wtime_() { return MPI_Wtime(); } double mpi_wtick_() { return MPI_Wtick(); } void mpi_group_incl_(int* group, int* n, int* ranks, int* group_out, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_incl(simgrid::smpi::Group::f2c(*group), *n, ranks, &tmp); if(*ierr == MPI_SUCCESS) { *group_out = tmp->add_f(); } } void mpi_initialized_(int* flag, int* ierr){ *ierr = MPI_Initialized(flag); } void mpi_get_processor_name_(char *name, int *resultlen, int* ierr){ *ierr = MPI_Get_processor_name(name, resultlen); } void mpi_get_count_(MPI_Status * status, int* datatype, int *count, int* ierr){ *ierr = MPI_Get_count(FORT_STATUS_IGNORE(status), simgrid::smpi::Datatype::f2c(*datatype), count); } void mpi_attr_get_(int* comm, int* keyval, void* attr_value, int* flag, int* ierr ){ *ierr = MPI_Attr_get(simgrid::smpi::Comm::f2c(*comm), *keyval, attr_value, flag); } void mpi_error_string_(int* errorcode, char* string, int* resultlen, int* ierr){ *ierr = MPI_Error_string(*errorcode, string, resultlen); } void mpi_win_fence_( int* assert, int* win, int* ierr){ *ierr = MPI_Win_fence(* assert, simgrid::smpi::Win::f2c(*win)); } void mpi_win_free_( int* win, int* ierr){ MPI_Win tmp = simgrid::smpi::Win::f2c(*win); *ierr = MPI_Win_free(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::F2C::free_f(*win); } } void mpi_win_create_( int *base, MPI_Aint* size, int* disp_unit, int* info, int* comm, int *win, int* ierr){ MPI_Win tmp; *ierr = MPI_Win_create( static_cast(base), *size, *disp_unit, simgrid::smpi::Info::f2c(*info), simgrid::smpi::Comm::f2c(*comm),&tmp); if(*ierr == MPI_SUCCESS) { *win = tmp->add_f(); } } void mpi_win_post_(int* group, int assert, int* win, int* ierr){ *ierr = MPI_Win_post(simgrid::smpi::Group::f2c(*group), assert, simgrid::smpi::Win::f2c(*win)); } void mpi_win_start_(int* group, int assert, int* win, int* ierr){ *ierr = MPI_Win_start(simgrid::smpi::Group::f2c(*group), assert, simgrid::smpi::Win::f2c(*win)); } void mpi_win_complete_(int* win, int* ierr){ *ierr = MPI_Win_complete(simgrid::smpi::Win::f2c(*win)); } void mpi_win_wait_(int* win, int* ierr){ *ierr = MPI_Win_wait(simgrid::smpi::Win::f2c(*win)); } void mpi_win_set_name_ (int* win, char * name, int* ierr, int size){ //handle trailing blanks while(name[size-1]==' ') size--; while(*name==' '){//handle leading blanks size--; name++; } char* tname = xbt_new(char,size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Win_set_name(simgrid::smpi::Win::f2c(*win), tname); xbt_free(tname); } void mpi_win_get_name_ (int* win, char * name, int* len, int* ierr){ *ierr = MPI_Win_get_name(simgrid::smpi::Win::f2c(*win),name,len); if(*len>0) name[*len]=' ';//blank padding, not \0 } void mpi_win_allocate_( MPI_Aint* size, int* disp_unit, int* info, int* comm, void* base, int* win, int* ierr){ MPI_Win tmp; *ierr = MPI_Win_allocate( *size, *disp_unit, simgrid::smpi::Info::f2c(*info), simgrid::smpi::Comm::f2c(*comm),static_cast(base),&tmp); if(*ierr == MPI_SUCCESS) { *win = tmp->add_f(); } } void mpi_win_attach_(int* win, int* base, MPI_Aint* size, int* ierr){ *ierr = MPI_Win_attach(simgrid::smpi::Win::f2c(*win), static_cast(base), *size); } void mpi_win_create_dynamic_( int* info, int* comm, int *win, int* ierr){ MPI_Win tmp; *ierr = MPI_Win_create_dynamic( simgrid::smpi::Info::f2c(*info), simgrid::smpi::Comm::f2c(*comm),&tmp); if(*ierr == MPI_SUCCESS) { *win = tmp->add_f(); } } void mpi_win_detach_(int* win, int* base, int* ierr){ *ierr = MPI_Win_detach(simgrid::smpi::Win::f2c(*win), static_cast(base)); } void mpi_win_set_info_(int* win, int* info, int* ierr){ *ierr = MPI_Win_set_info(simgrid::smpi::Win::f2c(*win), simgrid::smpi::Info::f2c(*info)); } void mpi_win_get_info_(int* win, int* info, int* ierr){ MPI_Info tmp; *ierr = MPI_Win_get_info(simgrid::smpi::Win::f2c(*win), &tmp); if(*ierr == MPI_SUCCESS) { *info = tmp->add_f(); } } void mpi_win_get_group_(int* win, int* group, int* ierr){ MPI_Group tmp; *ierr = MPI_Win_get_group(simgrid::smpi::Win::f2c(*win), &tmp); if(*ierr == MPI_SUCCESS) { *group = tmp->add_f(); } } void mpi_win_get_attr_(int* win, int* type_keyval, void* attribute_val, int* flag, int* ierr){ *ierr = MPI_Win_get_attr(simgrid::smpi::Win::f2c(*win), *type_keyval, attribute_val, flag); } void mpi_win_set_attr_(int* win, int* type_keyval, void* att, int* ierr){ *ierr = MPI_Win_set_attr(simgrid::smpi::Win::f2c(*win), *type_keyval, att); } void mpi_win_delete_attr_(int* win, int* comm_keyval, int* ierr){ *ierr = MPI_Win_delete_attr (simgrid::smpi::Win::f2c(*win), *comm_keyval); } void mpi_win_create_keyval_(void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr){ *ierr = MPI_Win_create_keyval(reinterpret_cast(copy_fn), reinterpret_cast(delete_fn), keyval, extra_state); } void mpi_win_free_keyval_(int* keyval, int* ierr){ *ierr = MPI_Win_free_keyval( keyval); } void mpi_win_lock_(int* lock_type, int* rank, int* assert, int* win, int* ierr){ *ierr = MPI_Win_lock(*lock_type, *rank, *assert, simgrid::smpi::Win::f2c(*win)); } void mpi_win_lock_all_(int* assert, int* win, int* ierr){ *ierr = MPI_Win_lock_all(*assert, simgrid::smpi::Win::f2c(*win)); } void mpi_win_unlock_(int* rank, int* win, int* ierr){ *ierr = MPI_Win_unlock(*rank, simgrid::smpi::Win::f2c(*win)); } void mpi_win_unlock_all_(int* win, int* ierr){ *ierr = MPI_Win_unlock_all(simgrid::smpi::Win::f2c(*win)); } void mpi_win_flush_(int* rank, int* win, int* ierr){ *ierr = MPI_Win_flush(*rank, simgrid::smpi::Win::f2c(*win)); } void mpi_win_flush_local_(int* rank, int* win, int* ierr){ *ierr = MPI_Win_flush_local(*rank, simgrid::smpi::Win::f2c(*win)); } void mpi_win_flush_all_(int* win, int* ierr){ *ierr = MPI_Win_flush_all(simgrid::smpi::Win::f2c(*win)); } void mpi_win_flush_local_all_(int* win, int* ierr){ *ierr = MPI_Win_flush_local_all(simgrid::smpi::Win::f2c(*win)); } void mpi_info_create_( int *info, int* ierr){ MPI_Info tmp; *ierr = MPI_Info_create(&tmp); if(*ierr == MPI_SUCCESS) { *info = tmp->add_f(); } } void mpi_info_set_( int *info, char *key, char *value, int* ierr, unsigned int keylen, unsigned int valuelen){ //handle trailing blanks while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char,keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; while(value[valuelen-1]==' ') valuelen--; while(*value==' '){//handle leading blanks valuelen--; value++; } char* tvalue = xbt_new(char,valuelen+1); strncpy(tvalue, value, valuelen); tvalue[valuelen]='\0'; *ierr = MPI_Info_set( simgrid::smpi::Info::f2c(*info), tkey, tvalue); xbt_free(tkey); xbt_free(tvalue); } void mpi_info_get_ (int* info,char *key,int* valuelen, char *value, int *flag, int* ierr, unsigned int keylen ){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char,keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_get(simgrid::smpi::Info::f2c(*info),tkey,*valuelen, value, flag); xbt_free(tkey); if(*flag!=0){ int replace=0; int i=0; for (i=0; i<*valuelen; i++){ if(value[i]=='\0') replace=1; if(replace) value[i]=' '; } } } void mpi_info_free_(int* info, int* ierr){ MPI_Info tmp = simgrid::smpi::Info::f2c(*info); *ierr = MPI_Info_free(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::F2C::free_f(*info); } } void mpi_get_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* ierr){ *ierr = MPI_Get( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Win::f2c(*win)); } void mpi_rget_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* request, int* ierr){ MPI_Request req; *ierr = MPI_Rget( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Win::f2c(*win), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_accumulate_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* op, int* win, int* ierr){ *ierr = MPI_Accumulate( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Win::f2c(*win)); } void mpi_raccumulate_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* op, int* win, int* request, int* ierr){ MPI_Request req; *ierr = MPI_Raccumulate( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Win::f2c(*win),&req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_put_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* ierr){ *ierr = MPI_Put( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Win::f2c(*win)); } void mpi_rput_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* request, int* ierr){ MPI_Request req; *ierr = MPI_Rput( static_cast(origin_addr),*origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype),*target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*tarsmpi_type_f2c), simgrid::smpi::Win::f2c(*win),&req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_fetch_and_op_( int *origin_addr, int* result_addr, int* datatype, int* target_rank, MPI_Aint* target_disp, int* op, int* win, int* ierr){ *ierr = MPI_Fetch_and_op( static_cast(origin_addr), static_cast(result_addr), simgrid::smpi::Datatype::f2c(*datatype),*target_rank, *target_disp, simgrid::smpi::Op::f2c(*op), simgrid::smpi::Win::f2c(*win)); } void mpi_compare_and_swap_(int* origin_addr, int* compare_addr, int* result_addr, int* datatype, int* target_rank, MPI_Aint* target_disp, int* win, int* ierr) { *ierr = MPI_Compare_and_swap( static_cast(origin_addr),static_cast(compare_addr), static_cast(result_addr), simgrid::smpi::Datatype::f2c(*datatype),*target_rank, *target_disp, simgrid::smpi::Win::f2c(*win)); } void mpi_get_accumulate_(int *origin_addr, int* origin_count, int* origin_datatype, int* result_addr, int* result_count, int* result_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* ierr){ *ierr = MPI_Get_accumulate(static_cast(origin_addr), *origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype), static_cast(result_addr), *result_count, simgrid::smpi::Datatype::f2c(*result_datatype), *target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*target_datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Win::f2c(*win)); } void mpi_rget_accumulate_(int *origin_addr, int* origin_count, int* origin_datatype, int* result_addr, int* result_count, int* result_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* request, int* ierr){ MPI_Request req; *ierr = MPI_Rget_accumulate(static_cast(origin_addr), *origin_count, simgrid::smpi::Datatype::f2c(*origin_datatype), static_cast(result_addr), *result_count, simgrid::smpi::Datatype::f2c(*result_datatype), *target_rank, *target_disp, *target_count, simgrid::smpi::Datatype::f2c(*target_datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Win::f2c(*win), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } //following are automatically generated, and have to be checked void mpi_finalized_ (int * flag, int* ierr){ *ierr = MPI_Finalized(flag); } void mpi_init_thread_ (int* required, int *provided, int* ierr){ smpi_init_fortran_types(); *ierr = MPI_Init_thread(nullptr, nullptr,*required, provided); running_processes++; } void mpi_query_thread_ (int *provided, int* ierr){ *ierr = MPI_Query_thread(provided); } void mpi_is_thread_main_ (int *flag, int* ierr){ *ierr = MPI_Is_thread_main(flag); } void mpi_address_ (void *location, MPI_Aint * address, int* ierr){ *ierr = MPI_Address(location, address); } void mpi_get_address_ (void *location, MPI_Aint * address, int* ierr){ *ierr = MPI_Get_address(location, address); } void mpi_pcontrol_ (int* level , int* ierr){ *ierr = MPI_Pcontrol(*static_cast(level)); } void mpi_op_create_ (void * function, int* commute, int* op, int* ierr){ MPI_Op tmp; *ierr = MPI_Op_create(reinterpret_cast(function),*commute, &tmp); if(*ierr == MPI_SUCCESS) { tmp->set_fortran_op(); *op = tmp->add_f(); } } void mpi_op_free_ (int* op, int* ierr){ MPI_Op tmp= simgrid::smpi::Op::f2c(*op); *ierr = MPI_Op_free(& tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::F2C::free_f(*op); } } void mpi_op_commutative_ (int* op, int* commute, int* ierr){ *ierr = MPI_Op_commutative(simgrid::smpi::Op::f2c(*op), commute); } void mpi_group_free_ (int* group, int* ierr){ MPI_Group tmp = simgrid::smpi::Group::f2c(*group); *ierr = MPI_Group_free(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::F2C::free_f(*group); } } void mpi_group_size_ (int* group, int *size, int* ierr){ *ierr = MPI_Group_size(simgrid::smpi::Group::f2c(*group), size); } void mpi_group_rank_ (int* group, int *rank, int* ierr){ *ierr = MPI_Group_rank(simgrid::smpi::Group::f2c(*group), rank); } void mpi_group_translate_ranks_ (int* group1, int* n, int *ranks1, int* group2, int *ranks2, int* ierr) { *ierr = MPI_Group_translate_ranks(simgrid::smpi::Group::f2c(*group1), *n, ranks1, simgrid::smpi::Group::f2c(*group2), ranks2); } void mpi_group_compare_ (int* group1, int* group2, int *result, int* ierr){ *ierr = MPI_Group_compare(simgrid::smpi::Group::f2c(*group1), simgrid::smpi::Group::f2c(*group2), result); } void mpi_group_union_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_union(simgrid::smpi::Group::f2c(*group1), simgrid::smpi::Group::f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_group_intersection_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_intersection(simgrid::smpi::Group::f2c(*group1), simgrid::smpi::Group::f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_group_difference_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_difference(simgrid::smpi::Group::f2c(*group1), simgrid::smpi::Group::f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_group_excl_ (int* group, int* n, int *ranks, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_excl(simgrid::smpi::Group::f2c(*group), *n, ranks, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_group_range_incl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_range_incl(simgrid::smpi::Group::f2c(*group), *n, ranges, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_group_range_excl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_range_excl(simgrid::smpi::Group::f2c(*group), *n, ranges, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = tmp->add_f(); } } void mpi_request_free_ (int* request, int* ierr){ MPI_Request tmp=simgrid::smpi::Request::f2c(*request); *ierr = MPI_Request_free(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::Request::free_f(*request); } } void mpi_pack_size_ (int* incount, int* datatype, int* comm, int* size, int* ierr) { *ierr = MPI_Pack_size(*incount, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Comm::f2c(*comm), size); } void mpi_cart_coords_ (int* comm, int* rank, int* maxdims, int* coords, int* ierr) { *ierr = MPI_Cart_coords(simgrid::smpi::Comm::f2c(*comm), *rank, *maxdims, coords); } void mpi_cart_create_ (int* comm_old, int* ndims, int* dims, int* periods, int* reorder, int* comm_cart, int* ierr) { MPI_Comm tmp; *ierr = MPI_Cart_create(simgrid::smpi::Comm::f2c(*comm_old), *ndims, dims, periods, *reorder, &tmp); if(*ierr == MPI_SUCCESS) { *comm_cart = tmp->add_f(); } } void mpi_cart_get_ (int* comm, int* maxdims, int* dims, int* periods, int* coords, int* ierr) { *ierr = MPI_Cart_get(simgrid::smpi::Comm::f2c(*comm), *maxdims, dims, periods, coords); } void mpi_cart_map_ (int* comm_old, int* ndims, int* dims, int* periods, int* newrank, int* ierr) { *ierr = MPI_Cart_map(simgrid::smpi::Comm::f2c(*comm_old), *ndims, dims, periods, newrank); } void mpi_cart_rank_ (int* comm, int* coords, int* rank, int* ierr) { *ierr = MPI_Cart_rank(simgrid::smpi::Comm::f2c(*comm), coords, rank); } void mpi_cart_shift_ (int* comm, int* direction, int* displ, int* source, int* dest, int* ierr) { *ierr = MPI_Cart_shift(simgrid::smpi::Comm::f2c(*comm), *direction, *displ, source, dest); } void mpi_cart_sub_ (int* comm, int* remain_dims, int* comm_new, int* ierr) { MPI_Comm tmp; *ierr = MPI_Cart_sub(simgrid::smpi::Comm::f2c(*comm), remain_dims, &tmp); if(*ierr == MPI_SUCCESS) { *comm_new = tmp->add_f(); } } void mpi_cartdim_get_ (int* comm, int* ndims, int* ierr) { *ierr = MPI_Cartdim_get(simgrid::smpi::Comm::f2c(*comm), ndims); } void mpi_graph_create_ (int* comm_old, int* nnodes, int* index, int* edges, int* reorder, int* comm_graph, int* ierr) { MPI_Comm tmp; *ierr = MPI_Graph_create(simgrid::smpi::Comm::f2c(*comm_old), *nnodes, index, edges, *reorder, &tmp); if(*ierr == MPI_SUCCESS) { *comm_graph = tmp->add_f(); } } void mpi_graph_get_ (int* comm, int* maxindex, int* maxedges, int* index, int* edges, int* ierr) { *ierr = MPI_Graph_get(simgrid::smpi::Comm::f2c(*comm), *maxindex, *maxedges, index, edges); } void mpi_graph_map_ (int* comm_old, int* nnodes, int* index, int* edges, int* newrank, int* ierr) { *ierr = MPI_Graph_map(simgrid::smpi::Comm::f2c(*comm_old), *nnodes, index, edges, newrank); } void mpi_graph_neighbors_ (int* comm, int* rank, int* maxneighbors, int* neighbors, int* ierr) { *ierr = MPI_Graph_neighbors(simgrid::smpi::Comm::f2c(*comm), *rank, *maxneighbors, neighbors); } void mpi_graph_neighbors_count_ (int* comm, int* rank, int* nneighbors, int* ierr) { *ierr = MPI_Graph_neighbors_count(simgrid::smpi::Comm::f2c(*comm), *rank, nneighbors); } void mpi_graphdims_get_ (int* comm, int* nnodes, int* nedges, int* ierr) { *ierr = MPI_Graphdims_get(simgrid::smpi::Comm::f2c(*comm), nnodes, nedges); } void mpi_topo_test_ (int* comm, int* top_type, int* ierr) { *ierr = MPI_Topo_test(simgrid::smpi::Comm::f2c(*comm), top_type); } void mpi_error_class_ (int* errorcode, int* errorclass, int* ierr) { *ierr = MPI_Error_class(*errorcode, errorclass); } void mpi_errhandler_create_ (void* function, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_create(reinterpret_cast(function), static_cast(errhandler)); } void mpi_errhandler_free_ (void* errhandler, int* ierr) { *ierr = MPI_Errhandler_free(static_cast(errhandler)); } void mpi_errhandler_get_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_get(simgrid::smpi::Comm::f2c(*comm), static_cast(errhandler)); } void mpi_errhandler_set_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(simgrid::smpi::Comm::f2c(*comm), *static_cast(errhandler)); } void mpi_cancel_ (int* request, int* ierr) { MPI_Request tmp=simgrid::smpi::Request::f2c(*request); *ierr = MPI_Cancel(&tmp); } void mpi_buffer_attach_ (void* buffer, int* size, int* ierr) { *ierr = MPI_Buffer_attach(buffer, *size); } void mpi_buffer_detach_ (void* buffer, int* size, int* ierr) { *ierr = MPI_Buffer_detach(buffer, size); } void mpi_intercomm_create_ (int* local_comm, int *local_leader, int* peer_comm, int* remote_leader, int* tag, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Intercomm_create(simgrid::smpi::Comm::f2c(*local_comm), *local_leader, simgrid::smpi::Comm::f2c(*peer_comm), *remote_leader, *tag, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = tmp->add_f(); } } void mpi_intercomm_merge_ (int* comm, int* high, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Intercomm_merge(simgrid::smpi::Comm::f2c(*comm), *high, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = tmp->add_f(); } } void mpi_attr_delete_ (int* comm, int* keyval, int* ierr) { *ierr = MPI_Attr_delete(simgrid::smpi::Comm::f2c(*comm), *keyval); } void mpi_attr_put_ (int* comm, int* keyval, void* attr_value, int* ierr) { *ierr = MPI_Attr_put(simgrid::smpi::Comm::f2c(*comm), *keyval, attr_value); } void mpi_keyval_create_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr) { *ierr = MPI_Keyval_create(reinterpret_cast(copy_fn),reinterpret_cast(delete_fn), keyval, extra_state); } void mpi_keyval_free_ (int* keyval, int* ierr) { *ierr = MPI_Keyval_free(keyval); } void mpi_test_cancelled_ (MPI_Status* status, int* flag, int* ierr) { *ierr = MPI_Test_cancelled(status, flag); } void mpi_get_elements_ (MPI_Status* status, int* datatype, int* elements, int* ierr) { *ierr = MPI_Get_elements(status, simgrid::smpi::Datatype::f2c(*datatype), elements); } void mpi_dims_create_ (int* nnodes, int* ndims, int* dims, int* ierr) { *ierr = MPI_Dims_create(*nnodes, *ndims, dims); } void mpi_add_error_class_ ( int *errorclass, int* ierr){ *ierr = MPI_Add_error_class( errorclass); } void mpi_add_error_code_ ( int* errorclass, int *errorcode, int* ierr){ *ierr = MPI_Add_error_code(*errorclass, errorcode); } void mpi_add_error_string_ ( int* errorcode, char *string, int* ierr){ *ierr = MPI_Add_error_string(*errorcode, string); } void mpi_info_dup_ (int* info, int* newinfo, int* ierr){ MPI_Info tmp; *ierr = MPI_Info_dup(simgrid::smpi::Info::f2c(*info), &tmp); if(*ierr==MPI_SUCCESS){ *newinfo= tmp->add_f(); } } void mpi_info_get_valuelen_ ( int* info, char *key, int *valuelen, int *flag, int* ierr, unsigned int keylen){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char, keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_get_valuelen( simgrid::smpi::Info::f2c(*info), tkey, valuelen, flag); xbt_free(tkey); } void mpi_info_delete_ (int* info, char *key, int* ierr, unsigned int keylen){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char, keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_delete(simgrid::smpi::Info::f2c(*info), tkey); xbt_free(tkey); } void mpi_info_get_nkeys_ ( int* info, int *nkeys, int* ierr){ *ierr = MPI_Info_get_nkeys( simgrid::smpi::Info::f2c(*info), nkeys); } void mpi_info_get_nthkey_ ( int* info, int* n, char *key, int* ierr, unsigned int keylen){ *ierr = MPI_Info_get_nthkey( simgrid::smpi::Info::f2c(*info), *n, key); unsigned int i = 0; for (i=strlen(key); i(query_fn), reinterpret_cast(free_fn), reinterpret_cast(cancel_fn), extra_state, &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_grequest_complete_ ( int* request, int* ierr){ *ierr = MPI_Grequest_complete( simgrid::smpi::Request::f2c(*request)); } void mpi_status_set_cancelled_ (MPI_Status* status,int* flag, int* ierr){ *ierr = MPI_Status_set_cancelled(status,*flag); } void mpi_status_set_elements_ ( MPI_Status* status, int* datatype, int* count, int* ierr){ *ierr = MPI_Status_set_elements( status, simgrid::smpi::Datatype::f2c(*datatype), *count); } void mpi_publish_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Publish_name( service_name, *reinterpret_cast(info), port_name); } void mpi_unpublish_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Unpublish_name( service_name, *reinterpret_cast(info), port_name); } void mpi_lookup_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Lookup_name( service_name, *reinterpret_cast(info), port_name); } void mpi_open_port_ ( int* info, char *port_name, int* ierr){ *ierr = MPI_Open_port( *reinterpret_cast(info),port_name); } void mpi_close_port_ ( char *port_name, int* ierr){ *ierr = MPI_Close_port( port_name); } void mpi_file_close_ ( int* file, int* ierr){ *ierr= MPI_File_close(reinterpret_cast(*file)); } void mpi_file_delete_ ( char* filename, int* info, int* ierr){ *ierr= MPI_File_delete(filename, simgrid::smpi::Info::f2c(*info)); } void mpi_file_open_ ( int* comm, char* filename, int* amode, int* info, int* fh, int* ierr){ *ierr= MPI_File_open(simgrid::smpi::Comm::f2c(*comm), filename, *amode, simgrid::smpi::Info::f2c(*info), reinterpret_cast(*fh)); } void mpi_file_set_view_ ( int* fh, long long int* offset, int* etype, int* filetype, char* datarep, int* info, int* ierr){ *ierr= MPI_File_set_view(reinterpret_cast(*fh) , reinterpret_cast(*offset), simgrid::smpi::Datatype::f2c(*etype), simgrid::smpi::Datatype::f2c(*filetype), datarep, simgrid::smpi::Info::f2c(*info)); } void mpi_file_read_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr){ *ierr= MPI_File_read(reinterpret_cast(*fh), buf, *count, simgrid::smpi::Datatype::f2c(*datatype), status); } void mpi_file_write_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr){ *ierr= MPI_File_write(reinterpret_cast(*fh), buf, *count, simgrid::smpi::Datatype::f2c(*datatype), status); } } // extern "C" SimGrid-3.18/src/smpi/bindings/smpi_pmpi_op.cpp0000644000175000017500000000240313217757317022056 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_op.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Op_create(MPI_User_function * function, int commute, MPI_Op * op) { if (function == nullptr || op == nullptr) { return MPI_ERR_ARG; } else { *op = new simgrid::smpi::Op(function, (commute!=0)); return MPI_SUCCESS; } } int PMPI_Op_free(MPI_Op * op) { if (op == nullptr) { return MPI_ERR_ARG; } else if (*op == MPI_OP_NULL) { return MPI_ERR_OP; } else { delete (*op); *op = MPI_OP_NULL; return MPI_SUCCESS; } } int PMPI_Op_commutative(MPI_Op op, int* commute){ if (op == MPI_OP_NULL) { return MPI_ERR_OP; } else if (commute==nullptr){ return MPI_ERR_ARG; } else { *commute = op->is_commutative(); return MPI_SUCCESS; } } MPI_Op PMPI_Op_f2c(MPI_Fint op){ return static_cast(simgrid::smpi::Op::f2c(op)); } MPI_Fint PMPI_Op_c2f(MPI_Op op){ return op->c2f(); } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_topo.cpp0000644000175000017500000001013213217757317022417 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage /* The topo part of MPI_COMM_WORLD should always be nullptr. When other topologies will be implemented, not only should we * check if the topology is nullptr, but we should check if it is the good topology type (so we have to add a * MPIR_Topo_Type field, and replace the MPI_Topology field by an union)*/ int PMPI_Cart_create(MPI_Comm comm_old, int ndims, int* dims, int* periodic, int reorder, MPI_Comm* comm_cart) { if (comm_old == MPI_COMM_NULL){ return MPI_ERR_COMM; } else if (ndims < 0 || (ndims > 0 && (dims == nullptr || periodic == nullptr)) || comm_cart == nullptr) { return MPI_ERR_ARG; } else{ simgrid::smpi::Topo_Cart* topo = new simgrid::smpi::Topo_Cart(comm_old, ndims, dims, periodic, reorder, comm_cart); if(*comm_cart==MPI_COMM_NULL) delete topo; return MPI_SUCCESS; } } int PMPI_Cart_rank(MPI_Comm comm, int* coords, int* rank) { if(comm == MPI_COMM_NULL || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if (coords == nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } return topo->rank(coords, rank); } int PMPI_Cart_shift(MPI_Comm comm, int direction, int displ, int* source, int* dest) { if(comm == MPI_COMM_NULL || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if (source == nullptr || dest == nullptr || direction < 0 ) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } return topo->shift(direction, displ, source, dest); } int PMPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int* coords) { if(comm == MPI_COMM_NULL || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if (rank < 0 || rank >= comm->size()) { return MPI_ERR_RANK; } if (maxdims <= 0) { return MPI_ERR_ARG; } if(coords == nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } return topo->coords(rank, maxdims, coords); } int PMPI_Cart_get(MPI_Comm comm, int maxdims, int* dims, int* periods, int* coords) { if(comm == nullptr || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if(maxdims <= 0 || dims == nullptr || periods == nullptr || coords == nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } return topo->get(maxdims, dims, periods, coords); } int PMPI_Cartdim_get(MPI_Comm comm, int* ndims) { if (comm == MPI_COMM_NULL || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if (ndims == nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } return topo->dim_get(ndims); } int PMPI_Dims_create(int nnodes, int ndims, int* dims) { if(dims == nullptr) { return MPI_ERR_ARG; } if (ndims < 1 || nnodes < 1) { return MPI_ERR_DIMS; } return simgrid::smpi::Topo_Cart::Dims_create(nnodes, ndims, dims); } int PMPI_Cart_sub(MPI_Comm comm, int* remain_dims, MPI_Comm* comm_new) { if(comm == MPI_COMM_NULL || comm->topo() == nullptr) { return MPI_ERR_TOPOLOGY; } if (comm_new == nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology topo = static_cast(comm->topo()); if (topo==nullptr) { return MPI_ERR_ARG; } MPIR_Cart_Topology cart = topo->sub(remain_dims, comm_new); if(*comm_new==MPI_COMM_NULL) delete cart; if(cart==nullptr) return MPI_ERR_ARG; return MPI_SUCCESS; } } SimGrid-3.18/src/smpi/bindings/smpi_f77_coll.cpp0000644000175000017500000001607013217757317022034 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" extern "C" { // This should really use the C linkage to be usable from Fortran void mpi_barrier_(int* comm, int* ierr) { *ierr = MPI_Barrier(simgrid::smpi::Comm::f2c(*comm)); } void mpi_bcast_(void *buf, int* count, int* datatype, int* root, int* comm, int* ierr) { *ierr = MPI_Bcast(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_reduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Reduce(sendbuf, recvbuf, *count, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_allreduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allreduce(sendbuf, recvbuf, *count, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Comm::f2c(*comm)); } void mpi_reduce_scatter_(void* sendbuf, void* recvbuf, int* recvcounts, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Reduce_scatter(sendbuf, recvbuf, recvcounts, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Comm::f2c(*comm)); } void mpi_scatter_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { recvbuf = static_cast( FORT_IN_PLACE(recvbuf)); *ierr = MPI_Scatter(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_scatterv_(void* sendbuf, int* sendcounts, int* displs, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { recvbuf = static_cast( FORT_IN_PLACE(recvbuf)); *ierr = MPI_Scatterv(sendbuf, sendcounts, displs, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_gather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = sendbuf!=MPI_IN_PLACE ? static_cast( FORT_BOTTOM(sendbuf)) : MPI_IN_PLACE; recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Gather(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_gatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts, int* displs, int* recvtype, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = sendbuf!=MPI_IN_PLACE ? static_cast( FORT_BOTTOM(sendbuf)) : MPI_IN_PLACE; recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Gatherv(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, recvcounts, displs, simgrid::smpi::Datatype::f2c(*recvtype), *root, simgrid::smpi::Comm::f2c(*comm)); } void mpi_allgather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allgather(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), simgrid::smpi::Comm::f2c(*comm)); } void mpi_allgatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts,int* displs, int* recvtype, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allgatherv(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, recvcounts, displs, simgrid::smpi::Datatype::f2c(*recvtype), simgrid::smpi::Comm::f2c(*comm)); } void mpi_scan_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr) { *ierr = MPI_Scan(sendbuf, recvbuf, *count, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Comm::f2c(*comm)); } void mpi_alltoall_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr) { *ierr = MPI_Alltoall(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), simgrid::smpi::Comm::f2c(*comm)); } void mpi_alltoallv_(void* sendbuf, int* sendcounts, int* senddisps, int* sendtype, void* recvbuf, int* recvcounts, int* recvdisps, int* recvtype, int* comm, int* ierr) { *ierr = MPI_Alltoallv(sendbuf, sendcounts, senddisps, simgrid::smpi::Datatype::f2c(*sendtype), recvbuf, recvcounts, recvdisps, simgrid::smpi::Datatype::f2c(*recvtype), simgrid::smpi::Comm::f2c(*comm)); } void mpi_reduce_local_ (void *inbuf, void *inoutbuf, int* count, int* datatype, int* op, int* ierr){ *ierr = MPI_Reduce_local(inbuf, inoutbuf, *count, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op)); } void mpi_reduce_scatter_block_ (void *sendbuf, void *recvbuf, int* recvcount, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Reduce_scatter_block(sendbuf, recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Comm::f2c(*comm)); } void mpi_alltoallw_ ( void *sendbuf, int *sendcnts, int *sdispls, int* sendtypes, void *recvbuf, int *recvcnts, int *rdispls, int* recvtypes, int* comm, int* ierr){ *ierr = MPI_Alltoallw( sendbuf, sendcnts, sdispls, reinterpret_cast(sendtypes), recvbuf, recvcnts, rdispls, reinterpret_cast(recvtypes), simgrid::smpi::Comm::f2c(*comm)); } void mpi_exscan_ (void *sendbuf, void *recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr){ *ierr = MPI_Exscan(sendbuf, recvbuf, *count, simgrid::smpi::Datatype::f2c(*datatype), simgrid::smpi::Op::f2c(*op), simgrid::smpi::Comm::f2c(*comm)); } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_coll.cpp0000644000175000017500000005634213217757317022404 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype_derived.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_ARG; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("bcast", comm->group()->index(root), -1.0, datatype->is_replayable() ? count : count * datatype->size(), -1, encode_datatype(datatype), "")); if (comm->size() > 1) simgrid::smpi::Colls::bcast(buf, count, datatype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Barrier(MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("barrier")); simgrid::smpi::Colls::barrier(comm); //Barrier can be used to synchronize RMA calls. Finish all requests from comm before. comm->finish_rma_calls(); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Gather(void *sendbuf, int sendcount, MPI_Datatype sendtype,void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || ((comm->rank() == root) && (recvtype == MPI_DATATYPE_NULL))){ retval = MPI_ERR_TYPE; } else if ((( sendbuf != MPI_IN_PLACE) && (sendcount <0)) || ((comm->rank() == root) && (recvcount <0))){ retval = MPI_ERR_COUNT; } else { char* sendtmpbuf = static_cast(sendbuf); int sendtmpcount = sendcount; MPI_Datatype sendtmptype = sendtype; if( (comm->rank() == root) && (sendbuf == MPI_IN_PLACE )) { sendtmpcount=0; sendtmptype=recvtype; } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData( "gather", comm->group()->index(root), -1.0, sendtmptype->is_replayable() ? sendtmpcount : sendtmpcount * sendtmptype->size(), (comm->rank() != root || recvtype->is_replayable()) ? recvcount : recvcount * recvtype->size(), encode_datatype(sendtmptype), encode_datatype(recvtype))); simgrid::smpi::Colls::gather(sendtmpbuf, sendtmpcount, sendtmptype, recvbuf, recvcount, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || ((comm->rank() == root) && (recvtype == MPI_DATATYPE_NULL))){ retval = MPI_ERR_TYPE; } else if (( sendbuf != MPI_IN_PLACE) && (sendcount <0)){ retval = MPI_ERR_COUNT; } else if (recvcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else { char* sendtmpbuf = static_cast(sendbuf); int sendtmpcount = sendcount; MPI_Datatype sendtmptype = sendtype; if( (comm->rank() == root) && (sendbuf == MPI_IN_PLACE )) { sendtmpcount=0; sendtmptype=recvtype; } int rank = smpi_process()->index(); int dt_size_recv = recvtype->is_replayable() ? 1 : recvtype->size(); std::vector* trace_recvcounts = new std::vector; if (comm->rank() == root) { for (int i = 0; i < comm->size(); i++) // copy data to avoid bad free trace_recvcounts->push_back(recvcounts[i] * dt_size_recv); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "gatherV", comm->group()->index(root), sendtmptype->is_replayable() ? sendtmpcount : sendtmpcount * sendtmptype->size(), nullptr, dt_size_recv, trace_recvcounts, encode_datatype(sendtmptype), encode_datatype(recvtype))); retval = simgrid::smpi::Colls::gatherv(sendtmpbuf, sendtmpcount, sendtmptype, recvbuf, recvcounts, displs, recvtype, root, comm); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int retval = MPI_SUCCESS; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || (recvtype == MPI_DATATYPE_NULL)){ retval = MPI_ERR_TYPE; } else if ((( sendbuf != MPI_IN_PLACE) && (sendcount <0)) || (recvcount <0)){ retval = MPI_ERR_COUNT; } else { if(sendbuf == MPI_IN_PLACE) { sendbuf=static_cast(recvbuf)+recvtype->get_extent()*recvcount*comm->rank(); sendcount=recvcount; sendtype=recvtype; } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("allGather", -1, -1.0, sendtype->is_replayable() ? sendcount : sendcount * sendtype->size(), recvtype->is_replayable() ? recvcount : recvcount * recvtype->size(), encode_datatype(sendtype), encode_datatype(recvtype))); simgrid::smpi::Colls::allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (((sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || (recvtype == MPI_DATATYPE_NULL)) { retval = MPI_ERR_TYPE; } else if (( sendbuf != MPI_IN_PLACE) && (sendcount <0)){ retval = MPI_ERR_COUNT; } else if (recvcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else { if(sendbuf == MPI_IN_PLACE) { sendbuf=static_cast(recvbuf)+recvtype->get_extent()*displs[comm->rank()]; sendcount=recvcounts[comm->rank()]; sendtype=recvtype; } int rank = smpi_process()->index(); int dt_size_recv = recvtype->is_replayable() ? 1 : recvtype->size(); std::vector* trace_recvcounts = new std::vector; for (int i = 0; i < comm->size(); i++) // copy data to avoid bad free trace_recvcounts->push_back(recvcounts[i] * dt_size_recv); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "allGatherV", -1, sendtype->is_replayable() ? sendcount : sendcount * sendtype->size(), nullptr, dt_size_recv, trace_recvcounts, encode_datatype(sendtype), encode_datatype(recvtype))); simgrid::smpi::Colls::allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Scatter(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (((comm->rank() == root) && (not sendtype->is_valid())) || ((recvbuf != MPI_IN_PLACE) && (not recvtype->is_valid()))) { retval = MPI_ERR_TYPE; } else if ((sendbuf == recvbuf) || ((comm->rank()==root) && sendcount>0 && (sendbuf == nullptr))){ retval = MPI_ERR_BUFFER; }else { if (recvbuf == MPI_IN_PLACE) { recvtype = sendtype; recvcount = sendcount; } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData( "scatter", comm->group()->index(root), -1.0, (comm->rank() != root || sendtype->is_replayable()) ? sendcount : sendcount * sendtype->size(), recvtype->is_replayable() ? recvcount : recvcount * recvtype->size(), encode_datatype(sendtype), encode_datatype(recvtype))); simgrid::smpi::Colls::scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Scatterv(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (sendcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else if (((comm->rank() == root) && (sendtype == MPI_DATATYPE_NULL)) || ((recvbuf != MPI_IN_PLACE) && (recvtype == MPI_DATATYPE_NULL))) { retval = MPI_ERR_TYPE; } else { if (recvbuf == MPI_IN_PLACE) { recvtype = sendtype; recvcount = sendcounts[comm->rank()]; } int rank = smpi_process()->index(); int dt_size_send = sendtype->is_replayable() ? 1 : sendtype->size(); std::vector* trace_sendcounts = new std::vector; if (comm->rank() == root) { for (int i = 0; i < comm->size(); i++) // copy data to avoid bad free trace_sendcounts->push_back(sendcounts[i] * dt_size_send); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "scatterV", comm->group()->index(root), dt_size_send, trace_sendcounts, recvtype->is_replayable() ? recvcount : recvcount * recvtype->size(), nullptr, encode_datatype(sendtype), encode_datatype(recvtype))); retval = simgrid::smpi::Colls::scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid() || op == MPI_OP_NULL) { retval = MPI_ERR_ARG; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("reduce", comm->group()->index(root), 0, datatype->is_replayable() ? count : count * datatype->size(), -1, encode_datatype(datatype), "")); simgrid::smpi::Colls::reduce(sendbuf, recvbuf, count, datatype, op, root, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Reduce_local(void *inbuf, void *inoutbuf, int count, MPI_Datatype datatype, MPI_Op op){ int retval = 0; smpi_bench_end(); if (not datatype->is_valid() || op == MPI_OP_NULL) { retval = MPI_ERR_ARG; } else { op->apply(inbuf, inoutbuf, &count, datatype); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { char* sendtmpbuf = static_cast(sendbuf); if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = static_cast(xbt_malloc(count*datatype->get_extent())); simgrid::smpi::Datatype::copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("allReduce", -1, 0, datatype->is_replayable() ? count : count * datatype->size(), -1, encode_datatype(datatype), "")); simgrid::smpi::Colls::allreduce(sendtmpbuf, recvbuf, count, datatype, op, comm); if( sendbuf == MPI_IN_PLACE ) xbt_free(sendtmpbuf); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData( "scan", -1, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); retval = simgrid::smpi::Colls::scan(sendbuf, recvbuf, count, datatype, op, comm); TRACE_smpi_comm_out(rank); } smpi_bench_begin(); return retval; } int PMPI_Exscan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = smpi_process()->index(); void* sendtmpbuf = sendbuf; if (sendbuf == MPI_IN_PLACE) { sendtmpbuf = static_cast(xbt_malloc(count * datatype->size())); memcpy(sendtmpbuf, recvbuf, count * datatype->size()); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData( "exscan", -1, datatype->is_replayable() ? count : count * datatype->size(), encode_datatype(datatype))); retval = simgrid::smpi::Colls::exscan(sendtmpbuf, recvbuf, count, datatype, op, comm); TRACE_smpi_comm_out(rank); if (sendbuf == MPI_IN_PLACE) xbt_free(sendtmpbuf); } smpi_bench_begin(); return retval; } int PMPI_Reduce_scatter(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if (recvcounts == nullptr) { retval = MPI_ERR_ARG; } else { int rank = smpi_process()->index(); std::vector* trace_recvcounts = new std::vector; int dt_send_size = datatype->is_replayable() ? 1 : datatype->size(); int totalcount = 0; for (int i = 0; i < comm->size(); i++) { // copy data to avoid bad free trace_recvcounts->push_back(recvcounts[i] * dt_send_size); totalcount += recvcounts[i]; } void* sendtmpbuf = sendbuf; if (sendbuf == MPI_IN_PLACE) { sendtmpbuf = static_cast(xbt_malloc(totalcount * datatype->size())); memcpy(sendtmpbuf, recvbuf, totalcount * datatype->size()); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData("reduceScatter", -1, dt_send_size, nullptr, -1, trace_recvcounts, encode_datatype(datatype), "")); simgrid::smpi::Colls::reduce_scatter(sendtmpbuf, recvbuf, recvcounts, datatype, op, comm); retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); if (sendbuf == MPI_IN_PLACE) xbt_free(sendtmpbuf); } smpi_bench_begin(); return retval; } int PMPI_Reduce_scatter_block(void *sendbuf, void *recvbuf, int recvcount, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (not datatype->is_valid()) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if (recvcount < 0) { retval = MPI_ERR_ARG; } else { int count = comm->size(); int rank = smpi_process()->index(); int dt_send_size = datatype->is_replayable() ? 1 : datatype->size(); std::vector* trace_recvcounts = new std::vector(recvcount * dt_send_size); // copy data to avoid bad free void* sendtmpbuf = sendbuf; if (sendbuf == MPI_IN_PLACE) { sendtmpbuf = static_cast(xbt_malloc(recvcount * count * datatype->size())); memcpy(sendtmpbuf, recvbuf, recvcount * count * datatype->size()); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData("reduceScatter", -1, 0, nullptr, -1, trace_recvcounts, encode_datatype(datatype), "")); int* recvcounts = new int[count]; for (int i = 0; i < count; i++) recvcounts[i] = recvcount; simgrid::smpi::Colls::reduce_scatter(sendtmpbuf, recvbuf, recvcounts, datatype, op, comm); delete[] recvcounts; retval = MPI_SUCCESS; TRACE_smpi_comm_out(rank); if (sendbuf == MPI_IN_PLACE) xbt_free(sendtmpbuf); } smpi_bench_begin(); return retval; } int PMPI_Alltoall(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((sendbuf != MPI_IN_PLACE && sendtype == MPI_DATATYPE_NULL) || recvtype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process()->index(); void* sendtmpbuf = static_cast(sendbuf); int sendtmpcount = sendcount; MPI_Datatype sendtmptype = sendtype; if (sendbuf == MPI_IN_PLACE) { sendtmpbuf = static_cast(xbt_malloc(recvcount * comm->size() * recvtype->size())); memcpy(sendtmpbuf, recvbuf, recvcount * comm->size() * recvtype->size()); sendtmpcount = recvcount; sendtmptype = recvtype; } TRACE_smpi_comm_in( rank, __FUNCTION__, new simgrid::instr::CollTIData("allToAll", -1, -1.0, sendtmptype->is_replayable() ? sendtmpcount : sendtmpcount * sendtmptype->size(), recvtype->is_replayable() ? recvcount : recvcount * recvtype->size(), encode_datatype(sendtmptype), encode_datatype(recvtype))); retval = simgrid::smpi::Colls::alltoall(sendtmpbuf, sendtmpcount, sendtmptype, recvbuf, recvcount, recvtype, comm); TRACE_smpi_comm_out(rank); if (sendbuf == MPI_IN_PLACE) xbt_free(sendtmpbuf); } smpi_bench_begin(); return retval; } int PMPI_Alltoallv(void* sendbuf, int* sendcounts, int* senddisps, MPI_Datatype sendtype, void* recvbuf, int* recvcounts, int* recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (sendtype == MPI_DATATYPE_NULL || recvtype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if ((sendbuf != MPI_IN_PLACE && (sendcounts == nullptr || senddisps == nullptr)) || recvcounts == nullptr || recvdisps == nullptr) { retval = MPI_ERR_ARG; } else { int rank = smpi_process()->index(); int size = comm->size(); int send_size = 0; int recv_size = 0; std::vector* trace_sendcounts = new std::vector; std::vector* trace_recvcounts = new std::vector; int dt_size_recv = recvtype->size(); void* sendtmpbuf = static_cast(sendbuf); int* sendtmpcounts = sendcounts; int* sendtmpdisps = senddisps; MPI_Datatype sendtmptype = sendtype; int maxsize = 0; for (int i = 0; i < size; i++) { // copy data to avoid bad free recv_size += recvcounts[i] * dt_size_recv; trace_recvcounts->push_back(recvcounts[i] * dt_size_recv); if (((recvdisps[i] + recvcounts[i]) * dt_size_recv) > maxsize) maxsize = (recvdisps[i] + recvcounts[i]) * dt_size_recv; } if (sendbuf == MPI_IN_PLACE) { sendtmpbuf = static_cast(xbt_malloc(maxsize)); memcpy(sendtmpbuf, recvbuf, maxsize); sendtmpcounts = static_cast(xbt_malloc(size * sizeof(int))); memcpy(sendtmpcounts, recvcounts, size * sizeof(int)); sendtmpdisps = static_cast(xbt_malloc(size * sizeof(int))); memcpy(sendtmpdisps, recvdisps, size * sizeof(int)); sendtmptype = recvtype; } int dt_size_send = sendtmptype->size(); for (int i = 0; i < size; i++) { // copy data to avoid bad free send_size += sendtmpcounts[i] * dt_size_send; trace_sendcounts->push_back(sendtmpcounts[i] * dt_size_send); } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "allToAllV", -1, send_size, trace_sendcounts, recv_size, trace_recvcounts, encode_datatype(sendtype), encode_datatype(recvtype))); retval = simgrid::smpi::Colls::alltoallv(sendtmpbuf, sendtmpcounts, sendtmpdisps, sendtmptype, recvbuf, recvcounts, recvdisps, recvtype, comm); TRACE_smpi_comm_out(rank); if (sendbuf == MPI_IN_PLACE) { xbt_free(sendtmpbuf); xbt_free(sendtmpcounts); xbt_free(sendtmpdisps); } } smpi_bench_begin(); return retval; } } SimGrid-3.18/src/smpi/bindings/smpi_f77_request.cpp0000644000175000017500000002506313217757317022575 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_request.hpp" extern "C" { // This should really use the C linkage to be usable from Fortran void mpi_send_init_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Send_init(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *tag, simgrid::smpi::Comm::f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_isend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Isend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *tag, simgrid::smpi::Comm::f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_irsend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Irsend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *tag, simgrid::smpi::Comm::f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_send_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr) { buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Send(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *tag, simgrid::smpi::Comm::f2c(*comm)); } void mpi_rsend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr) { buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Rsend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *tag, simgrid::smpi::Comm::f2c(*comm)); } void mpi_sendrecv_(void* sendbuf, int* sendcount, int* sendtype, int* dst, int* sendtag, void *recvbuf, int* recvcount, int* recvtype, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr) { sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Sendrecv(sendbuf, *sendcount, simgrid::smpi::Datatype::f2c(*sendtype), *dst, *sendtag, recvbuf, *recvcount, simgrid::smpi::Datatype::f2c(*recvtype), *src, *recvtag, simgrid::smpi::Comm::f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_recv_init_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Recv_init(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *src, *tag, simgrid::smpi::Comm::f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_irecv_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Irecv(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *src, *tag, simgrid::smpi::Comm::f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = req->add_f(); } } void mpi_recv_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, MPI_Status* status, int* ierr) { buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Recv(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *src, *tag, simgrid::smpi::Comm::f2c(*comm), status); } void mpi_sendrecv_replace_ (void *buf, int* count, int* datatype, int* dst, int* sendtag, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr) { *ierr = MPI_Sendrecv_replace(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dst, *sendtag, *src, *recvtag, simgrid::smpi::Comm::f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_ssend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr) { *ierr = MPI_Ssend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm)); } void mpi_ssend_init_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Ssend_init(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_bsend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* ierr) { *ierr = MPI_Bsend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm)); } void mpi_bsend_init_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Bsend_init(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_ibsend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Ibsend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_issend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Issend(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_rsend_init_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Rsend_init(buf, *count, simgrid::smpi::Datatype::f2c(*datatype), *dest, *tag, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = tmp->add_f(); } } void mpi_start_(int* request, int* ierr) { MPI_Request req = simgrid::smpi::Request::f2c(*request); *ierr = MPI_Start(&req); } void mpi_startall_(int* count, int* requests, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr = MPI_Startall(*count, reqs); xbt_free(reqs); } void mpi_wait_(int* request, MPI_Status* status, int* ierr) { MPI_Request req = simgrid::smpi::Request::f2c(*request); *ierr = MPI_Wait(&req, FORT_STATUS_IGNORE(status)); if(req==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(*request); *request=MPI_FORTRAN_REQUEST_NULL; } } void mpi_waitany_(int* count, int* requests, int* index, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr = MPI_Waitany(*count, reqs, index, status); if(reqs[*index]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[*index]); requests[*index]=MPI_FORTRAN_REQUEST_NULL; } xbt_free(reqs); } void mpi_waitall_(int* count, int* requests, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr = MPI_Waitall(*count, reqs, FORT_STATUSES_IGNORE(status)); for(i = 0; i < *count; i++) { if(reqs[i]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[i]); requests[i]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_waitsome_ (int* incount, int* requests, int *outcount, int *indices, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *incount); for(i = 0; i < *incount; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr = MPI_Waitsome(*incount, reqs, outcount, indices, status); for(i=0;i<*outcount;i++){ if(reqs[indices[i]]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[indices[i]]); requests[indices[i]]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_test_ (int * request, int *flag, MPI_Status * status, int* ierr){ MPI_Request req = simgrid::smpi::Request::f2c(*request); *ierr= MPI_Test(&req, flag, FORT_STATUS_IGNORE(status)); if(req==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(*request); *request=MPI_FORTRAN_REQUEST_NULL; } } void mpi_testall_ (int* count, int * requests, int *flag, MPI_Status * statuses, int* ierr){ int i; MPI_Request* reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr= MPI_Testall(*count, reqs, flag, FORT_STATUSES_IGNORE(statuses)); for(i = 0; i < *count; i++) { if(reqs[i]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[i]); requests[i]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_testany_ (int* count, int* requests, int *index, int *flag, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); } *ierr = MPI_Testany(*count, reqs, index, flag, FORT_STATUS_IGNORE(status)); if(*index!=MPI_UNDEFINED && reqs[*index]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[*index]); requests[*index]=MPI_FORTRAN_REQUEST_NULL; } xbt_free(reqs); } void mpi_testsome_ (int* incount, int* requests, int* outcount, int* indices, MPI_Status* statuses, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *incount); for(i = 0; i < *incount; i++) { reqs[i] = simgrid::smpi::Request::f2c(requests[i]); indices[i]=0; } *ierr = MPI_Testsome(*incount, reqs, outcount, indices, FORT_STATUSES_IGNORE(statuses)); for(i=0;i<*incount;i++){ if(indices[i] && reqs[indices[i]]==MPI_REQUEST_NULL){ simgrid::smpi::Request::free_f(requests[indices[i]]); requests[indices[i]]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_probe_ (int* source, int* tag, int* comm, MPI_Status* status, int* ierr) { *ierr = MPI_Probe(*source, *tag, simgrid::smpi::Comm::f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_iprobe_ (int* source, int* tag, int* comm, int* flag, MPI_Status* status, int* ierr) { *ierr = MPI_Iprobe(*source, *tag, simgrid::smpi::Comm::f2c(*comm), flag, status); } } SimGrid-3.18/src/smpi/bindings/smpi_f77_comm.cpp0000644000175000017500000001721213217757317022035 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_info.hpp" extern "C" { // This should really use the C linkage to be usable from Fortran void mpi_comm_rank_(int* comm, int* rank, int* ierr) { *ierr = MPI_Comm_rank(simgrid::smpi::Comm::f2c(*comm), rank); } void mpi_comm_size_(int* comm, int* size, int* ierr) { *ierr = MPI_Comm_size(simgrid::smpi::Comm::f2c(*comm), size); } void mpi_comm_dup_(int* comm, int* newcomm, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_dup(simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_create_(int* comm, int* group, int* newcomm, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_create(simgrid::smpi::Comm::f2c(*comm),simgrid::smpi::Group::f2c(*group), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_free_(int* comm, int* ierr) { MPI_Comm tmp = simgrid::smpi::Comm::f2c(*comm); *ierr = MPI_Comm_free(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::Comm::free_f(*comm); } } void mpi_comm_split_(int* comm, int* color, int* key, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_split(simgrid::smpi::Comm::f2c(*comm), *color, *key, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = tmp->add_f(); } } void mpi_comm_group_(int* comm, int* group_out, int* ierr) { MPI_Group tmp; *ierr = MPI_Comm_group(simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *group_out = tmp->c2f(); } } void mpi_comm_create_group_ (int* comm, int* group, int i, int* comm_out, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_create_group(simgrid::smpi::Comm::f2c(*comm),simgrid::smpi::Group::f2c(*group), i, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = tmp->c2f(); } } void mpi_comm_get_attr_ (int* comm, int* comm_keyval, void *attribute_val, int *flag, int* ierr){ *ierr = MPI_Comm_get_attr (simgrid::smpi::Comm::f2c(*comm), *comm_keyval, attribute_val, flag); } void mpi_comm_set_attr_ (int* comm, int* comm_keyval, void *attribute_val, int* ierr){ *ierr = MPI_Comm_set_attr ( simgrid::smpi::Comm::f2c(*comm), *comm_keyval, attribute_val); } void mpi_comm_delete_attr_ (int* comm, int* comm_keyval, int* ierr){ *ierr = MPI_Comm_delete_attr (simgrid::smpi::Comm::f2c(*comm), *comm_keyval); } void mpi_comm_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr){ *ierr = MPI_Comm_create_keyval(reinterpret_cast(copy_fn), reinterpret_cast(delete_fn), keyval, extra_state) ; } void mpi_comm_free_keyval_ (int* keyval, int* ierr) { *ierr = MPI_Comm_free_keyval( keyval); } void mpi_comm_get_name_ (int* comm, char* name, int* len, int* ierr){ *ierr = MPI_Comm_get_name(simgrid::smpi::Comm::f2c(*comm), name, len); if(*len>0) name[*len]=' '; } void mpi_comm_compare_ (int* comm1, int* comm2, int *result, int* ierr){ *ierr = MPI_Comm_compare(simgrid::smpi::Comm::f2c(*comm1), simgrid::smpi::Comm::f2c(*comm2), result); } void mpi_comm_disconnect_ (int* comm, int* ierr){ MPI_Comm tmp = simgrid::smpi::Comm::f2c(*comm); *ierr = MPI_Comm_disconnect(&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::Comm::free_f(*comm); } } void mpi_comm_set_errhandler_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(simgrid::smpi::Comm::f2c(*comm), *static_cast(errhandler)); } void mpi_comm_get_errhandler_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(simgrid::smpi::Comm::f2c(*comm), static_cast(errhandler)); } void mpi_comm_test_inter_ (int* comm, int* flag, int* ierr) { *ierr = MPI_Comm_test_inter(simgrid::smpi::Comm::f2c(*comm), flag); } void mpi_comm_remote_group_ (int* comm, int* group, int* ierr) { MPI_Group tmp; *ierr = MPI_Comm_remote_group(simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *group = tmp->c2f(); } } void mpi_comm_remote_size_ (int* comm, int* size, int* ierr) { *ierr = MPI_Comm_remote_size(simgrid::smpi::Comm::f2c(*comm), size); } void mpi_comm_set_name_ (int* comm, char* name, int* ierr, int size){ char* tname = xbt_new(char, size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Comm_set_name (simgrid::smpi::Comm::f2c(*comm), tname); xbt_free(tname); } void mpi_comm_dup_with_info_ (int* comm, int* info, int* newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_dup_with_info(simgrid::smpi::Comm::f2c(*comm), simgrid::smpi::Info::f2c(*info),&tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_split_type_ (int* comm, int* split_type, int* key, int* info, int* newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_split_type(simgrid::smpi::Comm::f2c(*comm), *split_type, *key, simgrid::smpi::Info::f2c(*info), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_set_info_ (int* comm, int* info, int* ierr){ *ierr = MPI_Comm_set_info (simgrid::smpi::Comm::f2c(*comm), simgrid::smpi::Info::f2c(*info)); } void mpi_comm_get_info_ (int* comm, int* info, int* ierr){ MPI_Info tmp; *ierr = MPI_Comm_get_info (simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr==MPI_SUCCESS){ *info = tmp->c2f(); } } void mpi_comm_create_errhandler_ ( void *function, void *errhandler, int* ierr){ *ierr = MPI_Comm_create_errhandler( reinterpret_cast(function), static_cast(errhandler)); } void mpi_comm_call_errhandler_ (int* comm,int* errorcode, int* ierr){ *ierr = MPI_Comm_call_errhandler(simgrid::smpi::Comm::f2c(*comm), *errorcode); } void mpi_comm_connect_ ( char *port_name, int* info, int* root, int* comm, int*newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_connect( port_name, *reinterpret_cast(info), *root, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_join_ ( int* fd, int* intercomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_join( *fd, &tmp); if(*ierr == MPI_SUCCESS) { *intercomm = tmp->add_f(); } } void mpi_comm_accept_ ( char *port_name, int* info, int* root, int* comm, int*newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_accept( port_name, *reinterpret_cast(info), *root, simgrid::smpi::Comm::f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = tmp->add_f(); } } void mpi_comm_spawn_ ( char *command, char *argv, int* maxprocs, int* info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_spawn( command, nullptr, *maxprocs, *reinterpret_cast(info), *root, simgrid::smpi::Comm::f2c(*comm), &tmp, array_of_errcodes); if(*ierr == MPI_SUCCESS) { *intercomm = tmp->add_f(); } } void mpi_comm_spawn_multiple_ ( int* count, char *array_of_commands, char** array_of_argv, int* array_of_maxprocs, int* array_of_info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_spawn_multiple(* count, &array_of_commands, &array_of_argv, array_of_maxprocs, reinterpret_cast(array_of_info), *root, simgrid::smpi::Comm::f2c(*comm), &tmp, array_of_errcodes); if(*ierr == MPI_SUCCESS) { *intercomm = tmp->add_f(); } } void mpi_comm_get_parent_ ( int* parent, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_get_parent( &tmp); if(*ierr == MPI_SUCCESS) { *parent = tmp->c2f(); } } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_info.cpp0000644000175000017500000000442613217757317022402 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_info.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Info_create( MPI_Info *info){ if (info == nullptr) return MPI_ERR_ARG; *info = new simgrid::smpi::Info(); return MPI_SUCCESS; } int PMPI_Info_set( MPI_Info info, char *key, char *value){ if (info == nullptr || key == nullptr || value == nullptr) return MPI_ERR_ARG; info->set(key, value); return MPI_SUCCESS; } int PMPI_Info_free( MPI_Info *info){ if (info == nullptr || *info==nullptr) return MPI_ERR_ARG; simgrid::smpi::Info::unref(*info); *info=MPI_INFO_NULL; return MPI_SUCCESS; } int PMPI_Info_get(MPI_Info info,char *key,int valuelen, char *value, int *flag){ *flag=false; if (info == nullptr || key == nullptr || valuelen <0) return MPI_ERR_ARG; if (value == nullptr) return MPI_ERR_INFO_VALUE; return info->get(key, valuelen, value, flag); } int PMPI_Info_dup(MPI_Info info, MPI_Info *newinfo){ if (info == nullptr || newinfo==nullptr) return MPI_ERR_ARG; *newinfo = new simgrid::smpi::Info(info); return MPI_SUCCESS; } int PMPI_Info_delete(MPI_Info info, char *key){ if (info == nullptr || key==nullptr) return MPI_ERR_ARG; return info->remove(key); } int PMPI_Info_get_nkeys( MPI_Info info, int *nkeys){ if (info == nullptr || nkeys==nullptr) return MPI_ERR_ARG; return info->get_nkeys(nkeys); } int PMPI_Info_get_nthkey( MPI_Info info, int n, char *key){ if (info == nullptr || key==nullptr || n<0 || n> MPI_MAX_INFO_KEY) return MPI_ERR_ARG; return info->get_nthkey(n, key); } int PMPI_Info_get_valuelen( MPI_Info info, char *key, int *valuelen, int *flag){ *flag=false; if (info == nullptr || key == nullptr || valuelen==nullptr) return MPI_ERR_ARG; return info->get_valuelen(key, valuelen, flag); } MPI_Info PMPI_Info_f2c(MPI_Fint info){ return static_cast(simgrid::smpi::Info::f2c(info)); } MPI_Fint PMPI_Info_c2f(MPI_Info info){ return info->c2f(); } } SimGrid-3.18/src/smpi/bindings/smpi_f77_type.cpp0000644000175000017500000002533713217757317022072 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" extern "C" { // This should really use the C linkage to be usable from Fortran void mpi_type_extent_(int* datatype, MPI_Aint * extent, int* ierr){ *ierr= MPI_Type_extent(simgrid::smpi::Datatype::f2c(*datatype), extent); } void mpi_type_free_(int* datatype, int* ierr){ MPI_Datatype tmp= simgrid::smpi::Datatype::f2c(*datatype); *ierr= MPI_Type_free (&tmp); if(*ierr == MPI_SUCCESS) { simgrid::smpi::F2C::free_f(*datatype); } } void mpi_type_ub_(int* datatype, MPI_Aint * disp, int* ierr){ *ierr= MPI_Type_ub(simgrid::smpi::Datatype::f2c(*datatype), disp); } void mpi_type_lb_(int* datatype, MPI_Aint * extent, int* ierr){ *ierr= MPI_Type_extent(simgrid::smpi::Datatype::f2c(*datatype), extent); } void mpi_type_size_(int* datatype, int *size, int* ierr) { *ierr = MPI_Type_size(simgrid::smpi::Datatype::f2c(*datatype), size); } void mpi_type_dup_ (int* datatype, int* newdatatype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_dup(simgrid::smpi::Datatype::f2c(*datatype), &tmp); if(*ierr == MPI_SUCCESS) { *newdatatype = tmp->add_f(); } } void mpi_type_set_name_ (int* datatype, char * name, int* ierr, int size){ char* tname = xbt_new(char, size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Type_set_name(simgrid::smpi::Datatype::f2c(*datatype), tname); xbt_free(tname); } void mpi_type_get_name_ (int* datatype, char * name, int* len, int* ierr){ *ierr = MPI_Type_get_name(simgrid::smpi::Datatype::f2c(*datatype),name,len); if(*len>0) name[*len]=' '; } void mpi_type_get_attr_ (int* type, int* type_keyval, void *attribute_val, int* flag, int* ierr){ *ierr = MPI_Type_get_attr ( simgrid::smpi::Datatype::f2c(*type), *type_keyval, attribute_val,flag); } void mpi_type_set_attr_ (int* type, int* type_keyval, void *attribute_val, int* ierr){ *ierr = MPI_Type_set_attr ( simgrid::smpi::Datatype::f2c(*type), *type_keyval, attribute_val); } void mpi_type_delete_attr_ (int* type, int* type_keyval, int* ierr){ *ierr = MPI_Type_delete_attr ( simgrid::smpi::Datatype::f2c(*type), *type_keyval); } void mpi_type_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr){ *ierr = MPI_Type_create_keyval(reinterpret_cast(copy_fn), reinterpret_cast(delete_fn), keyval, extra_state) ; } void mpi_type_free_keyval_ (int* keyval, int* ierr) { *ierr = MPI_Type_free_keyval( keyval); } void mpi_type_get_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr){ *ierr = MPI_Type_get_extent(simgrid::smpi::Datatype::f2c(*datatype), lb, extent); } void mpi_type_get_true_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr){ *ierr = MPI_Type_get_true_extent(simgrid::smpi::Datatype::f2c(*datatype), lb, extent); } void mpi_type_commit_(int* datatype, int* ierr){ MPI_Datatype tmp= simgrid::smpi::Datatype::f2c(*datatype); *ierr= MPI_Type_commit(&tmp); } void mpi_type_contiguous_ (int* count, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_contiguous(*count, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_vector(*count, *blocklen, *stride, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_hvector (*count, *blocklen, *stride, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_hvector(*count, *blocklen, *stride, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_hindexed_ (int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_hindexed(*count, blocklens, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_hindexed_(int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_hindexed(*count, blocklens, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_hindexed_block_ (int* count, int* blocklength, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_create_hindexed_block(*count, *blocklength, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_indexed_ (int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_indexed(*count, blocklens, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_indexed_(int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_indexed(*count, blocklens, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_indexed_block_ (int* count, int* blocklength, int* indices, int* old_type, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_indexed_block(*count, *blocklength, indices, simgrid::smpi::Datatype::f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_struct_ (int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr) { MPI_Datatype tmp; int i=0; MPI_Datatype* types = static_cast(xbt_malloc(*count*sizeof(MPI_Datatype))); for(i=0; i< *count; i++){ types[i] = simgrid::smpi::Datatype::f2c(old_types[i]); } *ierr = MPI_Type_struct(*count, blocklens, indices, types, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } xbt_free(types); } void mpi_type_create_struct_(int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr){ MPI_Datatype tmp; int i=0; MPI_Datatype* types = static_cast(xbt_malloc(*count*sizeof(MPI_Datatype))); for(i=0; i< *count; i++){ types[i] = simgrid::smpi::Datatype::f2c(old_types[i]); } *ierr = MPI_Type_create_struct(*count, blocklens, indices, types, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } xbt_free(types); } void mpi_pack_ (void* inbuf, int* incount, int* type, void* outbuf, int* outcount, int* position, int* comm, int* ierr) { *ierr = MPI_Pack(inbuf, *incount, simgrid::smpi::Datatype::f2c(*type), outbuf, *outcount, position, simgrid::smpi::Comm::f2c(*comm)); } void mpi_unpack_ (void* inbuf, int* insize, int* position, void* outbuf, int* outcount, int* type, int* comm, int* ierr) { *ierr = MPI_Unpack(inbuf, *insize, position, outbuf, *outcount, simgrid::smpi::Datatype::f2c(*type), simgrid::smpi::Comm::f2c(*comm)); } void mpi_pack_external_size_ (char *datarep, int* incount, int* datatype, MPI_Aint *size, int* ierr){ *ierr = MPI_Pack_external_size(datarep, *incount, simgrid::smpi::Datatype::f2c(*datatype), size); } void mpi_pack_external_ (char *datarep, void *inbuf, int* incount, int* datatype, void *outbuf, MPI_Aint* outcount, MPI_Aint *position, int* ierr){ *ierr = MPI_Pack_external(datarep, inbuf, *incount, simgrid::smpi::Datatype::f2c(*datatype), outbuf, *outcount, position); } void mpi_unpack_external_ ( char *datarep, void *inbuf, MPI_Aint* insize, MPI_Aint *position, void *outbuf, int* outcount, int* datatype, int* ierr){ *ierr = MPI_Unpack_external( datarep, inbuf, *insize, position, outbuf, *outcount, simgrid::smpi::Datatype::f2c(*datatype)); } void mpi_type_get_envelope_ ( int* datatype, int *num_integers, int *num_addresses, int *num_datatypes, int *combiner, int* ierr){ *ierr = MPI_Type_get_envelope( simgrid::smpi::Datatype::f2c(*datatype), num_integers, num_addresses, num_datatypes, combiner); } void mpi_type_get_contents_ (int* datatype, int* max_integers, int* max_addresses, int* max_datatypes, int* array_of_integers, MPI_Aint* array_of_addresses, int* array_of_datatypes, int* ierr){ *ierr = MPI_Type_get_contents(simgrid::smpi::Datatype::f2c(*datatype), *max_integers, *max_addresses,*max_datatypes, array_of_integers, array_of_addresses, reinterpret_cast(array_of_datatypes)); } void mpi_type_create_darray_ (int* size, int* rank, int* ndims, int* array_of_gsizes, int* array_of_distribs, int* array_of_dargs, int* array_of_psizes, int* order, int* oldtype, int*newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_create_darray(*size, *rank, *ndims, array_of_gsizes, array_of_distribs, array_of_dargs, array_of_psizes, *order, simgrid::smpi::Datatype::f2c(*oldtype), &tmp) ; if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_resized_ (int* oldtype,MPI_Aint* lb, MPI_Aint* extent, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_resized(simgrid::smpi::Datatype::f2c(*oldtype),*lb, *extent, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_create_subarray_ (int* ndims,int *array_of_sizes, int *array_of_subsizes, int *array_of_starts, int* order, int* oldtype, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_subarray(*ndims,array_of_sizes, array_of_subsizes, array_of_starts, *order, simgrid::smpi::Datatype::f2c(*oldtype), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = tmp->add_f(); } } void mpi_type_match_size_ (int* typeclass,int* size,int* datatype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_match_size(*typeclass,*size,&tmp); if(*ierr == MPI_SUCCESS) { *datatype = tmp->c2f(); } } } SimGrid-3.18/src/smpi/bindings/smpi_pmpi_type.cpp0000644000175000017500000002506213217757317022427 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_datatype_derived.hpp" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_pmpi); /* PMPI User level calls */ extern "C" { // Obviously, the C MPI interface should use the C linkage int PMPI_Type_free(MPI_Datatype * datatype) { /* Free a predefined datatype is an error according to the standard, and should be checked for */ if (*datatype == MPI_DATATYPE_NULL) { return MPI_ERR_ARG; } else { simgrid::smpi::Datatype::unref(*datatype); return MPI_SUCCESS; } } int PMPI_Type_size(MPI_Datatype datatype, int *size) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (size == nullptr) { return MPI_ERR_ARG; } else { *size = static_cast(datatype->size()); return MPI_SUCCESS; } } int PMPI_Type_size_x(MPI_Datatype datatype, MPI_Count *size) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (size == nullptr) { return MPI_ERR_ARG; } else { *size = static_cast(datatype->size()); return MPI_SUCCESS; } } int PMPI_Type_get_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (lb == nullptr || extent == nullptr) { return MPI_ERR_ARG; } else { return datatype->extent(lb, extent); } } int PMPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent) { return PMPI_Type_get_extent(datatype, lb, extent); } int PMPI_Type_extent(MPI_Datatype datatype, MPI_Aint * extent) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (extent == nullptr) { return MPI_ERR_ARG; } else { *extent = datatype->get_extent(); return MPI_SUCCESS; } } int PMPI_Type_lb(MPI_Datatype datatype, MPI_Aint * disp) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (disp == nullptr) { return MPI_ERR_ARG; } else { *disp = datatype->lb(); return MPI_SUCCESS; } } int PMPI_Type_ub(MPI_Datatype datatype, MPI_Aint * disp) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (disp == nullptr) { return MPI_ERR_ARG; } else { *disp = datatype->ub(); return MPI_SUCCESS; } } int PMPI_Type_dup(MPI_Datatype datatype, MPI_Datatype *newtype){ int retval = MPI_SUCCESS; if (datatype == MPI_DATATYPE_NULL) { retval=MPI_ERR_TYPE; } else { *newtype = new simgrid::smpi::Datatype(datatype, &retval); //error when duplicating, free the new datatype if(retval!=MPI_SUCCESS){ simgrid::smpi::Datatype::unref(*newtype); *newtype = MPI_DATATYPE_NULL; } } return retval; } int PMPI_Type_contiguous(int count, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_contiguous(count, old_type, 0, new_type); } } int PMPI_Type_commit(MPI_Datatype* datatype) { if (datatype == nullptr || *datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else { (*datatype)->commit(); return MPI_SUCCESS; } } int PMPI_Type_vector(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0 || blocklen<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_vector(count, blocklen, stride, old_type, new_type); } } int PMPI_Type_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0 || blocklen<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_hvector(count, blocklen, stride, old_type, new_type); } } int PMPI_Type_create_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { return MPI_Type_hvector(count, blocklen, stride, old_type, new_type); } int PMPI_Type_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_indexed(count, blocklens, indices, old_type, new_type); } } int PMPI_Type_create_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_indexed(count, blocklens, indices, old_type, new_type); } } int PMPI_Type_create_indexed_block(int count, int blocklength, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { int* blocklens=static_cast(xbt_malloc(blocklength*count*sizeof(int))); for (int i = 0; i < count; i++) blocklens[i]=blocklength; int retval = simgrid::smpi::Datatype::create_indexed(count, blocklens, indices, old_type, new_type); xbt_free(blocklens); return retval; } } int PMPI_Type_hindexed(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_hindexed(count, blocklens, indices, old_type, new_type); } } int PMPI_Type_create_hindexed(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { return PMPI_Type_hindexed(count, blocklens,indices,old_type,new_type); } int PMPI_Type_create_hindexed_block(int count, int blocklength, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { if (old_type == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (count<0){ return MPI_ERR_COUNT; } else { int* blocklens=(int*)xbt_malloc(blocklength*count*sizeof(int)); for (int i = 0; i < count; i++) blocklens[i] = blocklength; int retval = simgrid::smpi::Datatype::create_hindexed(count, blocklens, indices, old_type, new_type); xbt_free(blocklens); return retval; } } int PMPI_Type_struct(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type) { if (count<0){ return MPI_ERR_COUNT; } else { return simgrid::smpi::Datatype::create_struct(count, blocklens, indices, old_types, new_type); } } int PMPI_Type_create_struct(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type) { return PMPI_Type_struct(count, blocklens, indices, old_types, new_type); } int PMPI_Type_create_resized(MPI_Datatype oldtype,MPI_Aint lb, MPI_Aint extent, MPI_Datatype *newtype){ if (oldtype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } int blocks[3] = {1, 1, 1}; MPI_Aint disps[3] = {lb, 0, lb + extent}; MPI_Datatype types[3] = {MPI_LB, oldtype, MPI_UB}; *newtype = new simgrid::smpi::Type_Struct(oldtype->size(), lb, lb + extent, DT_FLAG_DERIVED, 3, blocks, disps, types); (*newtype)->addflag(~DT_FLAG_COMMITED); return MPI_SUCCESS; } int PMPI_Type_set_name(MPI_Datatype datatype, char * name) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (name == nullptr) { return MPI_ERR_ARG; } else { datatype->set_name(name); return MPI_SUCCESS; } } int PMPI_Type_get_name(MPI_Datatype datatype, char * name, int* len) { if (datatype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } else if (name == nullptr) { return MPI_ERR_ARG; } else { datatype->get_name(name, len); return MPI_SUCCESS; } } MPI_Datatype PMPI_Type_f2c(MPI_Fint datatype){ return static_cast(simgrid::smpi::F2C::f2c(datatype)); } MPI_Fint PMPI_Type_c2f(MPI_Datatype datatype){ return datatype->c2f(); } int PMPI_Type_get_attr (MPI_Datatype type, int type_keyval, void *attribute_val, int* flag) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return type->attr_get(type_keyval, attribute_val, flag); } int PMPI_Type_set_attr (MPI_Datatype type, int type_keyval, void *attribute_val) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return type->attr_put(type_keyval, attribute_val); } int PMPI_Type_delete_attr (MPI_Datatype type, int type_keyval) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return type->attr_delete(type_keyval); } int PMPI_Type_create_keyval(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval, void* extra_state) { smpi_copy_fn _copy_fn={nullptr,copy_fn,nullptr}; smpi_delete_fn _delete_fn={nullptr,delete_fn,nullptr}; return simgrid::smpi::Keyval::keyval_create(_copy_fn, _delete_fn, keyval, extra_state); } int PMPI_Type_free_keyval(int* keyval) { return simgrid::smpi::Keyval::keyval_free(keyval); } int PMPI_Unpack(void* inbuf, int incount, int* position, void* outbuf, int outcount, MPI_Datatype type, MPI_Comm comm) { if(incount<0 || outcount < 0 || inbuf==nullptr || outbuf==nullptr) return MPI_ERR_ARG; if (not type->is_valid()) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; return type->unpack(inbuf, incount, position, outbuf,outcount, comm); } int PMPI_Pack(void* inbuf, int incount, MPI_Datatype type, void* outbuf, int outcount, int* position, MPI_Comm comm) { if(incount<0 || outcount < 0|| inbuf==nullptr || outbuf==nullptr) return MPI_ERR_ARG; if (not type->is_valid()) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; return type->pack(inbuf == MPI_BOTTOM ? nullptr : inbuf, incount, outbuf, outcount, position, comm); } int PMPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm, int* size) { if(incount<0) return MPI_ERR_ARG; if (not datatype->is_valid()) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; *size=incount*datatype->size(); return MPI_SUCCESS; } } SimGrid-3.18/src/smpi/internals/0000755000175000017500000000000013217757320017054 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/internals/smpi_memory.cpp0000644000175000017500000002122513217757320022122 0ustar mquinsonmquinson/* Copyright (c) 2015-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include #ifndef WIN32 #include #include #include "src/internal_config.h" #include "src/xbt/memory_map.hpp" #include "private.hpp" #include "smpi_process.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_memory, smpi, "Memory layout support for SMPI"); int smpi_loaded_page = -1; char* smpi_data_exe_start = nullptr; int smpi_data_exe_size = 0; int smpi_privatize_global_variables; static void* smpi_data_exe_copy; // We keep a copy of all the privatization regions: We can then delete everything easily by iterating over this // collection and nothing can be leaked. We could also iterate over all actors but we would have to be diligent when two // actors use the same privatization region (so, smart pointers would have to be used etc.) // Use a std::deque so that pointers remain valid after push_back(). static std::deque smpi_privatization_regions; static const int PROT_RWX = (PROT_READ | PROT_WRITE | PROT_EXEC); static const int PROT_RW = (PROT_READ | PROT_WRITE ); XBT_ATTRIB_UNUSED static const int PROT_RX = (PROT_READ | PROT_EXEC ); void smpi_get_executable_global_size() { char buffer[PATH_MAX]; char* full_name = realpath(xbt_binary_name, buffer); if (full_name == nullptr) xbt_die("Could not resolve binary file name"); std::vector map = simgrid::xbt::get_memory_map(getpid()); for (auto i = map.begin(); i != map.end() ; ++i) { // TODO, In practice, this implementation would not detect a completely // anonymous data segment. This does not happen in practice, however. // File backed RW entry: if (i->pathname == full_name && (i->prot & PROT_RWX) == PROT_RW) { smpi_data_exe_start = (char*)i->start_addr; smpi_data_exe_size = i->end_addr - i->start_addr; ++i; /* Here we are making the assumption that a suitable empty region following the rw- area is the end of the data segment. It would be better to check with the size of the data segment. */ if (i != map.end() && i->pathname.empty() && (i->prot & PROT_RWX) == PROT_RW && (char*)i->start_addr == smpi_data_exe_start + smpi_data_exe_size) { smpi_data_exe_size = (char*)i->end_addr - smpi_data_exe_start; } return; } } xbt_die("Did not find my data segment."); } #endif #if HAVE_SANITIZE_ADDRESS #include static void* asan_safe_memcpy(void* dest, void* src, size_t n) { char* psrc = static_cast(src); char* pdest = static_cast(dest); for (size_t i = 0; i < n;) { while (i < n && __asan_address_is_poisoned(psrc + i)) ++i; if (i < n) { char* p = static_cast(__asan_region_is_poisoned(psrc + i, n - i)); size_t j = p ? (p - psrc) : n; memcpy(pdest + i, psrc + i, j - i); i = j; } } return dest; } #else #define asan_safe_memcpy(dest, src, n) memcpy(dest, src, n) #endif /** Map a given SMPI privatization segment (make a SMPI process active) */ void smpi_switch_data_segment(int dest) { if (smpi_loaded_page == dest)//no need to switch, we've already loaded the one we want return; // So the job: smpi_really_switch_data_segment(dest); } /** Map a given SMPI privatization segment (make a SMPI process active) even if SMPI thinks it is already active * * When doing a state restoration, the state of the restored variables might not be consistent with the state of the * virtual memory. In this case, we to change the data segment. */ void smpi_really_switch_data_segment(int dest) { if (smpi_data_exe_size == 0) // no need to switch return; #if HAVE_PRIVATIZATION // FIXME, cross-process support (mmap across process when necessary) simgrid::smpi::Process* process = smpi_process_remote(dest); int current = process->privatized_region()->file_descriptor; XBT_DEBUG("Switching data frame to the one of process %d", dest); void* tmp = mmap(TOPAGE(smpi_data_exe_start), smpi_data_exe_size, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_SHARED, current, 0); if (tmp != TOPAGE(smpi_data_exe_start)) xbt_die("Couldn't map the new region (errno %d): %s", errno, strerror(errno)); smpi_loaded_page = dest; #endif } int smpi_is_privatization_file(char* file) { const std::string buffer_path("/dev/shm/my-buffer-"); return buffer_path.compare(0, std::string::npos, file, buffer_path.length()) == 0; } // TODO: cheinrich: The behavior changed; this now only makes a backup of the // data segment. I think the function should be renamed. void smpi_backup_global_memory_segment() { #if HAVE_PRIVATIZATION smpi_get_executable_global_size(); XBT_DEBUG("bss+data segment found : size %d starting at %p", smpi_data_exe_size, smpi_data_exe_start); if (smpi_data_exe_size == 0) { // no need to do anything as global variables don't exist smpi_privatize_global_variables=false; return; } smpi_data_exe_copy = ::operator new(smpi_data_exe_size); // Make a copy of the data segment. This clean copy is retained over the whole runtime // of the simulation and can be used to initialize a dynamically added, new process. asan_safe_memcpy(smpi_data_exe_copy, TOPAGE(smpi_data_exe_start), smpi_data_exe_size); #else /* ! HAVE_PRIVATIZATION */ smpi_privatize_global_variables = false; xbt_die("You are trying to use privatization on a system that does not support it. Don't."); return; #endif } // Initializes the memory mapping for a single process and returns the privatization region smpi_privatization_region_t smpi_init_global_memory_segment_process() { int file_descriptor; void* address = nullptr; char path[24]; int status; do { snprintf(path, sizeof(path), "/smpi-buffer-%06x", rand() % 0xffffffU); file_descriptor = shm_open(path, O_RDWR | O_CREAT | O_EXCL, S_IRUSR | S_IWUSR); } while (file_descriptor == -1 && errno == EEXIST); if (file_descriptor < 0) { if (errno == EMFILE) { xbt_die("Impossible to create temporary file for memory mapping: %s\n\ The open() system call failed with the EMFILE error code (too many files). \n\n\ This means that you reached the system limits concerning the amount of files per process. \ This is not a surprise if you are trying to virtualize many processes on top of SMPI. \ Don't panic -- you should simply increase your system limits and try again. \n\n\ First, check what your limits are:\n\ cat /proc/sys/fs/file-max # Gives you the system-wide limit\n\ ulimit -Hn # Gives you the per process hard limit\n\ ulimit -Sn # Gives you the per process soft limit\n\ cat /proc/self/limits # Displays any per-process limitation (including the one given above)\n\n\ If one of these values is less than the amount of MPI processes that you try to run, then you got the explanation of this error. \ Ask the Internet about tutorials on how to increase the files limit such as: https://rtcamp.com/tutorials/linux/increase-open-files-limit/", strerror(errno)); } xbt_die("Impossible to create temporary file for memory mapping: %s", strerror(errno)); } status = ftruncate(file_descriptor, smpi_data_exe_size); if (status) xbt_die("Impossible to set the size of the temporary file for memory mapping"); /* Ask for a free region */ address = mmap(nullptr, smpi_data_exe_size, PROT_READ | PROT_WRITE, MAP_SHARED, file_descriptor, 0); if (address == MAP_FAILED) xbt_die("Couldn't find a free region for memory mapping"); status = shm_unlink(path); if (status) xbt_die("Impossible to unlink temporary file for memory mapping"); // initialize the values asan_safe_memcpy(address, smpi_data_exe_copy, smpi_data_exe_size); // store the address of the mapping for further switches smpi_privatization_regions.emplace_back(s_smpi_privatization_region_t{address, file_descriptor}); return &smpi_privatization_regions.back(); } void smpi_destroy_global_memory_segments(){ if (smpi_data_exe_size == 0) // no need to switch return; #if HAVE_PRIVATIZATION for (auto const& region : smpi_privatization_regions) { if (munmap(region.address, smpi_data_exe_size) < 0) XBT_WARN("Unmapping of fd %d failed: %s", region.file_descriptor, strerror(errno)); close(region.file_descriptor); } smpi_privatization_regions.clear(); ::operator delete(smpi_data_exe_copy); #endif } SimGrid-3.18/src/smpi/internals/smpi_static_variables.cpp0000644000175000017500000000171313217757320024131 0ustar mquinsonmquinson/* Copyright (c) 2011-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include struct s_smpi_static_t { void *ptr; void_f_pvoid_t free_fn; }; /** * \brief Holds a reference to all static variables that were registered * via smpi_register_static(). This helps to free them when * SMPI shuts down. */ static std::stack registered_static_variables_stack; void smpi_register_static(void* arg, void_f_pvoid_t free_fn) { s_smpi_static_t elm { arg, free_fn }; registered_static_variables_stack.push(elm); } void smpi_free_static() { while (not registered_static_variables_stack.empty()) { s_smpi_static_t elm = registered_static_variables_stack.top(); elm.free_fn(elm.ptr); registered_static_variables_stack.pop(); } } SimGrid-3.18/src/smpi/internals/smpi_global.cpp0000644000175000017500000006341113217757320022055 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "SmpiHost.hpp" #include "mc/mc.h" #include "private.hpp" #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_group.hpp" #include "smpi_info.hpp" #include "smpi_process.hpp" #include "src/msg/msg_private.hpp" #include "src/simix/smx_private.hpp" #include "src/surf/surf_interface.hpp" #include "xbt/config.hpp" #include /* DBL_MAX */ #include #include #include #include #if HAVE_SENDFILE #include #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_kernel, smpi, "Logging specific to SMPI (kernel)"); #include #include /* trim_right / trim_left */ #ifndef RTLD_DEEPBIND /* RTLD_DEEPBIND is a bad idea of GNU ld that obviously does not exist on other platforms * See https://www.akkadia.org/drepper/dsohowto.pdf * and https://lists.freebsd.org/pipermail/freebsd-current/2016-March/060284.html */ #define RTLD_DEEPBIND 0 #endif #if HAVE_PAPI #include "papi.h" const char* papi_default_config_name = "default"; struct papi_process_data { papi_counter_t counter_data; int event_set; }; #endif std::unordered_map location2speedup; static simgrid::smpi::Process** process_data = nullptr; int process_count = 0; int smpi_universe_size = 0; int* index_to_process_data = nullptr; extern double smpi_total_benched_time; xbt_os_timer_t global_timer; /** * Setting MPI_COMM_WORLD to MPI_COMM_UNINITIALIZED (it's a variable) * is important because the implementation of MPI_Comm checks * "this == MPI_COMM_UNINITIALIZED"? If yes, it uses smpi_process()->comm_world() * instead of "this". * This is basically how we only have one global variable but all processes have * different communicators (basically, the one their SMPI instance uses). * * See smpi_comm.cpp and the functions therein for details. */ MPI_Comm MPI_COMM_WORLD = MPI_COMM_UNINITIALIZED; MPI_Errhandler *MPI_ERRORS_RETURN = nullptr; MPI_Errhandler *MPI_ERRORS_ARE_FATAL = nullptr; MPI_Errhandler *MPI_ERRHANDLER_NULL = nullptr; // No instance gets manually created; check also the smpirun.in script as // this default name is used there as well (when the tag is generated). static const char* smpi_default_instance_name = "smpirun"; static simgrid::config::Flag smpi_wtime_sleep( "smpi/wtime", "Minimum time to inject inside a call to MPI_Wtime", 0.0); static simgrid::config::Flag smpi_init_sleep( "smpi/init", "Time to inject inside a call to MPI_Init", 0.0); void (*smpi_comm_copy_data_callback) (smx_activity_t, void*, size_t) = &smpi_comm_copy_buffer_callback; int smpi_process_count() { return process_count; } simgrid::smpi::Process* smpi_process() { smx_actor_t me = SIMIX_process_self(); if (me == nullptr) // This happens sometimes (eg, when linking against NS3 because it pulls openMPI...) return nullptr; simgrid::msg::ActorExt* msgExt = static_cast(me->userdata); return static_cast(msgExt->data); } simgrid::smpi::Process* smpi_process_remote(int index) { return process_data[index_to_process_data[index]]; } MPI_Comm smpi_process_comm_self(){ return smpi_process()->comm_self(); } void smpi_process_init(int *argc, char ***argv){ simgrid::smpi::Process::init(argc, argv); } int smpi_process_index(){ return smpi_process()->index(); } void * smpi_process_get_user_data(){ return smpi_process()->get_user_data(); } void smpi_process_set_user_data(void *data){ return smpi_process()->set_user_data(data); } int smpi_global_size() { char *value = getenv("SMPI_GLOBAL_SIZE"); xbt_assert(value,"Please set env var SMPI_GLOBAL_SIZE to the expected number of processes."); return xbt_str_parse_int(value, "SMPI_GLOBAL_SIZE contains a non-numerical value: %s"); } void smpi_comm_set_copy_data_callback(void (*callback) (smx_activity_t, void*, size_t)) { smpi_comm_copy_data_callback = callback; } static void print(std::vector> vec) { std::fprintf(stderr, "{"); for (auto const& elt : vec) { std::fprintf(stderr, "(0x%zx, 0x%zx),", elt.first, elt.second); } std::fprintf(stderr, "}\n"); } static void memcpy_private(void* dest, const void* src, std::vector>& private_blocks) { for (auto const& block : private_blocks) memcpy((uint8_t*)dest+block.first, (uint8_t*)src+block.first, block.second-block.first); } static void check_blocks(std::vector> &private_blocks, size_t buff_size) { for (auto const& block : private_blocks) xbt_assert(block.first <= block.second && block.second <= buff_size, "Oops, bug in shared malloc."); } void smpi_comm_copy_buffer_callback(smx_activity_t synchro, void *buff, size_t buff_size) { simgrid::kernel::activity::CommImplPtr comm = boost::dynamic_pointer_cast(synchro); int src_shared = 0; int dst_shared = 0; size_t src_offset = 0; size_t dst_offset = 0; std::vector> src_private_blocks; std::vector> dst_private_blocks; XBT_DEBUG("Copy the data over"); if((src_shared=smpi_is_shared(buff, src_private_blocks, &src_offset))) { XBT_DEBUG("Sender %p is shared. Let's ignore it.", buff); src_private_blocks = shift_and_frame_private_blocks(src_private_blocks, src_offset, buff_size); } else { src_private_blocks.clear(); src_private_blocks.push_back(std::make_pair(0, buff_size)); } if((dst_shared=smpi_is_shared((char*)comm->dst_buff, dst_private_blocks, &dst_offset))) { XBT_DEBUG("Receiver %p is shared. Let's ignore it.", (char*)comm->dst_buff); dst_private_blocks = shift_and_frame_private_blocks(dst_private_blocks, dst_offset, buff_size); } else { dst_private_blocks.clear(); dst_private_blocks.push_back(std::make_pair(0, buff_size)); } check_blocks(src_private_blocks, buff_size); check_blocks(dst_private_blocks, buff_size); auto private_blocks = merge_private_blocks(src_private_blocks, dst_private_blocks); check_blocks(private_blocks, buff_size); void* tmpbuff=buff; if ((smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) && (static_cast(buff) >= smpi_data_exe_start) && (static_cast(buff) < smpi_data_exe_start + smpi_data_exe_size)) { XBT_DEBUG("Privatization : We are copying from a zone inside global memory... Saving data to temp buffer !"); smpi_switch_data_segment( static_cast((static_cast(comm->src_proc->userdata)->data)) ->index()); tmpbuff = static_cast(xbt_malloc(buff_size)); memcpy_private(tmpbuff, buff, private_blocks); } if ((smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) && ((char*)comm->dst_buff >= smpi_data_exe_start) && ((char*)comm->dst_buff < smpi_data_exe_start + smpi_data_exe_size)) { XBT_DEBUG("Privatization : We are copying to a zone inside global memory - Switch data segment"); smpi_switch_data_segment( static_cast((static_cast(comm->dst_proc->userdata)->data)) ->index()); } XBT_DEBUG("Copying %zu bytes from %p to %p", buff_size, tmpbuff,comm->dst_buff); memcpy_private(comm->dst_buff, tmpbuff, private_blocks); if (comm->detached) { // if this is a detached send, the source buffer was duplicated by SMPI // sender to make the original buffer available to the application ASAP xbt_free(buff); //It seems that the request is used after the call there this should be free somewhere else but where??? //xbt_free(comm->comm.src_data);// inside SMPI the request is kept inside the user data and should be free comm->src_buff = nullptr; } if (tmpbuff != buff) xbt_free(tmpbuff); } void smpi_comm_null_copy_buffer_callback(smx_activity_t comm, void *buff, size_t buff_size) { /* nothing done in this version */ } static void smpi_check_options(){ //check correctness of MPI parameters xbt_assert(xbt_cfg_get_int("smpi/async-small-thresh") <= xbt_cfg_get_int("smpi/send-is-detached-thresh")); if (xbt_cfg_is_default_value("smpi/host-speed")) { XBT_INFO("You did not set the power of the host running the simulation. " "The timings will certainly not be accurate. " "Use the option \"--cfg=smpi/host-speed:\" to set its value." "Check http://simgrid.org/simgrid/latest/doc/options.html#options_smpi_bench for more information."); } xbt_assert(xbt_cfg_get_double("smpi/cpu-threshold") >=0, "The 'smpi/cpu-threshold' option cannot have negative values [anymore]. If you want to discard " "the simulation of any computation, please use 'smpi/simulate-computation:no' instead."); } int smpi_enabled() { return process_data != nullptr; } void smpi_global_init() { if (not MC_is_active()) { global_timer = xbt_os_timer_new(); xbt_os_walltimer_start(global_timer); } std::string filename = xbt_cfg_get_string("smpi/comp-adjustment-file"); if (not filename.empty()) { std::ifstream fstream(filename); if (not fstream.is_open()) { xbt_die("Could not open file %s. Does it exist?", filename.c_str()); } std::string line; typedef boost::tokenizer< boost::escaped_list_separator> Tokenizer; std::getline(fstream, line); // Skip the header line while (std::getline(fstream, line)) { Tokenizer tok(line); Tokenizer::iterator it = tok.begin(); Tokenizer::iterator end = std::next(tok.begin()); std::string location = *it; boost::trim(location); location2speedup.insert(std::pair(location, std::stod(*end))); } } #if HAVE_PAPI // This map holds for each computation unit (such as "default" or "process1" etc.) // the configuration as given by the user (counter data as a pair of (counter_name, counter_counter)) // and the (computed) event_set. std::map units2papi_setup; if (not xbt_cfg_get_string("smpi/papi-events").empty()) { if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) XBT_ERROR("Could not initialize PAPI library; is it correctly installed and linked?" " Expected version is %i", PAPI_VER_CURRENT); typedef boost::tokenizer> Tokenizer; boost::char_separator separator_units(";"); std::string str = xbt_cfg_get_string("smpi/papi-events"); Tokenizer tokens(str, separator_units); // Iterate over all the computational units. This could be processes, hosts, threads, ranks... You name it. // I'm not exactly sure what we will support eventually, so I'll leave it at the general term "units". for (auto const& unit_it : tokens) { boost::char_separator separator_events(":"); Tokenizer event_tokens(unit_it, separator_events); int event_set = PAPI_NULL; if (PAPI_create_eventset(&event_set) != PAPI_OK) { // TODO: Should this let the whole simulation die? XBT_CRITICAL("Could not create PAPI event set during init."); } // NOTE: We cannot use a map here, as we must obey the order of the counters // This is important for PAPI: We need to map the values of counters back // to the event_names (so, when PAPI_read() has finished)! papi_counter_t counters2values; // Iterate over all counters that were specified for this specific // unit. // Note that we need to remove the name of the unit // (that could also be the "default" value), which always comes first. // Hence, we start at ++(events.begin())! for (Tokenizer::iterator events_it = ++(event_tokens.begin()); events_it != event_tokens.end(); ++events_it) { int event_code = PAPI_NULL; char* event_name = const_cast((*events_it).c_str()); if (PAPI_event_name_to_code(event_name, &event_code) == PAPI_OK) { if (PAPI_add_event(event_set, event_code) != PAPI_OK) { XBT_ERROR("Could not add PAPI event '%s'. Skipping.", event_name); continue; } else { XBT_DEBUG("Successfully added PAPI event '%s' to the event set.", event_name); } } else { XBT_CRITICAL("Could not find PAPI event '%s'. Skipping.", event_name); continue; } counters2values.push_back( // We cannot just pass *events_it, as this is of type const basic_string std::make_pair(std::string(*events_it), 0)); } std::string unit_name = *(event_tokens.begin()); papi_process_data config = {.counter_data = std::move(counters2values), .event_set = event_set}; units2papi_setup.insert(std::make_pair(unit_name, std::move(config))); } } #endif if (index_to_process_data == nullptr) { index_to_process_data = new int[SIMIX_process_count()]; } bool smpirun = 0; if (process_count == 0) { // The program has been dispatched but no other // SMPI instances have been registered. We're using smpirun. smpirun = true; SMPI_app_instance_register(smpi_default_instance_name, nullptr, SIMIX_process_count()); // This call has a side effect on process_count... MPI_COMM_WORLD = *smpi_deployment_comm_world(smpi_default_instance_name); } smpi_universe_size = process_count; process_data = new simgrid::smpi::Process*[process_count]; for (int i = 0; i < process_count; i++) { if (smpirun) { process_data[i] = new simgrid::smpi::Process(i, smpi_deployment_finalization_barrier(smpi_default_instance_name)); smpi_deployment_register_process(smpi_default_instance_name, i, i); } else { // TODO We can pass a nullptr here because Process::set_data() assigns the // barrier from the instance anyway. This is ugly and should be changed process_data[i] = new simgrid::smpi::Process(i, nullptr); } } } void smpi_global_destroy() { smpi_bench_destroy(); smpi_shared_destroy(); smpi_deployment_cleanup_instances(); int count = smpi_process_count(); for (int i = 0; i < count; i++) { if(process_data[i]->comm_self()!=MPI_COMM_NULL){ simgrid::smpi::Comm::destroy(process_data[i]->comm_self()); } if(process_data[i]->comm_intra()!=MPI_COMM_NULL){ simgrid::smpi::Comm::destroy(process_data[i]->comm_intra()); } xbt_os_timer_free(process_data[i]->timer()); xbt_mutex_destroy(process_data[i]->mailboxes_mutex()); delete process_data[i]; } delete[] process_data; process_data = nullptr; if (simgrid::smpi::Colls::smpi_coll_cleanup_callback != nullptr) simgrid::smpi::Colls::smpi_coll_cleanup_callback(); MPI_COMM_WORLD = MPI_COMM_NULL; if (not MC_is_active()) { xbt_os_timer_free(global_timer); } delete[] index_to_process_data; if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) smpi_destroy_global_memory_segments(); smpi_free_static(); } extern "C" { static void smpi_init_logs(){ /* Connect log categories. See xbt/log.c */ XBT_LOG_CONNECT(smpi); /* Keep this line as soon as possible in this function: xbt_log_appender_file.c depends on it DO NOT connect this in XBT or so, or it will be useless to xbt_log_appender_file.c */ XBT_LOG_CONNECT(instr_smpi); XBT_LOG_CONNECT(smpi_bench); XBT_LOG_CONNECT(smpi_coll); XBT_LOG_CONNECT(smpi_colls); XBT_LOG_CONNECT(smpi_comm); XBT_LOG_CONNECT(smpi_datatype); XBT_LOG_CONNECT(smpi_dvfs); XBT_LOG_CONNECT(smpi_group); XBT_LOG_CONNECT(smpi_host); XBT_LOG_CONNECT(smpi_kernel); XBT_LOG_CONNECT(smpi_mpi); XBT_LOG_CONNECT(smpi_memory); XBT_LOG_CONNECT(smpi_op); XBT_LOG_CONNECT(smpi_pmpi); XBT_LOG_CONNECT(smpi_process); XBT_LOG_CONNECT(smpi_request); XBT_LOG_CONNECT(smpi_replay); XBT_LOG_CONNECT(smpi_rma); XBT_LOG_CONNECT(smpi_shared); XBT_LOG_CONNECT(smpi_utils); } } static void smpi_init_options(){ // return if already called if (smpi_cpu_threshold > -1) return; simgrid::smpi::Colls::set_collectives(); simgrid::smpi::Colls::smpi_coll_cleanup_callback = nullptr; smpi_cpu_threshold = xbt_cfg_get_double("smpi/cpu-threshold"); smpi_host_speed = xbt_cfg_get_double("smpi/host-speed"); std::string smpi_privatize_option = xbt_cfg_get_string("smpi/privatization"); if (smpi_privatize_option == "no" || smpi_privatize_option == "0") smpi_privatize_global_variables = SMPI_PRIVATIZE_NONE; else if (smpi_privatize_option == "yes" || smpi_privatize_option == "1") smpi_privatize_global_variables = SMPI_PRIVATIZE_DEFAULT; else if (smpi_privatize_option == "mmap") smpi_privatize_global_variables = SMPI_PRIVATIZE_MMAP; else if (smpi_privatize_option == "dlopen") smpi_privatize_global_variables = SMPI_PRIVATIZE_DLOPEN; else xbt_die("Invalid value for smpi/privatization: '%s'", smpi_privatize_option.c_str()); #if defined(__FreeBSD__) if (smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) { XBT_INFO("Mixing mmap privatization is broken on FreeBSD, switching to dlopen privatization instead."); smpi_privatize_global_variables = SMPI_PRIVATIZE_DLOPEN; } #endif if (smpi_cpu_threshold < 0) smpi_cpu_threshold = DBL_MAX; std::string val = xbt_cfg_get_string("smpi/shared-malloc"); if ((val == "yes") || (val == "1") || (val == "on") || (val == "global")) { smpi_cfg_shared_malloc = shmalloc_global; } else if (val == "local") { smpi_cfg_shared_malloc = shmalloc_local; } else if ((val == "no") || (val == "0") || (val == "off")) { smpi_cfg_shared_malloc = shmalloc_none; } else { xbt_die("Invalid value '%s' for option smpi/shared-malloc. Possible values: 'on' or 'global', 'local', 'off'", val.c_str()); } } typedef std::function smpi_entry_point_type; typedef int (* smpi_c_entry_point_type)(int argc, char **argv); typedef void (*smpi_fortran_entry_point_type)(); static int smpi_run_entry_point(smpi_entry_point_type entry_point, std::vector args) { char noarg[] = {'\0'}; const int argc = args.size(); std::unique_ptr argv(new char*[argc + 1]); for (int i = 0; i != argc; ++i) argv[i] = args[i].empty() ? noarg : &args[i].front(); argv[argc] = nullptr; int res = entry_point(argc, argv.get()); if (res != 0){ XBT_WARN("SMPI process did not return 0. Return value : %d", res); smpi_process()->set_return_value(res); } return 0; } // TODO, remove the number of functions involved here static smpi_entry_point_type smpi_resolve_function(void* handle) { smpi_fortran_entry_point_type entry_point_fortran = (smpi_fortran_entry_point_type)dlsym(handle, "user_main_"); if (entry_point_fortran != nullptr) { return [entry_point_fortran](int argc, char** argv) { smpi_process_init(&argc, &argv); entry_point_fortran(); return 0; }; } smpi_c_entry_point_type entry_point = (smpi_c_entry_point_type)dlsym(handle, "main"); if (entry_point != nullptr) { return entry_point; } return smpi_entry_point_type(); } int smpi_main(const char* executable, int argc, char *argv[]) { srand(SMPI_RAND_SEED); if (getenv("SMPI_PRETEND_CC") != nullptr) { /* Hack to ensure that smpicc can pretend to be a simple compiler. Particularly handy to pass it to the * configuration tools */ return 0; } TRACE_global_init(); SIMIX_global_init(&argc, argv); MSG_init(&argc,argv); SMPI_switch_data_segment = &smpi_switch_data_segment; simgrid::s4u::Host::onCreation.connect([](simgrid::s4u::Host& host) { host.extension_set(new simgrid::smpi::SmpiHost(&host)); }); // parse the platform file: get the host list SIMIX_create_environment(argv[1]); SIMIX_comm_set_copy_data_callback(smpi_comm_copy_buffer_callback); smpi_init_options(); if (smpi_privatize_global_variables == SMPI_PRIVATIZE_DLOPEN) { std::string executable_copy = executable; // Prepare the copy of the binary (get its size) struct stat fdin_stat; stat(executable_copy.c_str(), &fdin_stat); off_t fdin_size = fdin_stat.st_size; static std::size_t rank = 0; simix_global->default_function = [executable_copy, fdin_size](std::vector args) { return std::function([executable_copy, fdin_size, args] { // Copy the dynamic library: std::string target_executable = executable_copy + "_" + std::to_string(getpid()) + "_" + std::to_string(rank++) + ".so"; int fdin = open(executable_copy.c_str(), O_RDONLY); xbt_assert(fdin >= 0, "Cannot read from %s", executable_copy.c_str()); int fdout = open(target_executable.c_str(), O_CREAT | O_RDWR, S_IRWXU); xbt_assert(fdout >= 0, "Cannot write into %s", target_executable.c_str()); #if HAVE_SENDFILE ssize_t sent_size = sendfile(fdout, fdin, NULL, fdin_size); xbt_assert(sent_size == fdin_size, "Error while copying %s: only %zd bytes copied instead of %ld (errno: %d -- %s)", target_executable.c_str(), sent_size, fdin_size, errno, strerror(errno)); #else XBT_VERB("Copy %d bytes into %s", static_cast(fdin_size), target_executable.c_str()); const int bufsize = 1024 * 1024 * 4; char buf[bufsize]; while (int got = read(fdin, buf, bufsize)) { if (got == -1) { xbt_assert(errno == EINTR, "Cannot read from %s", executable_copy.c_str()); } else { char* p = buf; int todo = got; while (int done = write(fdout, p, todo)) { if (done == -1) { xbt_assert(errno == EINTR, "Cannot write into %s", target_executable.c_str()); } else { p += done; todo -= done; } } } } #endif close(fdin); close(fdout); // Load the copy and resolve the entry point: void* handle = dlopen(target_executable.c_str(), RTLD_LAZY | RTLD_LOCAL | RTLD_DEEPBIND); int saved_errno = errno; if (xbt_cfg_get_boolean("smpi/keep-temps") == false) unlink(target_executable.c_str()); if (handle == nullptr) xbt_die("dlopen failed: %s (errno: %d -- %s)", dlerror(), saved_errno, strerror(saved_errno)); smpi_entry_point_type entry_point = smpi_resolve_function(handle); if (not entry_point) xbt_die("Could not resolve entry point"); smpi_run_entry_point(entry_point, args); }); }; } else { // Load the dynamic library and resolve the entry point: void* handle = dlopen(executable, RTLD_LAZY | RTLD_LOCAL | RTLD_DEEPBIND); if (handle == nullptr) xbt_die("dlopen failed for %s: %s (errno: %d -- %s)", executable, dlerror(), errno, strerror(errno)); smpi_entry_point_type entry_point = smpi_resolve_function(handle); if (not entry_point) xbt_die("main not found in %s", executable); // TODO, register the executable for SMPI privatization // Execute the same entry point for each simulated process: simix_global->default_function = [entry_point](std::vector args) { return std::function([entry_point, args] { smpi_run_entry_point(entry_point, args); }); }; } SIMIX_launch_application(argv[2]); SMPI_init(); /* Clean IO before the run */ fflush(stdout); fflush(stderr); if (MC_is_active()) { MC_run(); } else { SIMIX_run(); xbt_os_walltimer_stop(global_timer); if (xbt_cfg_get_boolean("smpi/display-timing")){ double global_time = xbt_os_timer_elapsed(global_timer); XBT_INFO("Simulated time: %g seconds. \n\n" "The simulation took %g seconds (after parsing and platform setup)\n" "%g seconds were actual computation of the application", SIMIX_get_clock(), global_time , smpi_total_benched_time); if (smpi_total_benched_time/global_time>=0.75) XBT_INFO("More than 75%% of the time was spent inside the application code.\n" "You may want to use sampling functions or trace replay to reduce this."); } } int ret = 0; int count = smpi_process_count(); for (int i = 0; i < count; i++) { if(process_data[i]->return_value()!=0){ ret=process_data[i]->return_value();//return first non 0 value break; } } smpi_global_destroy(); TRACE_end(); return ret; } // Called either directly from the user code, or from the code called by smpirun void SMPI_init(){ smpi_init_logs(); smpi_init_options(); smpi_global_init(); smpi_check_options(); TRACE_smpi_alloc(); simgrid::surf::surfExitCallbacks.connect(TRACE_smpi_release); if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) smpi_backup_global_memory_segment(); } void SMPI_finalize(){ smpi_global_destroy(); } void smpi_mpi_init() { if(smpi_init_sleep > 0) simcall_process_sleep(smpi_init_sleep); } double smpi_mpi_wtime(){ double time; if (smpi_process()->initialized() != 0 && smpi_process()->finalized() == 0 && smpi_process()->sampling() == 0) { smpi_bench_end(); time = SIMIX_get_clock(); // to avoid deadlocks if used as a break condition, such as // while (MPI_Wtime(...) < time_limit) { // .... // } // because the time will not normally advance when only calls to MPI_Wtime // are made -> deadlock (MPI_Wtime never reaches the time limit) if(smpi_wtime_sleep > 0) simcall_process_sleep(smpi_wtime_sleep); smpi_bench_begin(); } else { time = SIMIX_get_clock(); } return time; } SimGrid-3.18/src/smpi/internals/smpi_bench.cpp0000644000175000017500000003407513217757320021700 0ustar mquinsonmquinson/* Copyright (c) 2007, 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "simgrid/host.h" #include "simgrid/modelchecker.h" #include "smpi_comm.hpp" #include "smpi_process.hpp" #include "src/internal_config.h" #include "src/mc/mc_replay.hpp" #include "src/simix/ActorImpl.hpp" #include #ifndef WIN32 #include #endif #include #if HAVE_PAPI #include #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_bench, smpi, "Logging specific to SMPI (benchmarking)"); double smpi_cpu_threshold = -1; double smpi_host_speed; shared_malloc_type smpi_cfg_shared_malloc = shmalloc_global; double smpi_total_benched_time = 0; extern "C" XBT_PUBLIC(void) smpi_execute_flops_(double *flops); void smpi_execute_flops_(double *flops) { smpi_execute_flops(*flops); } extern "C" XBT_PUBLIC(void) smpi_execute_(double *duration); void smpi_execute_(double *duration) { smpi_execute(*duration); } void smpi_execute_flops(double flops) { XBT_DEBUG("Handle real computation time: %f flops", flops); smx_activity_t action = simcall_execution_start("computation", flops, 1, 0, smpi_process()->process()->host); simcall_set_category (action, TRACE_internal_smpi_get_category()); simcall_execution_wait(action); smpi_switch_data_segment(smpi_process()->index()); } void smpi_execute(double duration) { if (duration >= smpi_cpu_threshold) { XBT_DEBUG("Sleep for %g to handle real computation time", duration); double flops = duration * smpi_host_speed; int rank = smpi_process()->index(); TRACE_smpi_computing_in(rank, flops); smpi_execute_flops(flops); TRACE_smpi_computing_out(rank); } else { XBT_DEBUG("Real computation took %g while option smpi/cpu-threshold is set to %g => ignore it", duration, smpi_cpu_threshold); } } void smpi_execute_benched(double duration) { smpi_bench_end(); double speed = sg_host_speed(sg_host_self()); smpi_execute_flops(duration*speed); smpi_bench_begin(); } void smpi_bench_begin() { if (smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP) { smpi_switch_data_segment(smpi_process()->index()); } if (MC_is_active() || MC_record_replay_is_active()) return; #if HAVE_PAPI if (not xbt_cfg_get_string("smpi/papi-events").empty()) { int event_set = smpi_process()->papi_event_set(); // PAPI_start sets everything to 0! See man(3) PAPI_start if (PAPI_LOW_LEVEL_INITED == PAPI_is_initialized()) { if (PAPI_start(event_set) != PAPI_OK) { // TODO This needs some proper handling. XBT_CRITICAL("Could not start PAPI counters.\n"); xbt_die("Error."); } } } #endif xbt_os_threadtimer_start(smpi_process()->timer()); } void smpi_bench_end() { if (MC_is_active() || MC_record_replay_is_active()) return; double speedup = 1; xbt_os_timer_t timer = smpi_process()->timer(); xbt_os_threadtimer_stop(timer); #if HAVE_PAPI /** * An MPI function has been called and now is the right time to update * our PAPI counters for this process. */ if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { papi_counter_t& counter_data = smpi_process()->papi_counters(); int event_set = smpi_process()->papi_event_set(); std::vector event_values = std::vector(counter_data.size()); if (PAPI_stop(event_set, &event_values[0]) != PAPI_OK) { // Error XBT_CRITICAL("Could not stop PAPI counters.\n"); xbt_die("Error."); } else { for (unsigned int i = 0; i < counter_data.size(); i++) { counter_data[i].second += event_values[i]; } } } #endif if (smpi_process()->sampling()) { XBT_CRITICAL("Cannot do recursive benchmarks."); XBT_CRITICAL("Are you trying to make a call to MPI within a SMPI_SAMPLE_ block?"); xbt_backtrace_display_current(); xbt_die("Aborting."); } if (xbt_cfg_get_string("smpi/comp-adjustment-file")[0] != '\0') { // Maybe we need to artificially speed up or slow // down our computation based on our statistical analysis. smpi_trace_call_location_t* loc = smpi_process()->call_location(); std::string key = loc->get_composed_key(); std::unordered_map::const_iterator it = location2speedup.find(key); if (it != location2speedup.end()) { speedup = it->second; } } // Simulate the benchmarked computation unless disabled via command-line argument if (xbt_cfg_get_boolean("smpi/simulate-computation")) { smpi_execute(xbt_os_timer_elapsed(timer)/speedup); } #if HAVE_PAPI if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0' && TRACE_smpi_is_enabled()) { container_t container = new simgrid::instr::Container(std::string("rank-") + std::to_string(smpi_process()->index())); papi_counter_t& counter_data = smpi_process()->papi_counters(); for (auto const& pair : counter_data) { new simgrid::instr::SetVariableEvent( surf_get_clock(), container, PJ_type_get(/* countername */ pair.first.c_str(), container->type), pair.second); } } #endif smpi_total_benched_time += xbt_os_timer_elapsed(timer); } /* Private sleep function used by smpi_sleep() and smpi_usleep() */ static unsigned int private_sleep(double secs) { smpi_bench_end(); XBT_DEBUG("Sleep for: %lf secs", secs); int rank = MPI_COMM_WORLD->rank(); TRACE_smpi_sleeping_in(rank, secs); simcall_process_sleep(secs); TRACE_smpi_sleeping_out(rank); smpi_bench_begin(); return 0; } unsigned int smpi_sleep(unsigned int secs) { return private_sleep(static_cast(secs)); } int smpi_usleep(useconds_t usecs) { return static_cast(private_sleep(static_cast(usecs) / 1000000.0)); } #if _POSIX_TIMERS > 0 int smpi_nanosleep(const struct timespec* tp, struct timespec* /*t*/) { return static_cast(private_sleep(static_cast(tp->tv_sec + tp->tv_nsec / 1000000000.0))); } #endif int smpi_gettimeofday(struct timeval* tv, void* /*tz*/) { smpi_bench_end(); double now = SIMIX_get_clock(); if (tv) { tv->tv_sec = static_cast(now); #ifdef WIN32 tv->tv_usec = static_cast((now - tv->tv_sec) * 1e6); #else tv->tv_usec = static_cast((now - tv->tv_sec) * 1e6); #endif } smpi_bench_begin(); return 0; } #if _POSIX_TIMERS > 0 int smpi_clock_gettime(clockid_t /*clk_id*/, struct timespec* tp) { //there is only one time in SMPI, so clk_id is ignored. smpi_bench_end(); double now = SIMIX_get_clock(); if (tp) { tp->tv_sec = static_cast(now); tp->tv_nsec = static_cast((now - tp->tv_sec) * 1e9); } smpi_bench_begin(); return 0; } #endif extern double sg_surf_precision; unsigned long long smpi_rastro_resolution () { smpi_bench_end(); double resolution = (1/sg_surf_precision); smpi_bench_begin(); return static_cast(resolution); } unsigned long long smpi_rastro_timestamp () { smpi_bench_end(); double now = SIMIX_get_clock(); unsigned long long sec = static_cast(now); unsigned long long pre = (now - sec) * smpi_rastro_resolution(); smpi_bench_begin(); return static_cast(sec) * smpi_rastro_resolution() + pre; } /* ****************************** Functions related to the SMPI_SAMPLE_ macros ************************************/ namespace { class SampleLocation : public std::string { public: SampleLocation(bool global, const char* file, int line) : std::string(std::string(file) + ":" + std::to_string(line)) { if (not global) this->append(":" + std::to_string(smpi_process()->index())); } }; class LocalData { public: double threshold; /* maximal stderr requested (if positive) */ double relstderr; /* observed stderr so far */ double mean; /* mean of benched times, to be used if the block is disabled */ double sum; /* sum of benched times (to compute the mean and stderr) */ double sum_pow2; /* sum of the square of the benched times (to compute the stderr) */ int iters; /* amount of requested iterations */ int count; /* amount of iterations done so far */ bool benching; /* true: we are benchmarking; false: we have enough data, no bench anymore */ bool need_more_benchs() const; }; } std::unordered_map> samples; bool LocalData::need_more_benchs() const { bool res = (count < iters) || (threshold > 0.0 && (count < 2 || // not enough data relstderr > threshold // stderr too high yet )); XBT_DEBUG("%s (count:%d iter:%d stderr:%f thres:%f mean:%fs)", (res ? "need more data" : "enough benchs"), count, iters, relstderr, threshold, mean); return res; } void smpi_sample_1(int global, const char *file, int line, int iters, double threshold) { SampleLocation loc(global, file, line); smpi_bench_end(); /* Take time from previous, unrelated computation into account */ smpi_process()->set_sampling(1); auto insert = samples.emplace(loc, LocalData{ threshold, // threshold 0.0, // relstderr 0.0, // mean 0.0, // sum 0.0, // sum_pow2 iters, // iters 0, // count true // benching (if we have no data, we need at least one) }); LocalData& data = insert.first->second; if (insert.second) { XBT_DEBUG("XXXXX First time ever on benched nest %s.", loc.c_str()); xbt_assert(threshold > 0 || iters > 0, "You should provide either a positive amount of iterations to bench, or a positive maximal stderr (or both)"); } else { if (data.iters != iters || data.threshold != threshold) { XBT_ERROR("Asked to bench block %s with different settings %d, %f is not %d, %f. " "How did you manage to give two numbers at the same line??", loc.c_str(), data.iters, data.threshold, iters, threshold); THROW_IMPOSSIBLE; } // if we already have some data, check whether sample_2 should get one more bench or whether it should emulate // the computation instead data.benching = data.need_more_benchs(); XBT_DEBUG("XXXX Re-entering the benched nest %s. %s", loc.c_str(), (data.benching ? "more benching needed" : "we have enough data, skip computes")); } } int smpi_sample_2(int global, const char *file, int line) { SampleLocation loc(global, file, line); int res; XBT_DEBUG("sample2 %s", loc.c_str()); auto sample = samples.find(loc); if (sample == samples.end()) xbt_die("Y U NO use SMPI_SAMPLE_* macros? Stop messing directly with smpi_sample_* functions!"); LocalData& data = sample->second; if (data.benching) { // we need to run a new bench XBT_DEBUG("benchmarking: count:%d iter:%d stderr:%f thres:%f; mean:%f", data.count, data.iters, data.relstderr, data.threshold, data.mean); res = 1; } else { // Enough data, no more bench (either we got enough data from previous visits to this benched nest, or we just //ran one bench and need to bail out now that our job is done). Just sleep instead XBT_DEBUG("No benchmark (either no need, or just ran one): count >= iter (%d >= %d) or stderrset_sampling(0); res = 0; // prepare to capture future, unrelated computations } smpi_bench_begin(); return res; } void smpi_sample_3(int global, const char *file, int line) { SampleLocation loc(global, file, line); XBT_DEBUG("sample3 %s", loc.c_str()); auto sample = samples.find(loc); if (sample == samples.end()) xbt_die("Y U NO use SMPI_SAMPLE_* macros? Stop messing directly with smpi_sample_* functions!"); LocalData& data = sample->second; if (not data.benching) THROW_IMPOSSIBLE; // ok, benchmarking this loop is over xbt_os_threadtimer_stop(smpi_process()->timer()); // update the stats data.count++; double period = xbt_os_timer_elapsed(smpi_process()->timer()); data.sum += period; data.sum_pow2 += period * period; double n = static_cast(data.count); data.mean = data.sum / n; data.relstderr = sqrt((data.sum_pow2 / n - data.mean * data.mean) / n) / data.mean; if (data.need_more_benchs()) { data.mean = period; // Still in benching process; We want sample_2 to simulate the exact time of this loop // occurrence before leaving, not the mean over the history } XBT_DEBUG("Average mean after %d steps is %f, relative standard error is %f (sample was %f)", data.count, data.mean, data.relstderr, period); // That's enough for now, prevent sample_2 to run the same code over and over data.benching = false; } extern "C" { /** These functions will be called from the user code **/ smpi_trace_call_location_t* smpi_trace_get_call_location() { return smpi_process()->call_location(); } void smpi_trace_set_call_location(const char* file, const int line) { smpi_trace_call_location_t* loc = smpi_process()->call_location(); loc->previous_filename = loc->filename; loc->previous_linenumber = loc->linenumber; loc->filename = file; loc->linenumber = line; } /** Required for Fortran bindings */ void smpi_trace_set_call_location_(const char* file, int* line) { smpi_trace_set_call_location(file, *line); } /** Required for Fortran if -fsecond-underscore is activated */ void smpi_trace_set_call_location__(const char* file, int* line) { smpi_trace_set_call_location(file, *line); } } void smpi_bench_destroy() { samples.clear(); } SimGrid-3.18/src/smpi/internals/smpi_deployment.cpp0000644000175000017500000000753413217757320023001 0ustar mquinsonmquinson/* Copyright (c) 2004-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "SmpiHost.hpp" #include "private.hpp" #include "simgrid/msg.h" /* barrier */ #include "smpi_comm.hpp" #include namespace simgrid { namespace smpi { namespace app { class Instance { public: Instance(const char* name, int max_no_processes, int process_count, MPI_Comm comm, msg_bar_t finalization_barrier) : name(name) , size(max_no_processes) , present_processes(0) , index(process_count) , comm_world(comm) , finalization_barrier(finalization_barrier) { } const char* name; int size; int present_processes; int index; // Badly named. This should be "no_processes_when_registering" ;) MPI_Comm comm_world; msg_bar_t finalization_barrier; }; } } namespace s4u { extern std::map host_list; } } using simgrid::smpi::app::Instance; static std::map smpi_instances; extern int process_count; // How many processes have been allocated over all instances? extern int* index_to_process_data; /** \ingroup smpi_simulation * \brief Registers a running instance of a MPI program. * * FIXME : remove MSG from the loop at some point. * \param name the reference name of the function. * \param code the main mpi function (must have a int ..(int argc, char *argv[])) prototype * \param num_processes the size of the instance we want to deploy */ void SMPI_app_instance_register(const char *name, xbt_main_func_t code, int num_processes) { if (code != nullptr) { // When started with smpirun, we will not execute a function SIMIX_function_register(name, code); } static int already_called = 0; if (not already_called) { already_called = 1; for (auto const& item : simgrid::s4u::host_list) { simgrid::s4u::Host* host = item.second; host->extension_set(new simgrid::smpi::SmpiHost(host)); } } Instance instance(name, num_processes, process_count, MPI_COMM_NULL, MSG_barrier_init(num_processes)); MPI_Group group = new simgrid::smpi::Group(instance.size); instance.comm_world = new simgrid::smpi::Comm(group, nullptr); MPI_Attr_put(instance.comm_world, MPI_UNIVERSE_SIZE, reinterpret_cast(instance.size)); process_count+=num_processes; smpi_instances.insert(std::pair(name, instance)); } //get the index of the process in the process_data array void smpi_deployment_register_process(const char* instance_id, int rank, int index) { if (smpi_instances.empty()) { // no instance registered, we probably used smpirun. index_to_process_data[index]=index; return; } Instance& instance = smpi_instances.at(instance_id); instance.present_processes++; index_to_process_data[index] = instance.index + rank; instance.comm_world->group()->set_mapping(index, rank); } //get the index of the process in the process_data array MPI_Comm* smpi_deployment_comm_world(const char* instance_id) { if (smpi_instances.empty()) { // no instance registered, we probably used smpirun. return nullptr; } Instance& instance = smpi_instances.at(instance_id); return &instance.comm_world; } msg_bar_t smpi_deployment_finalization_barrier(const char* instance_id) { if (smpi_instances.empty()) { // no instance registered, we probably used smpirun. return nullptr; } Instance& instance = smpi_instances.at(instance_id); return instance.finalization_barrier; } void smpi_deployment_cleanup_instances(){ for (auto const& item : smpi_instances) { Instance instance = item.second; MSG_barrier_destroy(instance.finalization_barrier); simgrid::smpi::Comm::destroy(instance.comm_world); } smpi_instances.clear(); } SimGrid-3.18/src/smpi/internals/smpi_shared.cpp0000644000175000017500000004754313217757320022073 0ustar mquinsonmquinson/* Copyright (c) 2007, 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Shared allocations are handled through shared memory segments. * Associated data and metadata are used as follows: * * mmap #1 * `allocs' map ---- -. * ---------- shared_data_t shared_metadata_t / | | | * .->| | ---> -------------------- <--. ----------------- | | | | * | ---------- | fd of | | | size of mmap | --| | | | * | | count (2) | |-- | data | \ | | | * `----------------- | | | ----------------- ---- | * -------------------- | ^ | * | | | * | | `allocs_metadata' map | * | | ---------------------- | * | `-- | |<-' * | .-- | |<-. * | | ---------------------- | * | | | * | | | * | | | * | | mmap #2 | * | v ---- -' * | shared_metadata_t / | | * | ----------------- | | | * | | size of mmap | --| | | * `-- | data | | | | * ----------------- | | | * \ | | * ---- */ #include #include #include "private.hpp" #include #include #ifndef WIN32 #include #endif #include #include #include #ifndef MAP_ANONYMOUS #define MAP_ANONYMOUS MAP_ANON #endif #ifndef MAP_POPULATE #define MAP_POPULATE 0 #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_shared, smpi, "Logging specific to SMPI (shared memory macros)"); #define PTR_STRLEN (2 + 2 * sizeof(void*) + 1) namespace{ /** Some location in the source code * * This information is used by SMPI_SHARED_MALLOC to allocate some shared memory for all simulated processes. */ class smpi_source_location : public std::string { public: smpi_source_location() = default; smpi_source_location(const char* filename, int line) : std::string(std::string(filename) + ":" + std::to_string(line)) { } }; struct shared_data_t { int fd = -1; int count = 0; }; std::unordered_map> allocs; typedef decltype(allocs)::value_type shared_data_key_type; struct shared_metadata_t { size_t size; size_t allocated_size; void *allocated_ptr; std::vector> private_blocks; shared_data_key_type* data; }; std::map allocs_metadata; std::map calls; #ifndef WIN32 static int smpi_shared_malloc_bogusfile = -1; static int smpi_shared_malloc_bogusfile_huge_page = -1; static unsigned long smpi_shared_malloc_blocksize = 1UL << 20; #endif } void smpi_shared_destroy() { allocs.clear(); allocs_metadata.clear(); calls.clear(); } static size_t shm_size(int fd) { struct stat st; if(fstat(fd, &st) < 0) { xbt_die("Could not stat fd %d: %s", fd, strerror(errno)); } return static_cast(st.st_size); } #ifndef WIN32 static void* shm_map(int fd, size_t size, shared_data_key_type* data) { char loc[PTR_STRLEN]; shared_metadata_t meta; if(size > shm_size(fd) && (ftruncate(fd, static_cast(size)) < 0)) { xbt_die("Could not truncate fd %d to %zu: %s", fd, size, strerror(errno)); } void* mem = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); if(mem == MAP_FAILED) { xbt_die( "Failed to map fd %d with size %zu: %s\n" "If you are running a lot of ranks, you may be exceeding the amount of mappings allowed per process.\n" "On Linux systems, change this value with sudo sysctl -w vm.max_map_count=newvalue (default value: 65536)\n" "Please see http://simgrid.gforge.inria.fr/simgrid/latest/doc/html/options.html#options_virt for more info.", fd, size, strerror(errno)); } snprintf(loc, PTR_STRLEN, "%p", mem); meta.size = size; meta.data = data; allocs_metadata[mem] = meta; XBT_DEBUG("MMAP %zu to %p", size, mem); return mem; } static void *smpi_shared_malloc_local(size_t size, const char *file, int line) { void* mem; smpi_source_location loc(file, line); auto res = allocs.insert(std::make_pair(loc, shared_data_t())); auto data = res.first; if (res.second) { // The new element was inserted. // Generate a shared memory name from the address of the shared_data: char shmname[32]; // cannot be longer than PSHMNAMLEN = 31 on Mac OS X (shm_open raises ENAMETOOLONG otherwise) snprintf(shmname, 31, "/shmalloc%p", &*data); int fd = shm_open(shmname, O_RDWR | O_CREAT | O_EXCL, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (fd < 0) { if (errno == EEXIST) xbt_die("Please cleanup /dev/shm/%s", shmname); else xbt_die("An unhandled error occurred while opening %s. shm_open: %s", shmname, strerror(errno)); } data->second.fd = fd; data->second.count = 1; mem = shm_map(fd, size, &*data); if (shm_unlink(shmname) < 0) { XBT_WARN("Could not early unlink %s. shm_unlink: %s", shmname, strerror(errno)); } XBT_DEBUG("Mapping %s at %p through %d", shmname, mem, fd); } else { mem = shm_map(data->second.fd, size, &*data); data->second.count++; } XBT_DEBUG("Shared malloc %zu in %p (metadata at %p)", size, mem, &*data); return mem; } // Align functions, from http://stackoverflow.com/questions/4840410/how-to-align-a-pointer-in-c #define PAGE_SIZE 0x1000 #define ALIGN_UP(n, align) (((n) + (align)-1) & -(align)) #define ALIGN_DOWN(n, align) ((n) & -(align)) #define HUGE_PAGE_SIZE 1<<21 /* Similar to smpi_shared_malloc, but only sharing the blocks described by shared_block_offsets. * This array contains the offsets (in bytes) of the block to share. * Even indices are the start offsets (included), odd indices are the stop offsets (excluded). * For instance, if shared_block_offsets == {27, 42}, then the elements mem[27], mem[28], ..., mem[41] are shared. * The others are not. */ void* smpi_shared_malloc_partial(size_t size, size_t* shared_block_offsets, int nb_shared_blocks) { std::string huge_page_mount_point = xbt_cfg_get_string("smpi/shared-malloc-hugepage"); bool use_huge_page = not huge_page_mount_point.empty(); #ifndef MAP_HUGETLB /* If the system header don't define that mmap flag */ xbt_assert(not use_huge_page, "Huge pages are not available on your system, you cannot use the smpi/shared-malloc-hugepage option."); use_huge_page = 0; #endif smpi_shared_malloc_blocksize = static_cast(xbt_cfg_get_double("smpi/shared-malloc-blocksize")); void* mem; size_t allocated_size; if(use_huge_page) { xbt_assert(smpi_shared_malloc_blocksize == HUGE_PAGE_SIZE, "the block size of shared malloc should be equal to the size of a huge page."); allocated_size = size + 2*smpi_shared_malloc_blocksize; } else { xbt_assert(smpi_shared_malloc_blocksize % PAGE_SIZE == 0, "the block size of shared malloc should be a multiple of the page size."); allocated_size = size; } /* First reserve memory area */ void* allocated_ptr = mmap(NULL, allocated_size, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); xbt_assert(allocated_ptr != MAP_FAILED, "Failed to allocate %zuMiB of memory. Run \"sysctl vm.overcommit_memory=1\" as root " "to allow big allocations.\n", size >> 20); if(use_huge_page) mem = (void*)ALIGN_UP((uint64_t)allocated_ptr, HUGE_PAGE_SIZE); else mem = allocated_ptr; XBT_DEBUG("global shared allocation. Blocksize %lu", smpi_shared_malloc_blocksize); /* Create a fd to a new file on disk, make it smpi_shared_malloc_blocksize big, and unlink it. * It still exists in memory but not in the file system (thus it cannot be leaked). */ /* Create bogus file if not done already * We need two different bogusfiles: * smpi_shared_malloc_bogusfile_huge_page is used for calls to mmap *with* MAP_HUGETLB, * smpi_shared_malloc_bogusfile is used for calls to mmap *without* MAP_HUGETLB. * We cannot use a same file for the two type of calls, since the first one needs to be * opened in a hugetlbfs mount point whereas the second needs to be a "classical" file. */ if(use_huge_page && smpi_shared_malloc_bogusfile_huge_page == -1) { std::string huge_page_filename = huge_page_mount_point + "/simgrid-shmalloc-XXXXXX"; smpi_shared_malloc_bogusfile_huge_page = mkstemp((char*)huge_page_filename.c_str()); XBT_DEBUG("bogusfile_huge_page: %s\n", huge_page_filename.c_str()); unlink(huge_page_filename.c_str()); } if(smpi_shared_malloc_bogusfile == -1) { char name[] = "/tmp/simgrid-shmalloc-XXXXXX"; smpi_shared_malloc_bogusfile = mkstemp(name); XBT_DEBUG("bogusfile : %s\n", name); unlink(name); char* dumb = new char[smpi_shared_malloc_blocksize](); // zero initialized ssize_t err = write(smpi_shared_malloc_bogusfile, dumb, smpi_shared_malloc_blocksize); if(err<0) xbt_die("Could not write bogus file for shared malloc"); delete[] dumb; } int mmap_base_flag = MAP_FIXED | MAP_SHARED | MAP_POPULATE; int mmap_flag = mmap_base_flag; int huge_fd = use_huge_page ? smpi_shared_malloc_bogusfile_huge_page : smpi_shared_malloc_bogusfile; #ifdef MAP_HUGETLB if(use_huge_page) mmap_flag |= MAP_HUGETLB; #endif XBT_DEBUG("global shared allocation, begin mmap"); /* Map the bogus file in place of the anonymous memory */ for(int i_block = 0; i_block < nb_shared_blocks; i_block ++) { XBT_DEBUG("\tglobal shared allocation, mmap block %d/%d", i_block+1, nb_shared_blocks); size_t start_offset = shared_block_offsets[2*i_block]; size_t stop_offset = shared_block_offsets[2*i_block+1]; xbt_assert(start_offset < stop_offset, "start_offset (%zu) should be lower than stop offset (%zu)", start_offset, stop_offset); xbt_assert(stop_offset <= size, "stop_offset (%zu) should be lower than size (%zu)", stop_offset, size); if(i_block < nb_shared_blocks-1) xbt_assert(stop_offset < shared_block_offsets[2*i_block+2], "stop_offset (%zu) should be lower than its successor start offset (%zu)", stop_offset, shared_block_offsets[2*i_block+2]); size_t start_block_offset = ALIGN_UP(start_offset, smpi_shared_malloc_blocksize); size_t stop_block_offset = ALIGN_DOWN(stop_offset, smpi_shared_malloc_blocksize); for (size_t offset = start_block_offset; offset < stop_block_offset; offset += smpi_shared_malloc_blocksize) { XBT_DEBUG("\t\tglobal shared allocation, mmap block offset %zx", offset); void* pos = (void*)((unsigned long)mem + offset); void* res = mmap(pos, smpi_shared_malloc_blocksize, PROT_READ | PROT_WRITE, mmap_flag, huge_fd, 0); xbt_assert(res == pos, "Could not map folded virtual memory (%s). Do you perhaps need to increase the " "size of the mapped file using --cfg=smpi/shared-malloc-blocksize=newvalue (default 1048576) ? " "You can also try using the sysctl vm.max_map_count. " "If you are using huge pages, check that you have at least one huge page (/proc/sys/vm/nr_hugepages) " "and that the directory you are passing is mounted correctly (mount /path/to/huge -t hugetlbfs -o rw,mode=0777).", strerror(errno)); } size_t low_page_start_offset = ALIGN_UP(start_offset, PAGE_SIZE); size_t low_page_stop_offset = start_block_offset < ALIGN_DOWN(stop_offset, PAGE_SIZE) ? start_block_offset : ALIGN_DOWN(stop_offset, PAGE_SIZE); if(low_page_start_offset < low_page_stop_offset) { XBT_DEBUG("\t\tglobal shared allocation, mmap block start"); void* pos = (void*)((unsigned long)mem + low_page_start_offset); void* res = mmap(pos, low_page_stop_offset-low_page_start_offset, PROT_READ | PROT_WRITE, mmap_base_flag, // not a full huge page smpi_shared_malloc_bogusfile, 0); xbt_assert(res == pos, "Could not map folded virtual memory (%s). Do you perhaps need to increase the " "size of the mapped file using --cfg=smpi/shared-malloc-blocksize=newvalue (default 1048576) ?" "You can also try using the sysctl vm.max_map_count", strerror(errno)); } if(low_page_stop_offset <= stop_block_offset) { XBT_DEBUG("\t\tglobal shared allocation, mmap block stop"); size_t high_page_stop_offset = stop_offset == size ? size : ALIGN_DOWN(stop_offset, PAGE_SIZE); if(high_page_stop_offset > stop_block_offset) { void* pos = (void*)((unsigned long)mem + stop_block_offset); void* res = mmap(pos, high_page_stop_offset-stop_block_offset, PROT_READ | PROT_WRITE, mmap_base_flag, // not a full huge page smpi_shared_malloc_bogusfile, 0); xbt_assert(res == pos, "Could not map folded virtual memory (%s). Do you perhaps need to increase the " "size of the mapped file using --cfg=smpi/shared-malloc-blocksize=newvalue (default 1048576) ?" "You can also try using the sysctl vm.max_map_count", strerror(errno)); } } } shared_metadata_t newmeta; //register metadata for memcpy avoidance shared_data_key_type* data = new shared_data_key_type; data->second.fd = -1; data->second.count = 1; newmeta.size = size; newmeta.data = data; newmeta.allocated_ptr = allocated_ptr; newmeta.allocated_size = allocated_size; if(shared_block_offsets[0] > 0) { newmeta.private_blocks.push_back(std::make_pair(0, shared_block_offsets[0])); } int i_block; for(i_block = 0; i_block < nb_shared_blocks-1; i_block ++) { newmeta.private_blocks.push_back(std::make_pair(shared_block_offsets[2*i_block+1], shared_block_offsets[2*i_block+2])); } if(shared_block_offsets[2*i_block+1] < size) { newmeta.private_blocks.push_back(std::make_pair(shared_block_offsets[2*i_block+1], size)); } allocs_metadata[mem] = newmeta; XBT_DEBUG("global shared allocation, allocated_ptr %p - %p", allocated_ptr, (void*)(((uint64_t)allocated_ptr)+allocated_size)); XBT_DEBUG("global shared allocation, returned_ptr %p - %p", mem, (void*)(((uint64_t)mem)+size)); return mem; } void *smpi_shared_malloc(size_t size, const char *file, int line) { if (size > 0 && smpi_cfg_shared_malloc == shmalloc_local) { return smpi_shared_malloc_local(size, file, line); } else if (smpi_cfg_shared_malloc == shmalloc_global) { int nb_shared_blocks = 1; size_t shared_block_offsets[2] = {0, size}; return smpi_shared_malloc_partial(size, shared_block_offsets, nb_shared_blocks); } XBT_DEBUG("Classic allocation of %zu bytes", size); return ::operator new(size); } int smpi_is_shared(void* ptr, std::vector> &private_blocks, size_t *offset){ private_blocks.clear(); // being paranoid if (allocs_metadata.empty()) return 0; if ( smpi_cfg_shared_malloc == shmalloc_local || smpi_cfg_shared_malloc == shmalloc_global) { auto low = allocs_metadata.lower_bound(ptr); if (low != allocs_metadata.end() && low->first == ptr) { private_blocks = low->second.private_blocks; *offset = 0; return 1; } if (low == allocs_metadata.begin()) return 0; low --; if (ptr < (char*)low->first + low->second.size) { xbt_assert(ptr > (char*)low->first, "Oops, there seems to be a bug in the shared memory metadata."); *offset = ((uint8_t*)ptr) - ((uint8_t*) low->first); private_blocks = low->second.private_blocks; return 1; } return 0; } else { return 0; } } std::vector> shift_and_frame_private_blocks(const std::vector> vec, size_t offset, size_t buff_size) { std::vector> result; for (auto const& block : vec) { auto new_block = std::make_pair(std::min(std::max((size_t)0, block.first - offset), buff_size), std::min(std::max((size_t)0, block.second - offset), buff_size)); if (new_block.second > 0 && new_block.first < buff_size) result.push_back(new_block); } return result; } std::vector> merge_private_blocks(std::vector> src, std::vector> dst) { std::vector> result; unsigned i_src = 0; unsigned i_dst = 0; while(i_src < src.size() && i_dst < dst.size()) { std::pair block; if(src[i_src].second <= dst[i_dst].first) { i_src++; } else if(dst[i_dst].second <= src[i_src].first) { i_dst++; } else { // src.second > dst.first && dst.second > src.first → the blocks are overlapping block = std::make_pair(std::max(src[i_src].first, dst[i_dst].first), std::min(src[i_src].second, dst[i_dst].second)); result.push_back(block); if(src[i_src].second < dst[i_dst].second) i_src ++; else i_dst ++; } } return result; } void smpi_shared_free(void *ptr) { if (smpi_cfg_shared_malloc == shmalloc_local) { char loc[PTR_STRLEN]; snprintf(loc, PTR_STRLEN, "%p", ptr); auto meta = allocs_metadata.find(ptr); if (meta == allocs_metadata.end()) { XBT_WARN("Cannot free: %p was not shared-allocated by SMPI - maybe its size was 0?", ptr); return; } shared_data_t* data = &meta->second.data->second; if (munmap(meta->second.allocated_ptr, meta->second.allocated_size) < 0) { XBT_WARN("Unmapping of fd %d failed: %s", data->fd, strerror(errno)); } data->count--; if (data->count <= 0) { close(data->fd); allocs.erase(allocs.find(meta->second.data->first)); allocs_metadata.erase(ptr); XBT_DEBUG("Shared free - with removal - of %p", ptr); } else { XBT_DEBUG("Shared free - no removal - of %p, count = %d", ptr, data->count); } } else if (smpi_cfg_shared_malloc == shmalloc_global) { auto meta = allocs_metadata.find(ptr); if (meta != allocs_metadata.end()){ meta->second.data->second.count--; if(meta->second.data->second.count==0) delete meta->second.data; } munmap(ptr, meta->second.size); } else { XBT_DEBUG("Classic deallocation of %p", ptr); ::operator delete(ptr); } } #endif int smpi_shared_known_call(const char* func, const char* input) { std::string loc = std::string(func) + ":" + input; return calls.find(loc) != calls.end(); } void* smpi_shared_get_call(const char* func, const char* input) { std::string loc = std::string(func) + ":" + input; return calls.at(loc); } void* smpi_shared_set_call(const char* func, const char* input, void* data) { std::string loc = std::string(func) + ":" + input; calls[loc] = data; return data; } SimGrid-3.18/src/smpi/internals/smpi_replay.cpp0000644000175000017500000010676413217757320022122 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_group.hpp" #include "smpi_process.hpp" #include "smpi_request.hpp" #include "xbt/replay.hpp" #include #include #define KEY_SIZE (sizeof(int) * 2 + 1) XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_replay,smpi,"Trace Replay with SMPI"); int communicator_size = 0; static int active_processes = 0; std::unordered_map*> reqq; MPI_Datatype MPI_DEFAULT_TYPE; MPI_Datatype MPI_CURRENT_TYPE; static int sendbuffer_size=0; char* sendbuffer=nullptr; static int recvbuffer_size=0; char* recvbuffer=nullptr; static void log_timed_action (const char *const *action, double clock){ if (XBT_LOG_ISENABLED(smpi_replay, xbt_log_priority_verbose)){ char *name = xbt_str_join_array(action, " "); XBT_VERB("%s %f", name, smpi_process()->simulated_elapsed()-clock); xbt_free(name); } } static std::vector* get_reqq_self() { return reqq.at(smpi_process()->index()); } static void set_reqq_self(std::vector *mpi_request) { reqq.insert({smpi_process()->index(), mpi_request}); } //allocate a single buffer for all sends, growing it if needed void* smpi_get_tmp_sendbuffer(int size) { if (not smpi_process()->replaying()) return xbt_malloc(size); if (sendbuffer_size(xbt_realloc(sendbuffer,size)); sendbuffer_size=size; } return sendbuffer; } //allocate a single buffer for all recv void* smpi_get_tmp_recvbuffer(int size){ if (not smpi_process()->replaying()) return xbt_malloc(size); if (recvbuffer_size(xbt_realloc(recvbuffer,size)); recvbuffer_size=size; } return recvbuffer; } void smpi_free_tmp_buffer(void* buf){ if (not smpi_process()->replaying()) xbt_free(buf); } /* Helper function */ static double parse_double(const char *string) { char *endptr; double value = strtod(string, &endptr); if (*endptr != '\0') THROWF(unknown_error, 0, "%s is not a double", string); return value; } //TODO: this logic should be moved inside the datatype class, to support all predefined types and get rid of is_replayable. static MPI_Datatype decode_datatype(const char *const action) { switch(atoi(action)) { case 0: MPI_CURRENT_TYPE=MPI_DOUBLE; break; case 1: MPI_CURRENT_TYPE=MPI_INT; break; case 2: MPI_CURRENT_TYPE=MPI_CHAR; break; case 3: MPI_CURRENT_TYPE=MPI_SHORT; break; case 4: MPI_CURRENT_TYPE=MPI_LONG; break; case 5: MPI_CURRENT_TYPE=MPI_FLOAT; break; case 6: MPI_CURRENT_TYPE=MPI_BYTE; break; default: MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; break; } return MPI_CURRENT_TYPE; } const char* encode_datatype(MPI_Datatype datatype) { if (datatype==MPI_BYTE) return ""; if(datatype==MPI_DOUBLE) return "0"; if(datatype==MPI_INT) return "1"; if(datatype==MPI_CHAR) return "2"; if(datatype==MPI_SHORT) return "3"; if(datatype==MPI_LONG) return "4"; if(datatype==MPI_FLOAT) return "5"; // default - not implemented. // do not warn here as we pass in this function even for other trace formats return "-1"; } #define CHECK_ACTION_PARAMS(action, mandatory, optional) {\ int i=0;\ while(action[i]!=nullptr)\ i++;\ if(isimulated_start(); /*initialize the number of active processes */ active_processes = smpi_process_count(); set_reqq_self(new std::vector); } static void action_finalize(const char *const *action) { /* Nothing to do */ } static void action_comm_size(const char *const *action) { communicator_size = parse_double(action[2]); log_timed_action (action, smpi_process()->simulated_elapsed()); } static void action_comm_split(const char *const *action) { log_timed_action (action, smpi_process()->simulated_elapsed()); } static void action_comm_dup(const char *const *action) { log_timed_action (action, smpi_process()->simulated_elapsed()); } static void action_compute(const char *const *action) { CHECK_ACTION_PARAMS(action, 1, 0) double clock = smpi_process()->simulated_elapsed(); double flops= parse_double(action[2]); int rank = smpi_process()->index(); TRACE_smpi_computing_in(rank, flops); smpi_execute_flops(flops); TRACE_smpi_computing_out(rank); log_timed_action (action, clock); } static void action_send(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int to = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process()->simulated_elapsed(); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process()->index(); int dst_traced = MPI_COMM_WORLD->group()->rank(to); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("send", dst_traced, size, encode_datatype(MPI_CURRENT_TYPE))); if (not TRACE_smpi_view_internals()) TRACE_smpi_send(rank, rank, dst_traced, 0, size*MPI_CURRENT_TYPE->size()); Request::send(nullptr, size, MPI_CURRENT_TYPE, to , 0, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action(action, clock); } static void action_Isend(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int to = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process()->simulated_elapsed(); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process()->index(); int dst_traced = MPI_COMM_WORLD->group()->rank(to); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Isend", dst_traced, size, encode_datatype(MPI_CURRENT_TYPE))); if (not TRACE_smpi_view_internals()) TRACE_smpi_send(rank, rank, dst_traced, 0, size*MPI_CURRENT_TYPE->size()); MPI_Request request = Request::isend(nullptr, size, MPI_CURRENT_TYPE, to, 0, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); get_reqq_self()->push_back(request); log_timed_action (action, clock); } static void action_recv(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int from = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process()->simulated_elapsed(); MPI_Status status; if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process()->index(); int src_traced = MPI_COMM_WORLD->group()->rank(from); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("recv", src_traced, size, encode_datatype(MPI_CURRENT_TYPE))); //unknown size from the receiver point of view if (size <= 0.0) { Request::probe(from, 0, MPI_COMM_WORLD, &status); size=status.count; } Request::recv(nullptr, size, MPI_CURRENT_TYPE, from, 0, MPI_COMM_WORLD, &status); TRACE_smpi_comm_out(rank); if (not TRACE_smpi_view_internals()) { TRACE_smpi_recv(src_traced, rank, 0); } log_timed_action (action, clock); } static void action_Irecv(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int from = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process()->simulated_elapsed(); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process()->index(); int src_traced = MPI_COMM_WORLD->group()->rank(from); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::Pt2PtTIData("Irecv", src_traced, size, encode_datatype(MPI_CURRENT_TYPE))); MPI_Status status; //unknow size from the receiver pov if (size <= 0.0) { Request::probe(from, 0, MPI_COMM_WORLD, &status); size = status.count; } MPI_Request request = Request::irecv(nullptr, size, MPI_CURRENT_TYPE, from, 0, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); get_reqq_self()->push_back(request); log_timed_action (action, clock); } static void action_test(const char* const* action) { CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process()->simulated_elapsed(); MPI_Status status; MPI_Request request = get_reqq_self()->back(); get_reqq_self()->pop_back(); //if request is null here, this may mean that a previous test has succeeded //Different times in traced application and replayed version may lead to this //In this case, ignore the extra calls. if(request!=nullptr){ int rank = smpi_process()->index(); TRACE_smpi_testing_in(rank); int flag = Request::test(&request, &status); XBT_DEBUG("MPI_Test result: %d", flag); /* push back request in vector to be caught by a subsequent wait. if the test did succeed, the request is now nullptr.*/ get_reqq_self()->push_back(request); TRACE_smpi_testing_out(rank); } log_timed_action (action, clock); } static void action_wait(const char *const *action){ CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process()->simulated_elapsed(); MPI_Status status; xbt_assert(get_reqq_self()->size(), "action wait not preceded by any irecv or isend: %s", xbt_str_join_array(action," ")); MPI_Request request = get_reqq_self()->back(); get_reqq_self()->pop_back(); if (request==nullptr){ /* Assume that the trace is well formed, meaning the comm might have been caught by a MPI_test. Then just return.*/ return; } int rank = request->comm() != MPI_COMM_NULL ? request->comm()->rank() : -1; MPI_Group group = request->comm()->group(); int src_traced = group->rank(request->src()); int dst_traced = group->rank(request->dst()); int is_wait_for_receive = (request->flags() & RECV); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("wait")); Request::wait(&request, &status); TRACE_smpi_comm_out(rank); if (is_wait_for_receive) TRACE_smpi_recv(src_traced, dst_traced, 0); log_timed_action (action, clock); } static void action_waitall(const char *const *action){ CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process()->simulated_elapsed(); const unsigned int count_requests = get_reqq_self()->size(); if (count_requests>0) { MPI_Status status[count_requests]; int rank_traced = smpi_process()->index(); TRACE_smpi_comm_in(rank_traced, __FUNCTION__, new simgrid::instr::Pt2PtTIData("waitAll", -1, count_requests, "")); int recvs_snd[count_requests]; int recvs_rcv[count_requests]; for (unsigned int i = 0; i < count_requests; i++) { const auto& req = (*get_reqq_self())[i]; if (req && (req->flags () & RECV)){ recvs_snd[i]=req->src(); recvs_rcv[i]=req->dst(); }else recvs_snd[i]=-100; } Request::waitall(count_requests, &(*get_reqq_self())[0], status); for (unsigned i = 0; i < count_requests; i++) { if (recvs_snd[i]!=-100) TRACE_smpi_recv(recvs_snd[i], recvs_rcv[i],0); } TRACE_smpi_comm_out(rank_traced); } log_timed_action (action, clock); } static void action_barrier(const char *const *action){ double clock = smpi_process()->simulated_elapsed(); int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::NoOpTIData("barrier")); Colls::barrier(MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_bcast(const char *const *action) { CHECK_ACTION_PARAMS(action, 1, 2) double size = parse_double(action[2]); double clock = smpi_process()->simulated_elapsed(); int root=0; /* Initialize MPI_CURRENT_TYPE in order to decrease the number of the checks */ MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; if(action[3]) { root= atoi(action[3]); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("bcast", MPI_COMM_WORLD->group()->index(root), -1.0, size, -1, encode_datatype(MPI_CURRENT_TYPE), "")); void *sendbuf = smpi_get_tmp_sendbuffer(size* MPI_CURRENT_TYPE->size()); Colls::bcast(sendbuf, size, MPI_CURRENT_TYPE, root, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_reduce(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 2) double comm_size = parse_double(action[2]); double comp_size = parse_double(action[3]); double clock = smpi_process()->simulated_elapsed(); int root=0; MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; if(action[4]) { root= atoi(action[4]); if(action[5]) MPI_CURRENT_TYPE=decode_datatype(action[5]); } int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("reduce", MPI_COMM_WORLD->group()->index(root), comp_size, comm_size, -1, encode_datatype(MPI_CURRENT_TYPE), "")); void *recvbuf = smpi_get_tmp_sendbuffer(comm_size* MPI_CURRENT_TYPE->size()); void *sendbuf = smpi_get_tmp_sendbuffer(comm_size* MPI_CURRENT_TYPE->size()); Colls::reduce(sendbuf, recvbuf, comm_size, MPI_CURRENT_TYPE, MPI_OP_NULL, root, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_allReduce(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) double comm_size = parse_double(action[2]); double comp_size = parse_double(action[3]); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; double clock = smpi_process()->simulated_elapsed(); int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("allReduce", -1, comp_size, comm_size, -1, encode_datatype(MPI_CURRENT_TYPE), "")); void *recvbuf = smpi_get_tmp_sendbuffer(comm_size* MPI_CURRENT_TYPE->size()); void *sendbuf = smpi_get_tmp_sendbuffer(comm_size* MPI_CURRENT_TYPE->size()); Colls::allreduce(sendbuf, recvbuf, comm_size, MPI_CURRENT_TYPE, MPI_OP_NULL, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_allToAll(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 2) //two mandatory (send and recv volumes) and two optional (corresponding datatypes) double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); int send_size = parse_double(action[2]); int recv_size = parse_double(action[3]); MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if(action[4] && action[5]) { MPI_CURRENT_TYPE=decode_datatype(action[4]); MPI_CURRENT_TYPE2=decode_datatype(action[5]); } else MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; void *send = smpi_get_tmp_sendbuffer(send_size*comm_size* MPI_CURRENT_TYPE->size()); void *recv = smpi_get_tmp_recvbuffer(recv_size*comm_size* MPI_CURRENT_TYPE2->size()); int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("allToAll", -1, -1.0, send_size, recv_size, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::alltoall(send, send_size, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_gather(const char *const *action) { /* The structure of the gather action for the rank 0 (total 4 processes) is the following: 0 gather 68 68 0 0 0 where: 1) 68 is the sendcounts 2) 68 is the recvcounts 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ CHECK_ACTION_PARAMS(action, 2, 3) double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); int send_size = parse_double(action[2]); int recv_size = parse_double(action[3]); MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if(action[4] && action[5]) { MPI_CURRENT_TYPE=decode_datatype(action[5]); MPI_CURRENT_TYPE2=decode_datatype(action[6]); } else { MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; } void *send = smpi_get_tmp_sendbuffer(send_size* MPI_CURRENT_TYPE->size()); void *recv = nullptr; int root=0; if(action[4]) root=atoi(action[4]); int rank = MPI_COMM_WORLD->rank(); if(rank==root) recv = smpi_get_tmp_recvbuffer(recv_size*comm_size* MPI_CURRENT_TYPE2->size()); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("gather", root, -1.0, send_size, recv_size, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::gather(send, send_size, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_comm_out(smpi_process()->index()); log_timed_action (action, clock); } static void action_scatter(const char* const* action) { /* The structure of the scatter action for the rank 0 (total 4 processes) is the following: 0 gather 68 68 0 0 0 where: 1) 68 is the sendcounts 2) 68 is the recvcounts 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ CHECK_ACTION_PARAMS(action, 2, 3) double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); int send_size = parse_double(action[2]); int recv_size = parse_double(action[3]); MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if (action[4] && action[5]) { MPI_CURRENT_TYPE = decode_datatype(action[5]); MPI_CURRENT_TYPE2 = decode_datatype(action[6]); } else { MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; } void* send = smpi_get_tmp_sendbuffer(send_size * MPI_CURRENT_TYPE->size()); void* recv = nullptr; int root = 0; if (action[4]) root = atoi(action[4]); int rank = MPI_COMM_WORLD->rank(); if (rank == root) recv = smpi_get_tmp_recvbuffer(recv_size * comm_size * MPI_CURRENT_TYPE2->size()); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("gather", root, -1.0, send_size, recv_size, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::scatter(send, send_size, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_comm_out(smpi_process()->index()); log_timed_action(action, clock); } static void action_gatherv(const char *const *action) { /* The structure of the gatherv action for the rank 0 (total 4 processes) is the following: 0 gather 68 68 10 10 10 0 0 0 where: 1) 68 is the sendcount 2) 68 10 10 10 is the recvcounts 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); CHECK_ACTION_PARAMS(action, comm_size+1, 2) int send_size = parse_double(action[2]); int disps[comm_size]; int recvcounts[comm_size]; int recv_sum=0; MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if(action[4+comm_size] && action[5+comm_size]) { MPI_CURRENT_TYPE=decode_datatype(action[4+comm_size]); MPI_CURRENT_TYPE2=decode_datatype(action[5+comm_size]); } else MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; void *send = smpi_get_tmp_sendbuffer(send_size* MPI_CURRENT_TYPE->size()); void *recv = nullptr; for(int i=0;irank(); if(rank==root) recv = smpi_get_tmp_recvbuffer(recv_sum* MPI_CURRENT_TYPE2->size()); std::vector* trace_recvcounts = new std::vector; for (int i = 0; i < comm_size; i++) // copy data to avoid bad free trace_recvcounts->push_back(recvcounts[i]); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "gatherV", root, send_size, nullptr, -1, trace_recvcounts, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::gatherv(send, send_size, MPI_CURRENT_TYPE, recv, recvcounts, disps, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_comm_out(smpi_process()->index()); log_timed_action (action, clock); } static void action_scatterv(const char* const* action) { /* The structure of the scatterv action for the rank 0 (total 4 processes) is the following: 0 gather 68 10 10 10 68 0 0 0 where: 1) 68 10 10 10 is the sendcounts 2) 68 is the recvcount 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); CHECK_ACTION_PARAMS(action, comm_size + 1, 2) int recv_size = parse_double(action[2 + comm_size]); int disps[comm_size]; int sendcounts[comm_size]; int send_sum = 0; MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if (action[4 + comm_size] && action[5 + comm_size]) { MPI_CURRENT_TYPE = decode_datatype(action[4 + comm_size]); MPI_CURRENT_TYPE2 = decode_datatype(action[5 + comm_size]); } else MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; void* send = nullptr; void* recv = smpi_get_tmp_recvbuffer(recv_size * MPI_CURRENT_TYPE->size()); for (int i = 0; i < comm_size; i++) { sendcounts[i] = atoi(action[i + 2]); send_sum += sendcounts[i]; disps[i] = 0; } int root = atoi(action[3 + comm_size]); int rank = MPI_COMM_WORLD->rank(); if (rank == root) send = smpi_get_tmp_sendbuffer(send_sum * MPI_CURRENT_TYPE2->size()); std::vector* trace_sendcounts = new std::vector; for (int i = 0; i < comm_size; i++) // copy data to avoid bad free trace_sendcounts->push_back(sendcounts[i]); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "gatherV", root, -1, trace_sendcounts, recv_size, nullptr, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::scatterv(send, sendcounts, disps, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_comm_out(smpi_process()->index()); log_timed_action(action, clock); } static void action_reducescatter(const char *const *action) { /* The structure of the reducescatter action for the rank 0 (total 4 processes) is the following: 0 reduceScatter 275427 275427 275427 204020 11346849 0 where: 1) The first four values after the name of the action declare the recvcounts array 2) The value 11346849 is the amount of instructions 3) The last value corresponds to the datatype, see decode_datatype(). */ double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); CHECK_ACTION_PARAMS(action, comm_size+1, 1) int comp_size = parse_double(action[2+comm_size]); int recvcounts[comm_size]; int rank = smpi_process()->index(); int size = 0; std::vector* trace_recvcounts = new std::vector; if(action[3+comm_size]) MPI_CURRENT_TYPE=decode_datatype(action[3+comm_size]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; for(int i=0;ipush_back(recvcounts[i]); size+=recvcounts[i]; } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData("reduceScatter", -1, 0, nullptr, -1, trace_recvcounts, std::to_string(comp_size), /* ugly hack to print comp_size */ encode_datatype(MPI_CURRENT_TYPE))); void *sendbuf = smpi_get_tmp_sendbuffer(size* MPI_CURRENT_TYPE->size()); void *recvbuf = smpi_get_tmp_recvbuffer(size* MPI_CURRENT_TYPE->size()); Colls::reduce_scatter(sendbuf, recvbuf, recvcounts, MPI_CURRENT_TYPE, MPI_OP_NULL, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_allgather(const char *const *action) { /* The structure of the allgather action for the rank 0 (total 4 processes) is the following: 0 allGather 275427 275427 where: 1) 275427 is the sendcount 2) 275427 is the recvcount 3) No more values mean that the datatype for sent and receive buffer is the default one, see decode_datatype(). */ double clock = smpi_process()->simulated_elapsed(); CHECK_ACTION_PARAMS(action, 2, 2) int sendcount=atoi(action[2]); int recvcount=atoi(action[3]); MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if(action[4] && action[5]) { MPI_CURRENT_TYPE = decode_datatype(action[4]); MPI_CURRENT_TYPE2 = decode_datatype(action[5]); } else MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; void *sendbuf = smpi_get_tmp_sendbuffer(sendcount* MPI_CURRENT_TYPE->size()); void *recvbuf = smpi_get_tmp_recvbuffer(recvcount* MPI_CURRENT_TYPE2->size()); int rank = smpi_process()->index(); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::CollTIData("allGather", -1, -1.0, sendcount, recvcount, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::allgather(sendbuf, sendcount, MPI_CURRENT_TYPE, recvbuf, recvcount, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_allgatherv(const char *const *action) { /* The structure of the allgatherv action for the rank 0 (total 4 processes) is the following: 0 allGatherV 275427 275427 275427 275427 204020 where: 1) 275427 is the sendcount 2) The next four elements declare the recvcounts array 3) No more values mean that the datatype for sent and receive buffer is the default one, see decode_datatype(). */ double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); CHECK_ACTION_PARAMS(action, comm_size+1, 2) int sendcount=atoi(action[2]); int recvcounts[comm_size]; int disps[comm_size]; int recv_sum=0; MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; if(action[3+comm_size] && action[4+comm_size]) { MPI_CURRENT_TYPE = decode_datatype(action[3+comm_size]); MPI_CURRENT_TYPE2 = decode_datatype(action[4+comm_size]); } else MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; void *sendbuf = smpi_get_tmp_sendbuffer(sendcount* MPI_CURRENT_TYPE->size()); for(int i=0;isize()); int rank = smpi_process()->index(); std::vector* trace_recvcounts = new std::vector; for (int i = 0; i < comm_size; i++) // copy data to avoid bad free trace_recvcounts->push_back(recvcounts[i]); TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "allGatherV", -1, sendcount, nullptr, -1, trace_recvcounts, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::allgatherv(sendbuf, sendcount, MPI_CURRENT_TYPE, recvbuf, recvcounts, disps, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } static void action_allToAllv(const char *const *action) { /* The structure of the allToAllV action for the rank 0 (total 4 processes) is the following: 0 allToAllV 100 1 7 10 12 100 1 70 10 5 where: 1) 100 is the size of the send buffer *sizeof(int), 2) 1 7 10 12 is the sendcounts array 3) 100*sizeof(int) is the size of the receiver buffer 4) 1 70 10 5 is the recvcounts array */ double clock = smpi_process()->simulated_elapsed(); int comm_size = MPI_COMM_WORLD->size(); CHECK_ACTION_PARAMS(action, 2*comm_size+2, 2) int send_size = 0; int recv_size = 0; int sendcounts[comm_size]; std::vector* trace_sendcounts = new std::vector; int recvcounts[comm_size]; std::vector* trace_recvcounts = new std::vector; int senddisps[comm_size]; int recvdisps[comm_size]; MPI_Datatype MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; int send_buf_size=parse_double(action[2]); int recv_buf_size=parse_double(action[3+comm_size]); if(action[4+2*comm_size] && action[5+2*comm_size]) { MPI_CURRENT_TYPE=decode_datatype(action[4+2*comm_size]); MPI_CURRENT_TYPE2=decode_datatype(action[5+2*comm_size]); } else MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; int rank = smpi_process()->index(); void *sendbuf = smpi_get_tmp_sendbuffer(send_buf_size* MPI_CURRENT_TYPE->size()); void *recvbuf = smpi_get_tmp_recvbuffer(recv_buf_size* MPI_CURRENT_TYPE2->size()); for(int i=0;ipush_back(sendcounts[i]); send_size += sendcounts[i]; recvcounts[i] = atoi(action[i+4+comm_size]); trace_recvcounts->push_back(recvcounts[i]); recv_size += recvcounts[i]; senddisps[i] = 0; recvdisps[i] = 0; } TRACE_smpi_comm_in(rank, __FUNCTION__, new simgrid::instr::VarCollTIData( "allToAllV", -1, send_size, trace_sendcounts, recv_size, trace_recvcounts, encode_datatype(MPI_CURRENT_TYPE), encode_datatype(MPI_CURRENT_TYPE2))); Colls::alltoallv(sendbuf, sendcounts, senddisps, MPI_CURRENT_TYPE,recvbuf, recvcounts, recvdisps, MPI_CURRENT_TYPE, MPI_COMM_WORLD); TRACE_smpi_comm_out(rank); log_timed_action (action, clock); } }} // namespace simgrid::smpi /** @brief Only initialize the replay, don't do it for real */ void smpi_replay_init(int* argc, char*** argv) { simgrid::smpi::Process::init(argc, argv); smpi_process()->mark_as_initialized(); smpi_process()->set_replaying(true); int rank = smpi_process()->index(); TRACE_smpi_init(rank); TRACE_smpi_computing_init(rank); TRACE_smpi_comm_in(rank, "smpi_replay_run_init", new simgrid::instr::NoOpTIData("init")); TRACE_smpi_comm_out(rank); xbt_replay_action_register("init", simgrid::smpi::action_init); xbt_replay_action_register("finalize", simgrid::smpi::action_finalize); xbt_replay_action_register("comm_size", simgrid::smpi::action_comm_size); xbt_replay_action_register("comm_split", simgrid::smpi::action_comm_split); xbt_replay_action_register("comm_dup", simgrid::smpi::action_comm_dup); xbt_replay_action_register("send", simgrid::smpi::action_send); xbt_replay_action_register("Isend", simgrid::smpi::action_Isend); xbt_replay_action_register("recv", simgrid::smpi::action_recv); xbt_replay_action_register("Irecv", simgrid::smpi::action_Irecv); xbt_replay_action_register("test", simgrid::smpi::action_test); xbt_replay_action_register("wait", simgrid::smpi::action_wait); xbt_replay_action_register("waitAll", simgrid::smpi::action_waitall); xbt_replay_action_register("barrier", simgrid::smpi::action_barrier); xbt_replay_action_register("bcast", simgrid::smpi::action_bcast); xbt_replay_action_register("reduce", simgrid::smpi::action_reduce); xbt_replay_action_register("allReduce", simgrid::smpi::action_allReduce); xbt_replay_action_register("allToAll", simgrid::smpi::action_allToAll); xbt_replay_action_register("allToAllV", simgrid::smpi::action_allToAllv); xbt_replay_action_register("gather", simgrid::smpi::action_gather); xbt_replay_action_register("scatter", simgrid::smpi::action_scatter); xbt_replay_action_register("gatherV", simgrid::smpi::action_gatherv); xbt_replay_action_register("scatterV", simgrid::smpi::action_scatterv); xbt_replay_action_register("allGather", simgrid::smpi::action_allgather); xbt_replay_action_register("allGatherV", simgrid::smpi::action_allgatherv); xbt_replay_action_register("reduceScatter", simgrid::smpi::action_reducescatter); xbt_replay_action_register("compute", simgrid::smpi::action_compute); //if we have a delayed start, sleep here. if(*argc>2){ double value = xbt_str_parse_double((*argv)[2], "%s is not a double"); XBT_VERB("Delayed start for instance - Sleeping for %f flops ",value ); smpi_execute_flops(value); } else { //UGLY: force a context switch to be sure that all MSG_processes begin initialization XBT_DEBUG("Force context switch by smpi_execute_flops - Sleeping for 0.0 flops "); smpi_execute_flops(0.0); } } /** @brief actually run the replay after initialization */ void smpi_replay_main(int* argc, char*** argv) { simgrid::xbt::replay_runner(*argc, *argv); /* and now, finalize everything */ /* One active process will stop. Decrease the counter*/ XBT_DEBUG("There are %zu elements in reqq[*]", get_reqq_self()->size()); if (not get_reqq_self()->empty()) { unsigned int count_requests=get_reqq_self()->size(); MPI_Request requests[count_requests]; MPI_Status status[count_requests]; unsigned int i=0; for (auto const& req : *get_reqq_self()) { requests[i] = req; i++; } simgrid::smpi::Request::waitall(count_requests, requests, status); } delete get_reqq_self(); active_processes--; if(active_processes==0){ /* Last process alive speaking: end the simulated timer */ XBT_INFO("Simulation time %f", smpi_process()->simulated_elapsed()); xbt_free(sendbuffer); xbt_free(recvbuffer); } TRACE_smpi_comm_in(smpi_process()->index(), "smpi_replay_run_finalize", new simgrid::instr::NoOpTIData("finalize")); smpi_process()->finalize(); TRACE_smpi_comm_out(smpi_process()->index()); TRACE_smpi_finalize(smpi_process()->index()); } /** @brief chain a replay initialization and a replay start */ void smpi_replay_run(int* argc, char*** argv) { smpi_replay_init(argc, argv); smpi_replay_main(argc, argv); } SimGrid-3.18/src/smpi/internals/smpi_process.cpp0000644000175000017500000001777413217757320022306 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_process.hpp" #include "mc/mc.h" #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_group.hpp" #include "src/mc/mc_replay.hpp" #include "src/msg/msg_private.hpp" #include "src/simix/smx_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_process, smpi, "Logging specific to SMPI (kernel)"); extern int* index_to_process_data; #define MAILBOX_NAME_MAXLEN (5 + sizeof(int) * 2 + 1) static char *get_mailbox_name(char *str, int index) { snprintf(str, MAILBOX_NAME_MAXLEN, "SMPI-%0*x", static_cast(sizeof(int) * 2), static_cast(index)); return str; } static char *get_mailbox_name_small(char *str, int index) { snprintf(str, MAILBOX_NAME_MAXLEN, "small%0*x", static_cast(sizeof(int) * 2), static_cast(index)); return str; } namespace simgrid{ namespace smpi{ Process::Process(int index, msg_bar_t finalization_barrier) : finalization_barrier_(finalization_barrier) { char name[MAILBOX_NAME_MAXLEN]; mailbox_ = simgrid::s4u::Mailbox::byName(get_mailbox_name(name, index)); mailbox_small_ = simgrid::s4u::Mailbox::byName(get_mailbox_name_small(name, index)); mailboxes_mutex_ = xbt_mutex_init(); timer_ = xbt_os_timer_new(); state_ = SMPI_UNINITIALIZED; if (MC_is_active()) MC_ignore_heap(timer_, xbt_os_timer_size()); #if HAVE_PAPI if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { // TODO: Implement host/process/thread based counters. This implementation // just always takes the values passed via "default", like this: // "default:COUNTER1:COUNTER2:COUNTER3;". auto it = units2papi_setup.find(papi_default_config_name); if (it != units2papi_setup.end()) { papi_event_set_ = it->second.event_set; papi_counter_data_ = it->second.counter_data; XBT_DEBUG("Setting PAPI set for process %i", i); } else { papi_event_set_ = PAPI_NULL; XBT_DEBUG("No PAPI set for process %i", i); } } #endif } void Process::set_data(int index, int* argc, char*** argv) { char* instance_id = (*argv)[1]; comm_world_ = smpi_deployment_comm_world(instance_id); msg_bar_t bar = smpi_deployment_finalization_barrier(instance_id); if (bar != nullptr) // don't overwrite the current one if the instance has none finalization_barrier_ = bar; instance_id_ = instance_id; index_ = index; static_cast(SIMIX_process_self()->userdata)->data = this; if (*argc > 3) { memmove(&(*argv)[0], &(*argv)[2], sizeof(char *) * (*argc - 2)); (*argv)[(*argc) - 1] = nullptr; (*argv)[(*argc) - 2] = nullptr; } (*argc)-=2; argc_ = argc; argv_ = argv; // set the process attached to the mailbox mailbox_small_->setReceiver(simgrid::s4u::Actor::self()); process_ = SIMIX_process_self(); XBT_DEBUG("<%d> New process in the game: %p", index_, SIMIX_process_self()); } /** @brief Prepares the current process for termination. */ void Process::finalize() { state_ = SMPI_FINALIZED; XBT_DEBUG("<%d> Process left the game", index_); // This leads to an explosion of the search graph which cannot be reduced: if(MC_is_active() || MC_record_replay_is_active()) return; // wait for all pending asynchronous comms to finish MSG_barrier_wait(finalization_barrier_); } /** @brief Check if a process is finalized */ int Process::finalized() { if (index_ != MPI_UNDEFINED) return (state_ == SMPI_FINALIZED); else return 0; } /** @brief Check if a process is initialized */ int Process::initialized() { if (index_to_process_data == nullptr){ return false; } else{ return ((index_ != MPI_UNDEFINED) && (state_ == SMPI_INITIALIZED)); } } /** @brief Mark a process as initialized (=MPI_Init called) */ void Process::mark_as_initialized() { if ((index_ != MPI_UNDEFINED) && (state_ != SMPI_FINALIZED)) state_ = SMPI_INITIALIZED; } void Process::set_replaying(bool value){ if ((index_ != MPI_UNDEFINED) && (state_ != SMPI_FINALIZED)) replaying_ = value; } bool Process::replaying(){ if (index_ != MPI_UNDEFINED) return replaying_; else return false; } void Process::set_user_data(void *data) { data_ = data; } void *Process::get_user_data() { return data_; } smx_actor_t Process::process(){ return process_; } /** * \brief Returns a structure that stores the location (filename + linenumber) of the last calls to MPI_* functions. * * \see smpi_trace_set_call_location */ smpi_trace_call_location_t* Process::call_location() { return &trace_call_loc_; } void Process::set_privatized_region(smpi_privatization_region_t region) { privatized_region_ = region; } smpi_privatization_region_t Process::privatized_region() { return privatized_region_; } int Process::index() { return index_; } MPI_Comm Process::comm_world() { return comm_world_==nullptr ? MPI_COMM_NULL : *comm_world_; } smx_mailbox_t Process::mailbox() { return mailbox_->getImpl(); } smx_mailbox_t Process::mailbox_small() { return mailbox_small_->getImpl(); } xbt_mutex_t Process::mailboxes_mutex() { return mailboxes_mutex_; } #if HAVE_PAPI int Process::papi_event_set() { return papi_event_set_; } papi_counter_t& smpi_process_papi_counters() { return papi_counter_data_; } #endif xbt_os_timer_t Process::timer() { return timer_; } void Process::simulated_start() { simulated_ = SIMIX_get_clock(); } double Process::simulated_elapsed() { return SIMIX_get_clock() - simulated_; } MPI_Comm Process::comm_self() { if(comm_self_==MPI_COMM_NULL){ MPI_Group group = new Group(1); comm_self_ = new Comm(group, nullptr); group->set_mapping(index_, 0); } return comm_self_; } MPI_Comm Process::comm_intra() { return comm_intra_; } void Process::set_comm_intra(MPI_Comm comm) { comm_intra_ = comm; } void Process::set_sampling(int s) { sampling_ = s; } int Process::sampling() { return sampling_; } msg_bar_t Process::finalization_barrier(){ return finalization_barrier_; } int Process::return_value(){ return return_value_; } void Process::set_return_value(int val){ return_value_=val; } void Process::init(int *argc, char ***argv){ if (smpi_process_count() == 0) { printf("SimGrid was not initialized properly before entering MPI_Init. Aborting, please check compilation process and use smpirun\n"); exit(1); } if (argc != nullptr && argv != nullptr) { smx_actor_t proc = SIMIX_process_self(); proc->context->set_cleanup(&MSG_process_cleanup_from_SIMIX); int index = proc->pid - 1; // The maestro process has always ID 0 but we don't need that process here if(index_to_process_data == nullptr){ index_to_process_data=static_cast(xbt_malloc(SIMIX_process_count()*sizeof(int))); } char* instance_id = (*argv)[1]; try { int rank = std::stoi(std::string((*argv)[2])); smpi_deployment_register_process(instance_id, rank, index); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Invalid rank: ") + (*argv)[2]); } // cheinrich: I'm not sure what the impact of the SMPI_switch_data_segment on this call is. I moved // this up here so that I can set the privatized region before the switch. Process* process = smpi_process_remote(index); if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ /* Now using segment index of the process */ index = proc->segment_index; process->set_privatized_region(smpi_init_global_memory_segment_process()); /* Done at the process's creation */ SMPI_switch_data_segment(index); } process->set_data(index, argc, argv); } xbt_assert(smpi_process(), "smpi_process() returned nullptr. You probably gave a nullptr parameter to MPI_Init. " "Although it's required by MPI-2, this is currently not supported by SMPI."); } } } SimGrid-3.18/src/smpi/internals/smpi_utils.cpp0000644000175000017500000000536113217757320021755 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_utils.hpp" #include "xbt/log.h" #include "xbt/sysdep.h" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_utils, smpi, "Logging specific to SMPI (utils)"); std::vector parse_factor(std::string smpi_coef_string) { std::vector smpi_factor; /** Setup the tokenizer that parses the string **/ typedef boost::tokenizer> Tokenizer; boost::char_separator sep(";"); boost::char_separator factor_separator(":"); Tokenizer tokens(smpi_coef_string, sep); /** * Iterate over patterns like A:B:C:D;E:F;G:H * These will be broken down into: * A --> B, C, D * E --> F * G --> H */ for (Tokenizer::iterator token_iter = tokens.begin(); token_iter != tokens.end(); ++token_iter) { XBT_DEBUG("token : %s", token_iter->c_str()); Tokenizer factor_values(*token_iter, factor_separator); s_smpi_factor_t fact; if (factor_values.begin() == factor_values.end()) { xbt_die("Malformed radical for smpi factor: '%s'", smpi_coef_string.c_str()); } unsigned int iteration = 0; for (Tokenizer::iterator factor_iter = factor_values.begin(); factor_iter != factor_values.end(); ++factor_iter) { iteration++; if (factor_iter == factor_values.begin()) { /* first element */ try { fact.factor = std::stoi(*factor_iter); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Invalid factor in chunk ") + std::to_string(smpi_factor.size() + 1) + ": " + *factor_iter); } } else { try { fact.values.push_back(std::stod(*factor_iter)); } catch (std::invalid_argument& ia) { throw std::invalid_argument(std::string("Invalid factor value ") + std::to_string(iteration) + " in chunk " + std::to_string(smpi_factor.size() + 1) + ": " + *factor_iter); } } } smpi_factor.push_back(fact); XBT_DEBUG("smpi_factor:\t%zu : %zu values, first: %f", fact.factor, smpi_factor.size(), fact.values[0]); } std::sort(smpi_factor.begin(), smpi_factor.end(), [](const s_smpi_factor_t &pa, const s_smpi_factor_t &pb) { return (pa.factor < pb.factor); }); for (auto const& fact : smpi_factor) { XBT_DEBUG("smpi_factor:\t%zu : %zu values, first: %f", fact.factor, smpi_factor.size() ,fact.values[0]); } smpi_factor.shrink_to_fit(); return smpi_factor; } SimGrid-3.18/src/smpi/internals/SmpiHost.cpp0000644000175000017500000001131413217757320021326 0ustar mquinsonmquinson/* Copyright (c) 2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "SmpiHost.hpp" #include "simgrid/s4u/VirtualMachine.hpp" #include "smpi_utils.hpp" #include #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_host, smpi, "Logging specific to SMPI (host)"); namespace simgrid { namespace smpi { simgrid::xbt::Extension SmpiHost::EXTENSION_ID; double SmpiHost::orecv(size_t size) { double current = orecv_parsed_values.empty() ? 0.0 : orecv_parsed_values.front().values[0] + orecv_parsed_values.front().values[1] * size; // Iterate over all the sections that were specified and find the right value. (fact.factor represents the interval // sizes; we want to find the section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto const& fact : orecv_parsed_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("or : %zu <= %zu return %.10f", size, fact.factor, current); return current; } else { // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current=fact.values[0]+fact.values[1]*size; } } XBT_DEBUG("smpi_or: %zu is larger than largest boundary, return %.10f", size, current); return current; } double SmpiHost::osend(size_t size) { double current = osend_parsed_values.empty() ? 0.0 : osend_parsed_values[0].values[0] + osend_parsed_values[0].values[1] * size; // Iterate over all the sections that were specified and find the right // value. (fact.factor represents the interval sizes; we want to find the // section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto const& fact : osend_parsed_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("os : %zu <= %zu return %.10f", size, fact.factor, current); return current; } else { // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current = fact.values[0] + fact.values[1] * size; } } XBT_DEBUG("Searching for smpi/os: %zu is larger than the largest boundary, return %.10f", size, current); return current; } double SmpiHost::oisend(size_t size) { double current = oisend_parsed_values.empty() ? 0.0 : oisend_parsed_values[0].values[0] + oisend_parsed_values[0].values[1] * size; // Iterate over all the sections that were specified and find the right value. (fact.factor represents the interval // sizes; we want to find the section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto const& fact : oisend_parsed_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("ois : %zu <= %zu return %.10f", size, fact.factor, current); return current; } else { // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current = fact.values[0] + fact.values[1] * size; } } XBT_DEBUG("Searching for smpi/ois: %zu is larger than the largest boundary, return %.10f", size, current); return current; } SmpiHost::SmpiHost(simgrid::s4u::Host *ptr) : host(ptr) { if (not SmpiHost::EXTENSION_ID.valid()) SmpiHost::EXTENSION_ID = simgrid::s4u::Host::extension_create(); const char* orecv_string = host->getProperty("smpi/or"); if (orecv_string != nullptr) { orecv_parsed_values = parse_factor(orecv_string); } else { orecv_parsed_values = parse_factor(xbt_cfg_get_string("smpi/or")); } const char* osend_string = host->getProperty("smpi/os"); if (osend_string != nullptr) { osend_parsed_values = parse_factor(osend_string); } else { osend_parsed_values = parse_factor(xbt_cfg_get_string("smpi/os")); } const char* oisend_string = host->getProperty("smpi/ois"); if (oisend_string != nullptr) { oisend_parsed_values = parse_factor(oisend_string); } else { oisend_parsed_values = parse_factor(xbt_cfg_get_string("smpi/ois")); } } SmpiHost::~SmpiHost()=default; } } SimGrid-3.18/src/smpi/internals/smpi_dvfs.cpp0000644000175000017500000000603513217757320021556 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include // FIXME: this plugin should be separated from the core #include "simgrid/s4u/Host.hpp" #include #include #include #include "src/internal_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_dvfs, smpi, "Logging specific to SMPI (experimental DVFS support)"); /** * \brief Return the speed of the processor (in flop/s) at a given pstate * * \param pstate_index pstate to test * \return Returns the processor speed associated with pstate_index */ double smpi_get_host_power_peak_at(int pstate_index) { return sg_host_self()->getPstateSpeed(pstate_index); } /** * \brief Return the current speed of the processor (in flop/s) * * \return Returns the current processor speed */ double smpi_get_host_current_power_peak() { return sg_host_self()->getSpeed(); } /** * \brief Return the number of pstates defined for the current host */ int smpi_get_host_nb_pstates() { return sg_host_get_nb_pstates(sg_host_self()); } /** * \brief Sets the pstate at which the processor should run * * \param pstate_index pstate to switch to */ void smpi_set_host_pstate(int pstate_index) { sg_host_set_pstate(sg_host_self(), pstate_index); } /** @brief Gets the pstate at which the processor currently running */ int smpi_get_host_pstate() { return sg_host_get_pstate(sg_host_self()); } /** * \brief Return the total energy consumed by a host (in Joules) * * \return Returns the consumed energy */ double smpi_get_host_consumed_energy() { return sg_host_get_consumed_energy(sg_host_self()); } #if SMPI_FORTRAN #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) typedef int integer; #else typedef long int integer; #endif typedef char *address; typedef float real; typedef double doublereal; struct complex { real r; real i; }; struct doublecomplex { doublereal r; doublereal i; }; extern "C" XBT_PUBLIC(doublereal) smpi_get_host_power_peak_at_(integer *pstate_index); doublereal smpi_get_host_power_peak_at_(integer *pstate_index) { return static_cast(smpi_get_host_power_peak_at((int)*pstate_index)); } extern "C" XBT_PUBLIC(doublereal) smpi_get_host_current_power_peak_(); doublereal smpi_get_host_current_power_peak_() { return smpi_get_host_current_power_peak(); } extern "C" XBT_PUBLIC(integer) smpi_get_host_nb_pstates_(); integer smpi_get_host_nb_pstates_() { return static_cast(smpi_get_host_nb_pstates()); } extern "C" XBT_PUBLIC(void) smpi_set_host_pstate_(integer *pstate_index); void smpi_set_host_pstate_(integer *pstate_index) { smpi_set_host_pstate(static_cast(*pstate_index)); } extern "C" XBT_PUBLIC(doublereal) smpi_get_host_consumed_energy_(); doublereal smpi_get_host_consumed_energy_() { return static_cast(smpi_get_host_consumed_energy()); } #endif SimGrid-3.18/src/smpi/internals/instr_smpi.cpp0000644000175000017500000002151513217757320021753 0ustar mquinsonmquinson/* Copyright (c) 2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include #include #include #include #include #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_smpi, instr, "Tracing SMPI"); static std::unordered_map*> keys; static const char* smpi_colors[] = { "recv", "1 0 0", "irecv", "1 0.52 0.52", "send", "0 0 1", "isend", "0.52 0.52 1", "sendrecv", "0 1 1", "wait", "1 1 0", "waitall", "0.78 0.78 0", "waitany", "0.78 0.78 0.58", "test", "0.52 0.52 0", "allgather", "1 0 0", "allgatherv", "1 0.52 0.52", "allreduce", "1 0 1", "alltoall", "0.52 0 1", "alltoallv", "0.78 0.52 1", "barrier", "0 0.78 0.78", "bcast", "0 0.78 0.39", "gather", "1 1 0", "gatherv", "1 1 0.52", "reduce", "0 1 0", "reducescatter", "0.52 1 0.52", "scan", "1 0.58 0.23", "exscan", "1 0.54 0.25", "scatterv", "0.52 0 0.52", "scatter", "1 0.74 0.54", "computing", "0 1 1", "sleeping", "0 0.5 0.5", "init", "0 1 0", "finalize", "0 1 0", "put", "0.3 1 0", "get", "0 1 0.3", "accumulate", "1 0.3 0", "rput", "0.3 1 0", "rget", "0 1 0.3", "raccumulate", "1 0.3 0", "compare_and_swap", "0.3 1 0", "get_accumulate", "0 1 0.3", "rget_accumulate", "1 0.3 0", "win_fence", "1 0 0.3", "win_post", "1 0 0.8", "win_wait", "1 0.8 0", "win_start", "0.8 0 1", "win_complete", "0.8 1 0", "win_lock", "1 0 0.3", "win_unlock", "1 0 0.3", "win_lock_all", "1 0 0.8", "win_unlock_all", "1 0.8 0", "win_flush", "1 0 0.3", "win_flush_local", "1 0 0.8", "win_flush_all", "1 0.8 0", "win_flush_local_all", "1 0 0.3", "" , "" }; static const char* instr_find_color(const char* state) { std::string target = std::string(state); boost::algorithm::to_lower(target); const char* ret = nullptr; unsigned int i = 0; const char* current = smpi_colors[i]; while (current != nullptr) { if (target == current // exact match || strstr(target.c_str(), current) != 0) { // as substring ret = smpi_colors[i + 1]; break; } i+=2; current = smpi_colors[i]; } return ret; } XBT_PRIVATE container_t smpi_container(int rank) { return simgrid::instr::Container::byName(std::string("rank-") + std::to_string(rank)); } static std::string TRACE_smpi_put_key(int src, int dst, int tag, int send) { // get the deque for src#dst std::string aux = std::to_string(src) + "#" + std::to_string(dst) + "#" + std::to_string(tag) + "#" + std::to_string(send); auto it = keys.find(aux); std::deque* d; if (it == keys.end()) { d = new std::deque; keys[aux] = d; } else d = it->second; //generate the key static unsigned long long counter = 0; counter++; std::string key = std::to_string(src) + "_" + std::to_string(dst) + "_" + std::to_string(tag) + "_" + std::to_string(counter); //push it d->push_back(key); return key; } static std::string TRACE_smpi_get_key(int src, int dst, int tag, int send) { std::string key; std::string aux = std::to_string(src) + "#" + std::to_string(dst) + "#" + std::to_string(tag) + "#" + std::to_string(send == 1 ? 0 : 1); auto it = keys.find(aux); if (it == keys.end()) { // first posted key = TRACE_smpi_put_key(src, dst, tag, send); } else { key = it->second->front(); it->second->pop_front(); } return key; } static std::unordered_map process_category; void TRACE_internal_smpi_set_category (const char *category) { if (not TRACE_smpi_is_enabled()) return; //declare category TRACE_category (category); if (category != nullptr) process_category[SIMIX_process_self()] = category; } const char *TRACE_internal_smpi_get_category () { if (not TRACE_smpi_is_enabled()) return nullptr; auto it = process_category.find(SIMIX_process_self()); return (it == process_category.end()) ? nullptr : it->second.c_str(); } void TRACE_smpi_alloc() { // for symmetry } void TRACE_smpi_release() { for (auto const& elm : keys) delete elm.second; } void TRACE_smpi_init(int rank) { if (not TRACE_smpi_is_enabled()) return; std::string str = std::string("rank-") + std::to_string(rank); container_t father; if (TRACE_smpi_is_grouped()){ father = simgrid::instr::Container::byNameOrNull(sg_host_self_get_name()); }else{ father = simgrid::instr::Container::getRoot(); } xbt_assert(father != nullptr, "Could not find a parent for mpi rank %s at function %s", str.c_str(), __FUNCTION__); #if HAVE_PAPI container_t container = #endif new simgrid::instr::Container(str, "MPI", father); #if HAVE_PAPI papi_counter_t counters = smpi_process()->papi_counters(); for (auto const& it : counters) { /** * Check whether this variable already exists or not. Otherwise, it will be created * multiple times but only the last one would be used... */ if (s_type::getOrNull(it.first.c_str(), container->type_) == nullptr) { Type::variableNew(it.first.c_str(), "", container->type_); } } #endif } void TRACE_smpi_finalize(int rank) { if (not TRACE_smpi_is_enabled()) return; container_t container = smpi_container(rank); container->removeFromParent(); delete container; } void TRACE_smpi_computing_init(int rank) { //first use, initialize the color in the trace if (TRACE_smpi_is_enabled() && TRACE_smpi_is_computing()) smpi_container(rank)->getState("MPI_STATE")->addEntityValue("computing", instr_find_color("computing")); } void TRACE_smpi_computing_in(int rank, double amount) { if (TRACE_smpi_is_enabled() && TRACE_smpi_is_computing()) smpi_container(rank) ->getState("MPI_STATE") ->pushEvent("computing", new simgrid::instr::CpuTIData("compute", amount)); } void TRACE_smpi_computing_out(int rank) { if (TRACE_smpi_is_enabled() && TRACE_smpi_is_computing()) smpi_container(rank)->getState("MPI_STATE")->popEvent(); } void TRACE_smpi_sleeping_init(int rank) { //first use, initialize the color in the trace if (TRACE_smpi_is_enabled() && TRACE_smpi_is_sleeping()) smpi_container(rank)->getState("MPI_STATE")->addEntityValue("sleeping", instr_find_color("sleeping")); } void TRACE_smpi_sleeping_in(int rank, double duration) { if (TRACE_smpi_is_enabled() && TRACE_smpi_is_sleeping()) smpi_container(rank) ->getState("MPI_STATE") ->pushEvent("sleeping", new simgrid::instr::CpuTIData("sleep", duration)); } void TRACE_smpi_sleeping_out(int rank) { if (TRACE_smpi_is_enabled() && not TRACE_smpi_is_sleeping()) smpi_container(rank)->getState("MPI_STATE")->popEvent(); } void TRACE_smpi_testing_in(int rank) { //do not forget to set the color first, otherwise this will explode if (not TRACE_smpi_is_enabled()) return; simgrid::instr::StateType* state = smpi_container(rank)->getState("MPI_STATE"); state->addEntityValue("test"); state->pushEvent("test", new simgrid::instr::NoOpTIData("test")); } void TRACE_smpi_testing_out(int rank) { if (TRACE_smpi_is_enabled()) smpi_container(rank)->getState("MPI_STATE")->popEvent(); } void TRACE_smpi_comm_in(int rank, const char* operation, simgrid::instr::TIData* extra) { if (not TRACE_smpi_is_enabled()) { delete extra; return; } simgrid::instr::StateType* state = smpi_container(rank)->getState("MPI_STATE"); state->addEntityValue(operation, instr_find_color(operation)); state->pushEvent(operation, extra); } void TRACE_smpi_comm_out(int rank) { if (TRACE_smpi_is_enabled()) smpi_container(rank)->getState("MPI_STATE")->popEvent(); } void TRACE_smpi_send(int rank, int src, int dst, int tag, int size) { if (not TRACE_smpi_is_enabled()) return; std::string key = TRACE_smpi_get_key(src, dst, tag, 1); XBT_DEBUG("Send tracing from %d to %d, tag %d, with key %s", src, dst, tag, key.c_str()); simgrid::instr::Container::getRoot()->getLink("MPI_LINK")->startEvent(smpi_container(rank), "PTP", key, size); } void TRACE_smpi_recv(int src, int dst, int tag) { if (not TRACE_smpi_is_enabled()) return; std::string key = TRACE_smpi_get_key(src, dst, tag, 0); XBT_DEBUG("Recv tracing from %d to %d, tag %d, with key %s", src, dst, tag, key.c_str()); simgrid::instr::Container::getRoot()->getLink("MPI_LINK")->endEvent(smpi_container(dst), "PTP", key); } SimGrid-3.18/src/smpi/smpirun.in0000755000175000017500000003666613217757325017133 0ustar mquinsonmquinson#! /bin/sh # Copyright (c) 2007-2016, The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. @CMAKE_SMPI_COMMAND@ SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" DEFAULT_LOOPBACK_BANDWIDTH="498000000Bps" DEFAULT_LOOPBACK_LATENCY="0.000004s" DEFAULT_NETWORK_BANDWIDTH="$((26 * 1024 * 1024))Bps" DEFAULT_NETWORK_LATENCY="0.000005s" DEFAULT_NUMPROCS="4" DEFAULT_SPEED="100flops" LOOPBACK_BANDWIDTH="${DEFAULT_LOOPBACK_BANDWIDTH}" LOOPBACK_LATENCY="${DEFAULT_LOOPBACK_LATENCY}" NETWORK_BANDWIDTH="${DEFAULT_NETWORK_BANDWIDTH}" NETWORK_LATENCY="${DEFAULT_NETWORK_LATENCY}" SPEED="${DEFAULT_SPEED}" PRIVATIZE="--cfg=smpi/privatization:@HAVE_PRIVATIZATION@" SIMOPTS="--cfg=surf/precision:1e-9 --cfg=network/model:SMPI" #usage to print the way this script should be called usage () { cat < -hostfile program [program-options] Options: -keep-temps # don't remove the generated files after execution -wrapper # use command to run the program (e.g. "valgrind" or "gdb --args") -map # display the machine on which each process rank is mapped -np # use that amount of processes from the hostfile. # By default, all processes of the hostfile are used. -no-privatize # Disable the globals privatization, that is activated by default -trace-ti # activate time independant tracing (for replay, default in smpi_simgrid.txt) -trace # activate tracing (Paje, default in smpi_simgrid.trace) -trace-comment # put a comment on the top of the trace file -trace-comment-file # put file contents on the top of the trace file as comment -trace-grouped # group MPI processes by location -trace-resource # trace resource utilization -trace-file # name of the tracefile (simgrid_smpi.trace) -ext # additional parameter (reserved) -version # Displays the SimGrid version (human readable) -git-version # Displays the git hash of SimGrid or (deprecated usage): $0 [-keep-temps] [-np ] [-bandwidth ] [-latency ] program [program-options] EOF } #check if we have at least one parameter if [ $# -eq 0 ] then usage exit fi EXTOPT="" WRAPPER="" HOSTFILE="" HOSTFILETMP=0 MAPOPT=0 REPLAY=0 unset pid trapped_signals="HUP INT QUIT ILL ABRT SEGV FPE ALRM TERM USR1 USR2 BUS" die () { printf '[%s] ** error: %s. Aborting.\n' "$(basename $0)" "$*" >&2 exit 1 } smpirun_cleanup() { if [ -z "${KEEP}" ] ; then if [ -z "${PLATFORM}" -a -n "$PLATFORMTMP" ]; then rm -f ${PLATFORMTMP} PLATFORMTMP="" fi if [ ${HOSTFILETMP} = 1 -a -n "$HOSTFILE" ] ; then rm -f ${HOSTFILE} HOSTFILE="" fi if [ ${UNROLLEDHOSTFILETMP} = 1 -a -n "$UNROLLEDHOSTFILE" ] ; then rm -f ${UNROLLEDHOSTFILE} UNROLLEDHOSTFILE="" fi if [ -n ${APPLICATIONTMP} ]; then rm -f ${APPLICATIONTMP} APPLICATIONTMP="" fi fi } smpirun_trap() { local sig sig="$1" # Cleanup and kill the child process: smpirun_cleanup if ! [ -z "$pid" ]; then kill -TERM $pid fi unset pid # Raise the same signal again (remove the traps first): trap - $trapped_signals kill -$sig $$ # This should never happen: kill -ABRT $$ kill -TERM $$ } for s in $trapped_signals; do trap "smpirun_trap $s" $s done while true; do case "$1" in "-np" | "-n") NUMPROCS="$2" shift 2 ;; "-bandwidth") NETWORK_BANDWIDTH="$2" shift 2 ;; "-latency") NETWORK_LATENCY="$2" shift 2 ;; "-platform") PLATFORM="$2" if [ ! -f "${PLATFORM}" ]; then die "the file '${PLATFORM}' does not exist" fi shift 2 ;; "-hostfile" | "-machinefile") HOSTFILE="$2" if [ ! -f "${HOSTFILE}" ]; then die "the file '${HOSTFILE}' does not exist" fi shift 2 ;; "-ext") EXTOPT="$2" shift 2 ;; "-no-privatize") PRIVATIZE="" shift 1 ;; "-map") MAPOPT=1 shift 1 ;; "-trace") TRACE_ACTIVE="true" shift 1 ;; "-trace-ti") TRACE_ACTIVE="true" TRACE_TI_ACTIVE="true" shift 1 ;; "-trace-comment") TRACE_COMMENT="$2" shift 2 ;; "-trace-comment-file") TRACE_COMMENT_FILE="$2" shift 2 ;; "-trace-file") TRACE_FILENAME="$2" shift 2 ;; "-trace-grouped") TRACE_GROUPED="true" shift 1 ;; "-trace-resource") TRACE_RESOURCE="true" shift 1 ;; "-keep-temps") KEEP="true" SIMOPTS="$SIMOPTS --cfg=smpi/keep-temps:yes" shift 1 ;; "-wrapper") WRAPPER="$2" shift 2 ;; "-help" | "--help" | "-h") usage exit 0 ;; "-version" | "--version" | "-v") printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; "--cfg="*|"--log="*) for OPT in ${1#*=} do SIMOPTS="$SIMOPTS ${1%%=*}=$OPT" done shift 1 ;; "-foreground") # Nothing to do, compatibility. shift 1 ;; *) break ;; esac done # check if we still have at least one parameter beyond options if [ $# -eq 0 ] then echo "Error: no program to execute!" usage exit fi EXEC="$1" shift # steel --cfg and --logs options while [ $# -gt 0 ]; do case "$1" in "--cfg="*|"--log="*) for OPT in ${1#*=} do SIMOPTS="$SIMOPTS ${1%%=*}=$OPT" done shift 1 ;; *) PROC_ARGS="${PROC_ARGS:+$PROC_ARGS }$1" shift ;; esac done if [ -z "${HOSTFILE}" ] && [ -z "${PLATFORM}" ] ; then echo "No hostfile nor platform specified." usage exit 1 fi if [ -z "${HOSTFILE}" ] ; then HOSTFILETMP=1 HOSTFILE="$(mktemp smpitmp-hostfXXXXXX)" perl -ne 'print "$1\n" if /.*.*/' ${PLATFORM} > ${HOSTFILE} perl -ne 'if (/.*> ${HOSTFILE} fi UNROLLEDHOSTFILETMP=0 #parse if our lines are terminated by :num_process multiple_processes=$(grep -c ":" $HOSTFILE) if [ "${multiple_processes}" -gt 0 ] ; then UNROLLEDHOSTFILETMP=1 UNROLLEDHOSTFILE="$(mktemp smpitmp-hostfXXXXXX)" perl -ne ' do{ for ( 1 .. $2 ) { print "$1\n" } } if /(.*?):(\d+).*/' ${HOSTFILE} > ${UNROLLEDHOSTFILE} if [ ${HOSTFILETMP} = 1 ] ; then rm ${HOSTFILE} HOSTFILETMP=0 fi HOSTFILE=$UNROLLEDHOSTFILE fi # Don't use wc -l to compute it to avoid issues with trailing \n at EOF hostfile_procs=$(grep -c "[a-zA-Z0-9]" $HOSTFILE) if [ ${hostfile_procs} = 0 ] ; then die "the hostfile '${HOSTFILE}' is empty" fi if [ -z "${NUMPROCS}" ] ; then # Use the amount of processes in the hostfile as default value for the -np parameter NUMPROCS=$hostfile_procs fi if [ ${NUMPROCS} -gt ${hostfile_procs} ] ; then echo "You requested to use ${NUMPROCS} ranks, but there is only ${hostfile_procs} processes in your hostfile..." >&2 fi ##-------------------------------- DEFAULT or SPECIFIED PLATFORM -------------------------------------- if [ -z "${PLATFORM}" ]; then PLATFORMTMP="$(mktemp smpitmp-platfXXXXXX)" cat > ${PLATFORMTMP} < PLATFORMHEAD i=${NUMPROCS} while [ $i -gt 0 ]; do echo " " >> ${PLATFORMTMP} echo " " >> ${PLATFORMTMP} echo " " >> ${PLATFORMTMP} i=$((i - 1)) done i=${NUMPROCS} while [ $i -gt 0 ]; do j=${NUMPROCS} while [ $j -gt 0 ]; do if [ $i -eq $j ]; then echo " " >> ${PLATFORMTMP} else echo " " >> ${PLATFORMTMP} fi j=$((j - 1)) done i=$((i - 1)) done cat >> ${PLATFORMTMP} < PLATFORMFOOT else PLATFORMTMP=${PLATFORM} fi ##-------------------------------- end DEFAULT or SPECIFIED PLATFORM -------------------------------------- ##-------------------------------- DEFAULT APPLICATION -------------------------------------- APPLICATIONTMP="$(mktemp smpitmp-appXXXXXX)" #APPLICATIONTMP="app.xml" cat > ${APPLICATIONTMP} < APPLICATIONHEAD ##---- cache hostnames of hostfile--------------- if [ -n "${HOSTFILE}" ] && [ -f ${HOSTFILE} ]; then hostnames=$(cat ${HOSTFILE} | tr '\n\r' ' ') fi if [ "${EXTOPT}" = "smpi_replay" ]; then APP_TRACES=$PROC_ARGS if [ -n "${APP_TRACES}" ] && [ -f "${APP_TRACES}" ]; then hosttraces=$(cat ${APP_TRACES} | tr '\n\r' ' ' ) NUMTRACES=$(cat ${APP_TRACES} | wc -l) REPLAY=1 else printf "File not found: %s\n" "${APP_TRACES:-\${APP_TRACES\}}" >&2 exit 1 fi fi ##---------------------------------------------------------- ## generate application.xml with hostnames from hostfile: ## the name of host_i (1<=i<=p, where -np p) is the line i in hostfile (where -hostfile hostfile), or "host$i" if ## hostfile has less than i lines. ##---------------------------------------------------------- HAVE_SEQ="$(which seq 2>/dev/null)" if [ -n "${HAVE_SEQ}" ]; then SEQ=$(${HAVE_SEQ} 0 $(( NUMPROCS - 1))) else cnt=0 while [ $cnt -lt ${NUMPROCS} ] ; do SEQ="$SEQ $cnt" cnt=$((cnt + 1)); done fi set -- $hostnames ##---- generate tags------------------------------ #prepare arguments at once for ARG in $PROC_ARGS; do XML_ARGS="${XML_ARGS}"" " done for i in ${SEQ} do j=$(( i % hostfile_procs + 1 )) host=$(eval "echo \${$j}") ##---- optional display of ranks to actor mapping if [ ${MAPOPT} = 1 ]; then echo "[rank $i] -> $host" fi echo " " >> ${APPLICATIONTMP} if [ ${REPLAY} = 1 ]; then if [ ${NUMTRACES} -gt 1 ]; then echo " " >> ${APPLICATIONTMP} else echo " " >> ${APPLICATIONTMP} fi else echo ${XML_ARGS} >> ${APPLICATIONTMP} fi echo " " >> ${APPLICATIONTMP} done cat >> ${APPLICATIONTMP} < APPLICATIONFOOT ##-------------------------------- end DEFAULT APPLICATION -------------------------------------- ##---------------------- SMPI TRACING OPTIONS --------------------------------- if [ -n "${TRACE_ACTIVE}" ]; then #define trace filename if [ -n "${TRACE_TI_ACTIVE}" ]; then if [ -z "${TRACE_FILENAME}" ]; then TRACE_FILENAME="smpi_simgrid.txt" fi TRACEOPTIONS="--cfg=tracing:yes --cfg=tracing/filename:${TRACE_FILENAME} --cfg=tracing/smpi:yes --cfg=tracing/smpi/format:TI --cfg=tracing/smpi/computing:yes" else if [ -z "${TRACE_FILENAME}" ]; then TRACE_FILENAME="smpi_simgrid.trace" fi TRACEOPTIONS="--cfg=tracing:yes --cfg=tracing/filename:${TRACE_FILENAME} --cfg=tracing/smpi:yes" fi if [ -n "${TRACE_COMMENT}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/comment:${TRACE_COMMENT}" fi if [ -n "${TRACE_COMMENT_FILE}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/comment-file:${TRACE_COMMENT_FILE}" fi if [ -n "${TRACE_GROUPED}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/smpi/group:yes" fi if [ -n "${TRACE_RESOURCE}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/categorized:yes --cfg=tracing/uncategorized:yes" fi fi ##---------------------- end SMPI TRACING OPTIONS --------------------------------- export SMPI_GLOBAL_SIZE=${NUMPROCS} if [ -n "${KEEP}" ] ; then echo ${EXEC} ${PRIVATIZE} ${TRACEOPTIONS} ${SIMOPTS} ${PLATFORMTMP} ${APPLICATIONTMP} if [ ${HOSTFILETMP} = 1 ] ; then echo "Generated hostfile ${HOSTFILE} kept." fi if [ ${UNROLLEDHOSTFILETMP} = 1 ] ; then echo "Generated unrolled hostfile ${UNROLLEDHOSTFILE} kept." fi fi # Execute the process # # The shell still need to be alive for the duration in order to do some cleanup after the process. # # We are going through great lengths in order to both keep stdin and be able to handle signals: # # * The job is launched in the background in order to be able to handle signals. # # * The FD 3 is used to temporarily store FD 1. This is because the shell connects FD 1 to /dev/null when the command # is launched in the background: this can be overriden in bash but not in standard bourne shell. exec 3<&0 ${WRAPPER} "@SMPIMAIN@" ${EXEC} ${TRACEOPTIONS} ${SIMOPTS} ${PLATFORMTMP} ${APPLICATIONTMP} <&3 3>&- & pid=$! exec 3>&- wait $pid status=$? pid="" # Keep temporary files on failures to help debugging # if [ ${status} -ne 0 ] ; then if [ -z "${KEEP}" ]; then echo ${EXEC} ${PRIVATIZE} ${TRACEOPTIONS} ${SIMOPTS} ${PLATFORMTMP} ${APPLICATIONTMP} if [ ${HOSTFILETMP} = 1 ] ; then echo "Generated hostfile ${HOSTFILE} kept." fi if [ ${UNROLLEDHOSTFILETMP} = 1 ] ; then echo "Generated unrolled hostfile ${UNROLLEDHOSTFILE} kept." fi fi echo "Execution failed with code ${status}." KEEP=true fi smpirun_cleanup exit $status SimGrid-3.18/src/smpi/include/0000755000175000017500000000000013217757321016501 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/include/smpi_request.hpp0000644000175000017500000001067013217757320021735 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_REQUEST_HPP_INCLUDED #define SMPI_REQUEST_HPP_INCLUDED #include "smpi/smpi.h" #include "smpi_f2c.hpp" namespace simgrid{ namespace smpi{ class Request : public F2C { private : void *buf_; /* in the case of non-contiguous memory the user address should be keep * to unserialize the data inside the user memory*/ void *old_buf_; /* this let us know how to unserialize at the end of * the communication*/ MPI_Datatype old_type_; size_t size_; int src_; int dst_; int tag_; //to handle cases where we have an unknown sender //We can't override src, tag, and size, because the request may be reused later int real_src_; int real_tag_; int truncated_; size_t real_size_; MPI_Comm comm_; smx_activity_t action_; unsigned flags_; int detached_; MPI_Request detached_sender_; int refcount_; MPI_Op op_; public: Request()=default; Request(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, unsigned flags); MPI_Comm comm(); size_t size(); size_t real_size(); int src(); int dst(); int tag(); int flags(); int detached(); void print_request(const char *message); void start(); static void finish_wait(MPI_Request* request, MPI_Status * status); static void unref(MPI_Request* request); static void wait(MPI_Request* req, MPI_Status * status); static MPI_Request send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static MPI_Request isend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static MPI_Request ssend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static MPI_Request rma_send_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm,MPI_Op op); static MPI_Request recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); static MPI_Request rma_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm,MPI_Op op); static MPI_Request irecv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); static MPI_Request isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static MPI_Request issend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static MPI_Request irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); static void recv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status); static void send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static void ssend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); static void sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype,int dst, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status); static void startall(int count, MPI_Request * requests); static int test(MPI_Request * request,MPI_Status * status); static int testsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]); static int testany(int count, MPI_Request requests[], int *index, MPI_Status * status); static int testall(int count, MPI_Request requests[], MPI_Status status[]); static void probe(int source, int tag, MPI_Comm comm, MPI_Status* status); static void iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status); static int waitany(int count, MPI_Request requests[], MPI_Status * status); static int waitall(int count, MPI_Request requests[], MPI_Status status[]); static int waitsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]); static int match_send(void* a, void* b, simgrid::kernel::activity::CommImpl* ignored); static int match_recv(void* a, void* b, simgrid::kernel::activity::CommImpl* ignored); int add_f() override; static void free_f(int id); static Request* f2c(int); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_datatype_derived.hpp0000644000175000017500000000550413217757320023562 0ustar mquinsonmquinson/* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_DATATYPE_DERIVED_HPP #define SMPI_DATATYPE_DERIVED_HPP #include "smpi_datatype.hpp" namespace simgrid{ namespace smpi{ class Type_Contiguous: public Datatype { int block_count_; MPI_Datatype old_type_; public: Type_Contiguous(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, MPI_Datatype old_type); ~Type_Contiguous(); void serialize(void* noncontiguous, void* contiguous, int count); void unserialize(void* contiguous_vector, void* noncontiguous_vector, int count, MPI_Op op); }; class Type_Hvector: public Datatype{ int block_count_; int block_length_; MPI_Aint block_stride_; MPI_Datatype old_type_; public: Type_Hvector(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, int block_length, MPI_Aint block_stride, MPI_Datatype old_type); ~Type_Hvector(); void serialize(void* noncontiguous, void* contiguous, int count); void unserialize(void* contiguous_vector, void* noncontiguous_vector, int count, MPI_Op op); }; class Type_Vector : public Type_Hvector { public: Type_Vector(int size, MPI_Aint lb, MPI_Aint ub, int flags, int count, int blocklen, int stride, MPI_Datatype old_type); }; class Type_Hindexed: public Datatype{ int block_count_; int* block_lengths_; MPI_Aint* block_indices_; MPI_Datatype old_type_; public: Type_Hindexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, int* block_lengths, MPI_Aint* block_indices, MPI_Datatype old_type); Type_Hindexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, int* block_lengths, int* block_indices, MPI_Datatype old_type, MPI_Aint factor); ~Type_Hindexed(); void serialize(void* noncontiguous, void* contiguous, int count); void unserialize(void* contiguous_vector, void* noncontiguous_vector, int count, MPI_Op op); }; class Type_Indexed : public Type_Hindexed { public: Type_Indexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, int* block_lengths, int* block_indices, MPI_Datatype old_type); }; class Type_Struct: public Datatype{ int block_count_; int* block_lengths_; MPI_Aint* block_indices_; MPI_Datatype* old_types_; public: Type_Struct(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, int* block_lengths, MPI_Aint* block_indices, MPI_Datatype* old_types); ~Type_Struct(); void serialize(void* noncontiguous, void* contiguous, int count); void unserialize(void* contiguous_vector, void* noncontiguous_vector, int count, MPI_Op op); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_datatype.hpp0000644000175000017500000001107213217757320022055 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_DATATYPE_HPP #define SMPI_DATATYPE_HPP #include "smpi_f2c.hpp" #include "smpi_keyvals.hpp" #define DT_FLAG_DESTROYED 0x0001 /**< user destroyed but some other layers still have a reference */ #define DT_FLAG_COMMITED 0x0002 /**< ready to be used for a send/recv operation */ #define DT_FLAG_CONTIGUOUS 0x0004 /**< contiguous datatype */ #define DT_FLAG_OVERLAP 0x0008 /**< datatype is unpropper for a recv operation */ #define DT_FLAG_USER_LB 0x0010 /**< has a user defined LB */ #define DT_FLAG_USER_UB 0x0020 /**< has a user defined UB */ #define DT_FLAG_PREDEFINED 0x0040 /**< cannot be removed: initial and predefined datatypes */ #define DT_FLAG_NO_GAPS 0x0080 /**< no gaps around the datatype */ #define DT_FLAG_DATA 0x0100 /**< data or control structure */ #define DT_FLAG_ONE_SIDED 0x0200 /**< datatype can be used for one sided operations */ #define DT_FLAG_UNAVAILABLE 0x0400 /**< datatypes unavailable on the build (OS or compiler dependant) */ #define DT_FLAG_DERIVED 0x0800 /**< is the datatype derived ? */ /* * We should make the difference here between the predefined contiguous and non contiguous * datatypes. The DT_FLAG_BASIC is held by all predefined contiguous datatypes. */ #define DT_FLAG_BASIC (DT_FLAG_PREDEFINED | DT_FLAG_CONTIGUOUS | DT_FLAG_NO_GAPS | DT_FLAG_DATA | DT_FLAG_COMMITED) extern const MPI_Datatype MPI_PTR; //The following are datatypes for the MPI functions MPI_MAXLOC and MPI_MINLOC. struct float_int { float value; int index; }; struct float_float { float value; float index; }; struct long_long { long value; long index; }; struct double_double { double value; double index; }; struct long_int { long value; int index; }; struct double_int { double value; int index; }; struct short_int { short value; int index; }; struct int_int { int value; int index; }; struct long_double_int { long double value; int index; }; struct integer128_t { int64_t value; int64_t index; }; namespace simgrid{ namespace smpi{ class Datatype : public F2C, public Keyval{ char* name_; size_t size_; MPI_Aint lb_; MPI_Aint ub_; int flags_; int refcount_; public: static std::unordered_map keyvals_; static int keyval_id_; Datatype(int size, MPI_Aint lb, MPI_Aint ub, int flags); Datatype(char* name, int size, MPI_Aint lb, MPI_Aint ub, int flags); Datatype(Datatype* datatype, int* ret); virtual ~Datatype(); char* name(); size_t size(); MPI_Aint lb(); MPI_Aint ub(); int flags(); int refcount(); void ref(); static void unref(MPI_Datatype datatype); void commit(); bool is_valid(); bool is_basic(); bool is_replayable(); void addflag(int flag); int extent(MPI_Aint* lb, MPI_Aint* extent); MPI_Aint get_extent(); void get_name(char* name, int* length); void set_name(char* name); static int copy(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype); virtual void serialize(void* noncontiguous, void* contiguous, int count); virtual void unserialize(void* contiguous, void* noncontiguous, int count, MPI_Op op); static int keyval_create(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval, void* extra_state); static int keyval_free(int* keyval); int pack(void* inbuf, int incount, void* outbuf, int outcount, int* position, MPI_Comm comm); int unpack(void* inbuf, int insize, int* position, void* outbuf, int outcount, MPI_Comm comm); static int create_contiguous(int count, MPI_Datatype old_type, MPI_Aint lb, MPI_Datatype* new_type); static int create_vector(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* new_type); static int create_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type); static int create_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type); static int create_hindexed(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type); static int create_struct(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type); static Datatype* f2c(int id); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_keyvals.hpp0000644000175000017500000001065013217757320021721 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_KEYVALS_HPP_INCLUDED #define SMPI_KEYVALS_HPP_INCLUDED #include "smpi/smpi.h" #include struct smpi_delete_fn { MPI_Comm_delete_attr_function *comm_delete_fn; MPI_Type_delete_attr_function *type_delete_fn; MPI_Win_delete_attr_function *win_delete_fn; }; struct smpi_copy_fn { MPI_Comm_copy_attr_function *comm_copy_fn; MPI_Type_copy_attr_function *type_copy_fn; MPI_Win_copy_attr_function *win_copy_fn; }; struct s_smpi_key_elem_t { smpi_copy_fn copy_fn; smpi_delete_fn delete_fn; int refcount; }; typedef s_smpi_key_elem_t* smpi_key_elem; namespace simgrid{ namespace smpi{ class Keyval{ private: std::unordered_map attributes_; protected: std::unordered_map* attributes(); public: // Each subclass should have two members, as we want to separate the ones for Win, Comm, and Datatypes : // static std::unordered_map keyvals_; // static int keyval_id_; template static int keyval_create(smpi_copy_fn copy_fn, smpi_delete_fn delete_fn, int* keyval, void* extra_statee); template static int keyval_free(int* keyval); template int attr_delete(int keyval); template int attr_get(int keyval, void* attr_value, int* flag); template int attr_put(int keyval, void* attr_value); template static int call_deleter(T* obj, smpi_key_elem elem, int keyval, void * value, int* flag); template void cleanup_attr(); }; template int Keyval::keyval_create(smpi_copy_fn copy_fn, smpi_delete_fn delete_fn, int* keyval, void* extra_state){ smpi_key_elem value = new s_smpi_key_elem_t; value->copy_fn=copy_fn; value->delete_fn=delete_fn; value->refcount=1; *keyval = T::keyval_id_; T::keyvals_.insert({*keyval, value}); T::keyval_id_++; return MPI_SUCCESS; } template int Keyval::keyval_free(int* keyval){ /* See MPI-1, 5.7.1. Freeing the keyval does not remove it if it * is in use in an attribute */ smpi_key_elem elem = T::keyvals_.at(*keyval); if(elem==0){ return MPI_ERR_ARG; } if(elem->refcount==1){ T::keyvals_.erase(*keyval); delete elem; }else{ elem->refcount--; } return MPI_SUCCESS; } template int Keyval::attr_delete(int keyval){ smpi_key_elem elem = T::keyvals_.at(keyval); if(elem==nullptr) return MPI_ERR_ARG; elem->refcount--; void * value = nullptr; int flag=0; if(this->attr_get(keyval, &value, &flag)==MPI_SUCCESS){ int ret = call_deleter((T*)this, elem, keyval,value,&flag); if(ret!=MPI_SUCCESS) return ret; } if(attributes()->empty()) return MPI_ERR_ARG; attributes()->erase(keyval); return MPI_SUCCESS; } template int Keyval::attr_get(int keyval, void* attr_value, int* flag){ smpi_key_elem elem = T::keyvals_.at(keyval); if(elem==nullptr) return MPI_ERR_ARG; if(attributes()->empty()){ *flag=0; return MPI_SUCCESS; } const auto& attribs = attributes(); auto attr = attribs->find(keyval); if (attr != attribs->end()) { *static_cast(attr_value) = attr->second; *flag=1; } else { *flag=0; } return MPI_SUCCESS; } template int Keyval::attr_put(int keyval, void* attr_value){ smpi_key_elem elem = T::keyvals_.at(keyval); if(elem==nullptr) return MPI_ERR_ARG; elem->refcount++; void * value = nullptr; int flag=0; this->attr_get(keyval, &value, &flag); if(flag!=0){ int ret = call_deleter((T*)this, elem, keyval,value,&flag); if(ret!=MPI_SUCCESS) return ret; } attributes()->insert({keyval, attr_value}); return MPI_SUCCESS; } template void Keyval::cleanup_attr(){ if (not attributes()->empty()) { int flag=0; for (auto const& it : attributes_) { auto elm = T::keyvals_.find(it.first); if (elm != T::keyvals_.end()) { smpi_key_elem elem = elm->second; if(elem != nullptr){ call_deleter((T*)this, elem, it.first,it.second,&flag); } } else { //already deleted, not a problem; flag=0; } } } } } } #endif SimGrid-3.18/src/smpi/include/smpi_coll.hpp0000644000175000017500000004420713217757320021201 0ustar mquinsonmquinson/*High level handling of collective algorithms*/ /* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_COLL_HPP #define SMPI_COLL_HPP #include "private.hpp" #include "xbt/base.h" /** \brief MPI collective description */ #define COLL_DEFS(cat, ret, args, args2) \ static void set_##cat(std::string name); \ static s_mpi_coll_description_t mpi_coll_##cat##_description[]; \ static int(*cat) args; #define COLL_SIG(cat, ret, args, args2)\ static int cat args; #define COLL_DESCRIPTION(cat, ret, args, name) \ {# name,\ # cat " " # name " collective",\ (void*) Coll_ ## cat ## _ ## name::cat } #define COLL_PROTO(cat, ret, args, name) \ class Coll_ ## cat ## _ ## name : public Coll { \ public: \ static ret cat (COLL_UNPAREN args); \ }; #define COLL_UNPAREN(...) __VA_ARGS__ #define COLL_APPLY(action, sig, name) action(sig, name) #define COLL_COMMA , #define COLL_NOsep #define COLL_NOTHING(...) #define COLL_GATHER_SIG gather, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ int root, MPI_Comm comm) #define COLL_ALLGATHER_SIG allgather, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_ALLGATHERV_SIG allgatherv, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int *recv_count, int *recv_disps, \ MPI_Datatype recv_type, MPI_Comm comm) #define COLL_ALLTOALL_SIG alltoall, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_ALLTOALLV_SIG alltoallv, int, \ (void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, \ void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_BCAST_SIG bcast, int, \ (void *buf, int count, MPI_Datatype datatype, \ int root, MPI_Comm comm) #define COLL_REDUCE_SIG reduce, int, \ (void *buf, void *rbuf, int count, MPI_Datatype datatype, \ MPI_Op op, int root, MPI_Comm comm) #define COLL_ALLREDUCE_SIG allreduce, int, \ (void *sbuf, void *rbuf, int rcount, \ MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) #define COLL_REDUCE_SCATTER_SIG reduce_scatter, int, \ (void *sbuf, void *rbuf, int *rcounts,\ MPI_Datatype dtype,MPI_Op op,MPI_Comm comm) #define COLL_SCATTER_SIG scatter, int, \ (void *sendbuf, int sendcount, MPI_Datatype sendtype,\ void *recvbuf, int recvcount, MPI_Datatype recvtype,\ int root, MPI_Comm comm) #define COLL_BARRIER_SIG barrier, int, \ (MPI_Comm comm) namespace simgrid{ namespace smpi{ struct s_mpi_coll_description_t { const char *name; const char *description; void *coll; }; class Colls{ public: static XBT_PUBLIC(void) coll_help(const char *category, s_mpi_coll_description_t * table); static XBT_PUBLIC(int) find_coll_description(s_mpi_coll_description_t* table, std::string name, const char* desc); static void set_collectives(); // for each collective type, create the set_* prototype, the description array and the function pointer COLL_APPLY(COLL_DEFS, COLL_GATHER_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_ALLGATHER_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_ALLGATHERV_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_REDUCE_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_ALLREDUCE_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_REDUCE_SCATTER_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_SCATTER_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_BARRIER_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_BCAST_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_ALLTOALL_SIG, ""); COLL_APPLY(COLL_DEFS, COLL_ALLTOALLV_SIG, ""); // These fairly unused collectives only have one implementation in SMPI static int gatherv(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int* recvcounts, int* displs, MPI_Datatype recvtype, int root, MPI_Comm comm); static int scatterv(void* sendbuf, int* sendcounts, int* displs, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm); static int scan(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); static int exscan(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); static void (*smpi_coll_cleanup_callback)(); }; class Coll { public: // for each collective type, create a function member COLL_APPLY(COLL_SIG,COLL_GATHER_SIG,""); COLL_APPLY(COLL_SIG,COLL_ALLGATHER_SIG,""); COLL_APPLY(COLL_SIG,COLL_ALLGATHERV_SIG,""); COLL_APPLY(COLL_SIG,COLL_REDUCE_SIG,""); COLL_APPLY(COLL_SIG,COLL_ALLREDUCE_SIG,""); COLL_APPLY(COLL_SIG,COLL_REDUCE_SCATTER_SIG,""); COLL_APPLY(COLL_SIG,COLL_SCATTER_SIG,""); COLL_APPLY(COLL_SIG,COLL_BARRIER_SIG,""); COLL_APPLY(COLL_SIG,COLL_BCAST_SIG,""); COLL_APPLY(COLL_SIG,COLL_ALLTOALL_SIG,""); COLL_APPLY(COLL_SIG,COLL_ALLTOALLV_SIG,""); }; /************* * GATHER * *************/ #define COLL_GATHERS(action, COLL_sep) \ COLL_APPLY(action, COLL_GATHER_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_linear_sync) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, automatic) COLL_GATHERS(COLL_PROTO, COLL_NOsep) /************* * ALLGATHER * *************/ #define COLL_ALLGATHERS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLGATHER_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, 2dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, 3dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, GB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, loosely_lr) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, NTSLR) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, NTSLR_NB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, rhv) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, SMP_NTS) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, smp_simple) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, spreading_simple) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ompi_neighborexchange) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mvapich2_smp) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, automatic) COLL_ALLGATHERS(COLL_PROTO, COLL_NOsep) /************** * ALLGATHERV * **************/ #define COLL_ALLGATHERVS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, GB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi_neighborexchange) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi_bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich_ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, automatic) COLL_ALLGATHERVS(COLL_PROTO, COLL_NOsep) /************* * ALLREDUCE * *************/ #define COLL_ALLREDUCES(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, lr) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab1) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab2) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_binomial) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_binomial_pipeline) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag_lr) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag_rab) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, redbcast) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, ompi_ring_segmented) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2_rs) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, automatic) COLL_ALLREDUCES(COLL_PROTO, COLL_NOsep) /************ * ALLTOALL * ************/ #define COLL_ALLTOALLS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLTOALL_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, 2dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, 3dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, basic_linear) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_rma) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mvapich2_scatter_dest) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, automatic) COLL_ALLTOALLS(COLL_PROTO, COLL_NOsep) /************* * ALLTOALLV * *************/ #define COLL_ALLTOALLVS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, automatic) COLL_ALLTOALLVS(COLL_PROTO, COLL_NOsep) /********* * BCAST * *********/ #define COLL_BCASTS(action, COLL_sep) \ COLL_APPLY(action, COLL_BCAST_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_pattern_aware) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_pattern_aware_wait) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_scatter) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, binomial_tree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, flattree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, flattree_pipeline) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSB) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSL) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSL_Isend) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, scatter_LR_allgather) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, scatter_rdb_allgather) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_binary) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_binomial) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_linear) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi_split_bintree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi_pipeline) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_inter_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_intra_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_knomial_intra_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, automatic) COLL_BCASTS(COLL_PROTO, COLL_NOsep) /********** * REDUCE * **********/ #define COLL_REDUCES(action, COLL_sep) \ COLL_APPLY(action, COLL_REDUCE_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, arrival_pattern_aware) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, binomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, flat_tree) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, NTSL) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, scatter_gather) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_chain) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_pipeline) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_in_order_binary) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_binary) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2_knomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, rab) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, automatic) COLL_REDUCES(COLL_PROTO, COLL_NOsep) /************* * REDUCE_SCATTER * *************/ #define COLL_REDUCE_SCATTERS(action, COLL_sep) \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi_basic_recursivehalving) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi_ring) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_pair) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_rdb) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_noncomm) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, automatic) COLL_REDUCE_SCATTERS(COLL_PROTO, COLL_NOsep) /************* * SCATTER * *************/ #define COLL_SCATTERS(action, COLL_sep) \ COLL_APPLY(action, COLL_SCATTER_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2_two_level_binomial) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2_two_level_direct) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, automatic) COLL_SCATTERS(COLL_PROTO, COLL_NOsep) /************* * BARRIER * *************/ #define COLL_BARRIERS(action, COLL_sep) \ COLL_APPLY(action, COLL_BARRIER_SIG, default) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_two_procs) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_tree) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_bruck) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_recursivedoubling) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_doublering) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mvapich2_pair) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, automatic) COLL_BARRIERS(COLL_PROTO, COLL_NOsep) } } #endif SimGrid-3.18/src/smpi/include/smpi_topo.hpp0000644000175000017500000000316013217757320021222 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_TOPO_HPP_INCLUDED #define SMPI_TOPO_HPP_INCLUDED #include "smpi_comm.hpp" #include "smpi_status.hpp" typedef SMPI_Topology *MPI_Topology; namespace simgrid{ namespace smpi{ class Topo { public: virtual ~Topo()=default; MPI_Comm getComm() const { return comm_; } void setComm(MPI_Comm comm) { comm_ = comm; } private: MPI_Comm comm_; }; class Topo_Cart: public Topo { private: int nnodes_ = 0; int ndims_; int *dims_; int *periodic_; int *position_; public: explicit Topo_Cart(int ndims); ~Topo_Cart(); Topo_Cart(MPI_Comm comm_old, int ndims, int dims[], int periods[], int reorder, MPI_Comm *comm_cart); Topo_Cart* sub(const int remain_dims[], MPI_Comm *newcomm) ; int coords(int rank, int maxdims, int coords[]) ; int get(int maxdims, int* dims, int* periods, int* coords); int rank(int* coords, int* rank); int shift(int direction, int disp, int *rank_source, int *rank_dest) ; int dim_get(int *ndims); static int Dims_create(int nnodes, int ndims, int dims[]); }; class Topo_Graph: public Topo { private: int nnodes_; int *index_; int *edges_; public: Topo_Graph(); ~Topo_Graph(); }; class Topo_Dist_Graph: public Topo { private: int *in_; int *in_weights_; int *out_; int *out_weights_; public: Topo_Dist_Graph(); ~Topo_Dist_Graph(); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_info.hpp0000644000175000017500000000174713217757320021205 0ustar mquinsonmquinson/* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_INFO_HPP #define SMPI_INFO_HPP #include "smpi/smpi.h" #include "smpi_f2c.hpp" #include #include namespace simgrid{ namespace smpi{ class Info : public F2C{ private: std::map map_; int refcount_ = 1; public: Info() = default; explicit Info(Info* orig); ~Info() = default; void ref(); static void unref(MPI_Info info); void set(char *key, char *value); int get(char *key,int valuelen, char *value, int *flag); int remove(char* key); int get_nkeys(int *nkeys); int get_nthkey(int n, char *key); int get_valuelen(char *key, int *valuelen, int *flag); static Info* f2c(int id); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_win.hpp0000644000175000017500000000631713217757320021045 0ustar mquinsonmquinson/* Copyright (c) 2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_WIN_HPP_INCLUDED #define SMPI_WIN_HPP_INCLUDED #include "smpi_f2c.hpp" #include "smpi_keyvals.hpp" #include "xbt/synchro.h" #include #include #include namespace simgrid{ namespace smpi{ class Win : public F2C, public Keyval { private : void* base_; MPI_Aint size_; int disp_unit_; int assert_; MPI_Info info_; MPI_Comm comm_; std::vector *requests_; xbt_mutex_t mut_; msg_bar_t bar_; MPI_Win* connected_wins_; char* name_; int opened_; MPI_Group group_; int count_; //for ordering the accs xbt_mutex_t lock_mut_; xbt_mutex_t atomic_mut_; std::list lockers_; int rank_; // to identify owner for barriers in MPI_COMM_WORLD int mode_; // exclusive or shared lock int allocated_; int dynamic_; public: static std::unordered_map keyvals_; static int keyval_id_; Win(void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, int allocated = 0, int dynamic = 0); Win(MPI_Info info, MPI_Comm comm) : Win(MPI_BOTTOM, 0, 1, info, comm, 0, 1) {}; ~Win(); int attach (void *base, MPI_Aint size); int detach (void *base); void get_name( char* name, int* length); void get_group( MPI_Group* group); void set_name( char* name); int rank(); int dynamic(); int start(MPI_Group group, int assert); int post(MPI_Group group, int assert); int complete(); MPI_Info info(); void set_info( MPI_Info info); int wait(); MPI_Aint size(); void* base(); int disp_unit(); int fence(int assert); int put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Request* request=nullptr); int get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Request* request=nullptr); int accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Request* request=nullptr); int get_accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Request* request=nullptr); int compare_and_swap(void *origin_addr, void *compare_addr, void *result_addr, MPI_Datatype datatype, int target_rank, MPI_Aint target_disp); static Win* f2c(int id); int lock(int lock_type, int rank, int assert); int unlock(int rank); int lock_all(int assert); int unlock_all(); int flush(int rank); int flush_local(int rank); int flush_all(); int flush_local_all(); int finish_comms(); int finish_comms(int rank); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_process.hpp0000644000175000017500000000526613217757320021730 0ustar mquinsonmquinson/* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_PROCESS_HPP #define SMPI_PROCESS_HPP #include "private.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "src/instr/instr_smpi.hpp" #include "xbt/synchro.h" namespace simgrid{ namespace smpi{ class Process { private: double simulated_ = 0 /* Used to time with simulated_start/elapsed */; int* argc_ = nullptr; char*** argv_ = nullptr; simgrid::s4u::MailboxPtr mailbox_; simgrid::s4u::MailboxPtr mailbox_small_; xbt_mutex_t mailboxes_mutex_; xbt_os_timer_t timer_; MPI_Comm comm_self_ = MPI_COMM_NULL; MPI_Comm comm_intra_ = MPI_COMM_NULL; MPI_Comm* comm_world_ = nullptr; void* data_ = nullptr; /* user data */ int index_ = MPI_UNDEFINED; char state_; int sampling_ = 0; /* inside an SMPI_SAMPLE_ block? */ char* instance_id_ = nullptr; bool replaying_ = false; /* is the process replaying a trace */ msg_bar_t finalization_barrier_; int return_value_ = 0; smpi_trace_call_location_t trace_call_loc_; smx_actor_t process_ = nullptr; smpi_privatization_region_t privatized_region_; #if HAVE_PAPI /** Contains hardware data as read by PAPI **/ int papi_event_set_; papi_counter_t papi_counter_data_; #endif public: explicit Process(int index, msg_bar_t barrier); void set_data(int index, int* argc, char*** argv); void finalize(); int finalized(); int initialized(); void mark_as_initialized(); void set_replaying(bool value); bool replaying(); void set_user_data(void *data); void *get_user_data(); smpi_trace_call_location_t* call_location(); void set_privatized_region(smpi_privatization_region_t region); smpi_privatization_region_t privatized_region(); int index(); smx_mailbox_t mailbox(); smx_mailbox_t mailbox_small(); xbt_mutex_t mailboxes_mutex(); #if HAVE_PAPI int papi_event_set(); papi_counter_t& papi_counters(); #endif xbt_os_timer_t timer(); void simulated_start(); double simulated_elapsed(); MPI_Comm comm_world(); MPI_Comm comm_self(); MPI_Comm comm_intra(); void set_comm_intra(MPI_Comm comm); void set_sampling(int s); int sampling(); msg_bar_t finalization_barrier(); int return_value(); void set_return_value(int val); static void init(int *argc, char ***argv); smx_actor_t process(); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_group.hpp0000644000175000017500000000305113217757320021374 0ustar mquinsonmquinson/* Copyright (c) 2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_GROUP_HPP_INCLUDED #define SMPI_GROUP_HPP_INCLUDED #include "smpi_f2c.hpp" #include #include namespace simgrid{ namespace smpi{ class Group : public F2C{ private: int size_; /* This is actually a map from int to int. We could use * std::map here, but looking up a value there costs O(log(n)). * For a vector, this costs O(1). We hence go with the vector. */ std::vector rank_to_index_map_; std::vector index_to_rank_map_; int refcount_; public: explicit Group(); explicit Group(int size); explicit Group(Group* origin); void set_mapping(int index, int rank); int index(int rank); int rank(int index); void ref(); static void unref(MPI_Group group); int size(); int compare(MPI_Group group2); int incl(int n, int* ranks, MPI_Group* newgroup); int excl(int n, int *ranks, MPI_Group * newgroup); int group_union(MPI_Group group2, MPI_Group* newgroup); int intersection(MPI_Group group2, MPI_Group* newgroup); int difference(MPI_Group group2, MPI_Group* newgroup); int range_incl(int n, int ranges[][3], MPI_Group * newgroup); int range_excl(int n, int ranges[][3], MPI_Group * newgroup); static Group* f2c(int id); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_op.hpp0000644000175000017500000000141313217757320020656 0ustar mquinsonmquinson/* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_OP_HPP #define SMPI_OP_HPP #include "smpi_info.hpp" namespace simgrid{ namespace smpi{ class Op : public F2C{ private: MPI_User_function *func_; bool is_commutative_; bool is_fortran_op_; public: Op(MPI_User_function * function, bool commutative); bool is_commutative(); bool is_fortran_op(); void set_fortran_op(); void apply(void *invec, void *inoutvec, int *len, MPI_Datatype datatype); static Op* f2c(int id); }; } } #endif SimGrid-3.18/src/smpi/include/private.hpp0000644000175000017500000007037113217757321020674 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_PRIVATE_HPP #define SMPI_PRIVATE_HPP #include "include/xbt/config.hpp" #include "simgrid/msg.h" // msg_bar_t #include "smpi/smpi.h" #include "src/instr/instr_smpi.hpp" #include "src/internal_config.h" #include #include extern "C" { #define PERSISTENT 0x1 #define NON_PERSISTENT 0x2 #define SEND 0x4 #define RECV 0x8 #define RECV_DELETE 0x10 #define ISEND 0x20 #define SSEND 0x40 #define PREPARED 0x80 #define FINISHED 0x100 #define RMA 0x200 #define ACCUMULATE 0x400 enum smpi_process_state { SMPI_UNINITIALIZED, SMPI_INITIALIZED, SMPI_FINALIZED }; #define COLL_TAG_REDUCE -112 #define COLL_TAG_SCATTER -223 #define COLL_TAG_SCATTERV -334 #define COLL_TAG_GATHER -445 #define COLL_TAG_ALLGATHER -556 #define COLL_TAG_ALLGATHERV -667 #define COLL_TAG_BARRIER -778 #define COLL_TAG_REDUCE_SCATTER -889 #define COLL_TAG_ALLTOALLV -1000 #define COLL_TAG_ALLTOALL -1112 #define COLL_TAG_GATHERV -2223 #define COLL_TAG_BCAST -3334 #define COLL_TAG_ALLREDUCE -4445 // SMPI_RMA_TAG has to be the smallest one, as it will be decremented for accumulate ordering. #define SMPI_RMA_TAG -6666 /* Convert between Fortran and C */ #define FORT_BOTTOM(addr) ((*(int*)addr) == -200 ? MPI_BOTTOM : (void*)addr) #define FORT_IN_PLACE(addr) ((*(int*)addr) == -100 ? MPI_IN_PLACE : (void*)addr) #define FORT_STATUS_IGNORE(addr) (static_cast((*(int*)addr) == -300 ? MPI_STATUS_IGNORE : (void*)addr)) #define FORT_STATUSES_IGNORE(addr) (static_cast((*(int*)addr) == -400 ? MPI_STATUSES_IGNORE : (void*)addr)) extern XBT_PRIVATE MPI_Comm MPI_COMM_UNINITIALIZED; typedef SMPI_Cart_topology* MPIR_Cart_Topology; typedef SMPI_Graph_topology* MPIR_Graph_Topology; typedef SMPI_Dist_Graph_topology* MPIR_Dist_Graph_Topology; XBT_PRIVATE SMPI_Process* smpi_process(); XBT_PRIVATE SMPI_Process* smpi_process_remote(int index); XBT_PRIVATE int smpi_process_count(); XBT_PRIVATE void smpi_deployment_register_process(const char* instance_id, int rank, int index); XBT_PRIVATE MPI_Comm* smpi_deployment_comm_world(const char* instance_id); XBT_PRIVATE msg_bar_t smpi_deployment_finalization_barrier(const char* instance_id); XBT_PRIVATE void smpi_deployment_cleanup_instances(); XBT_PRIVATE void smpi_comm_copy_buffer_callback(smx_activity_t comm, void* buff, size_t buff_size); XBT_PRIVATE void smpi_comm_null_copy_buffer_callback(smx_activity_t comm, void* buff, size_t buff_size); XBT_PRIVATE int smpi_enabled(); XBT_PRIVATE void smpi_global_init(); XBT_PRIVATE void smpi_global_destroy(); XBT_PRIVATE double smpi_mpi_wtime(); XBT_PRIVATE void smpi_mpi_init(); // utilities extern XBT_PRIVATE double smpi_cpu_threshold; extern XBT_PRIVATE double smpi_host_speed; extern XBT_PRIVATE char* smpi_data_exe_start; // start of the data+bss segment of the executable extern XBT_PRIVATE int smpi_data_exe_size; // size of the data+bss segment of the executable enum shared_malloc_type { shmalloc_none, shmalloc_local, shmalloc_global }; extern XBT_PRIVATE shared_malloc_type smpi_cfg_shared_malloc; // Whether to activate shared malloc XBT_PRIVATE void smpi_switch_data_segment(int dest); XBT_PRIVATE void smpi_really_switch_data_segment(int dest); XBT_PRIVATE int smpi_is_privatization_file(char* file); XBT_PRIVATE void smpi_get_executable_global_size(); XBT_PRIVATE void smpi_backup_global_memory_segment(); XBT_PRIVATE void smpi_destroy_global_memory_segments(); XBT_PRIVATE void smpi_bench_destroy(); XBT_PRIVATE void smpi_bench_begin(); XBT_PRIVATE void smpi_bench_end(); XBT_PRIVATE void smpi_shared_destroy(); XBT_PRIVATE void* smpi_get_tmp_sendbuffer(int size); XBT_PRIVATE void* smpi_get_tmp_recvbuffer(int size); XBT_PRIVATE void smpi_free_tmp_buffer(void* buf); // f77 wrappers void mpi_init_(int* ierr); void mpi_finalize_(int* ierr); void mpi_abort_(int* comm, int* errorcode, int* ierr); void mpi_comm_rank_(int* comm, int* rank, int* ierr); void mpi_comm_size_(int* comm, int* size, int* ierr); double mpi_wtime_(); double mpi_wtick_(); void mpi_initialized_(int* flag, int* ierr); void mpi_comm_dup_(int* comm, int* newcomm, int* ierr); void mpi_comm_create_(int* comm, int* group, int* newcomm, int* ierr); void mpi_comm_free_(int* comm, int* ierr); void mpi_comm_split_(int* comm, int* color, int* key, int* comm_out, int* ierr); void mpi_group_incl_(int* group, int* n, int* key, int* group_out, int* ierr); void mpi_comm_group_(int* comm, int* group_out, int* ierr); void mpi_comm_create_group_(int* comm, int* group, int, int* comm_out, int* ierr); void mpi_send_init_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_isend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_irsend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_send_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr); void mpi_rsend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr); void mpi_recv_init_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr); void mpi_irecv_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr); void mpi_recv_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, MPI_Status* status, int* ierr); void mpi_start_(int* request, int* ierr); void mpi_startall_(int* count, int* requests, int* ierr); void mpi_wait_(int* request, MPI_Status* status, int* ierr); void mpi_waitany_(int* count, int* requests, int* index, MPI_Status* status, int* ierr); void mpi_waitall_(int* count, int* requests, MPI_Status* status, int* ierr); void mpi_barrier_(int* comm, int* ierr); void mpi_bcast_(void* buf, int* count, int* datatype, int* root, int* comm, int* ierr); void mpi_reduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* root, int* comm, int* ierr); void mpi_allreduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_reduce_scatter_(void* sendbuf, void* recvbuf, int* recvcounts, int* datatype, int* op, int* comm, int* ierr); void mpi_scatter_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_scatterv_(void* sendbuf, int* sendcounts, int* displs, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_gather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_gatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts, int* displs, int* recvtype, int* root, int* comm, int* ierr); void mpi_allgather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr); void mpi_allgatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* displs, int* recvtype, int* comm, int* ierr); void mpi_type_size_(int* datatype, int* size, int* ierr); void mpi_scan_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_alltoall_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr); void mpi_alltoallv_(void* sendbuf, int* sendcounts, int* senddisps, int* sendtype, void* recvbuf, int* recvcounts, int* recvdisps, int* recvtype, int* comm, int* ierr); void mpi_get_processor_name_(char* name, int* resultlen, int* ierr); void mpi_test_(int* request, int* flag, MPI_Status* status, int* ierr); void mpi_testall_(int* count, int* requests, int* flag, MPI_Status* statuses, int* ierr); void mpi_get_count_(MPI_Status* status, int* datatype, int* count, int* ierr); void mpi_type_extent_(int* datatype, MPI_Aint* extent, int* ierr); void mpi_attr_get_(int* comm, int* keyval, void* attr_value, int* flag, int* ierr); void mpi_type_commit_(int* datatype, int* ierr); void mpi_type_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr); void mpi_type_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr); void mpi_type_create_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr); void mpi_type_free_(int* datatype, int* ierr); void mpi_type_lb_(int* datatype, MPI_Aint* extent, int* ierr); void mpi_type_ub_(int* datatype, MPI_Aint* extent, int* ierr); void mpi_win_fence_(int* assert, int* win, int* ierr); void mpi_win_free_(int* win, int* ierr); void mpi_win_create_(int* base, MPI_Aint* size, int* disp_unit, int* info, int* comm, int* win, int* ierr); void mpi_win_set_name_(int* win, char* name, int* ierr, int size); void mpi_win_get_name_(int* win, char* name, int* len, int* ierr); void mpi_win_post_(int* group, int assert, int* win, int* ierr); void mpi_win_start_(int* group, int assert, int* win, int* ierr); void mpi_win_complete_(int* win, int* ierr); void mpi_win_wait_(int* win, int* ierr); void mpi_win_allocate_(MPI_Aint* size, int* disp_unit, int* info, int* comm, void* base, int* win, int* ierr); void mpi_win_attach_(int* win, int* base, MPI_Aint* size, int* ierr); void mpi_win_create_dynamic_(int* info, int* comm, int* win, int* ierr); void mpi_win_detach_(int* win, int* base, int* ierr); void mpi_win_set_info_(int* win, int* info, int* ierr); void mpi_win_get_info_(int* win, int* info, int* ierr); void mpi_win_get_group_(int* win, int* group, int* ierr); void mpi_win_get_attr_(int* win, int* type_keyval, void* attribute_val, int* flag, int* ierr); void mpi_win_set_attr_(int* win, int* type_keyval, void* att, int* ierr); void mpi_win_delete_attr_(int* win, int* comm_keyval, int* ierr); void mpi_win_create_keyval_(void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_win_free_keyval_(int* keyval, int* ierr); void mpi_win_lock_(int* lock_type, int* rank, int* assert, int* win, int* ierr); void mpi_win_lock_all_(int* assert, int* win, int* ierr); void mpi_win_unlock_(int* rank, int* win, int* ierr); void mpi_win_unlock_all_(int* win, int* ierr); void mpi_win_flush_(int* rank, int* win, int* ierr); void mpi_win_flush_local_(int* rank, int* win, int* ierr); void mpi_win_flush_all_(int* win, int* ierr); void mpi_win_flush_local_all_(int* win, int* ierr); void mpi_info_create_(int* info, int* ierr); void mpi_info_set_(int* info, char* key, char* value, int* ierr, unsigned int keylen, unsigned int valuelen); void mpi_info_free_(int* info, int* ierr); void mpi_get_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* ierr); void mpi_put_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* ierr); void mpi_rget_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* request, int* ierr); void mpi_rput_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* request, int* ierr); void mpi_fetch_and_op_(int* origin_addr, int* result_addr, int* datatype, int* target_rank, MPI_Aint* target_disp, int* op, int* win, int* ierr); void mpi_compare_and_swap_(int* origin_addr, int* compare_addr, int* result_addr, int* datatype, int* target_rank, MPI_Aint* target_disp, int* win, int* ierr); void mpi_get_accumulate_(int* origin_addr, int* origin_count, int* origin_datatype, int* result_addr, int* result_count, int* result_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* ierr); void mpi_rget_accumulate_(int* origin_addr, int* origin_count, int* origin_datatype, int* result_addr, int* result_count, int* result_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* request, int* ierr); void mpi_accumulate_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* ierr); void mpi_raccumulate_(int* origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* request, int* ierr); void mpi_error_string_(int* errorcode, char* string, int* resultlen, int* ierr); void mpi_sendrecv_(void* sendbuf, int* sendcount, int* sendtype, int* dst, int* sendtag, void* recvbuf, int* recvcount, int* recvtype, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr); void mpi_finalized_(int* flag, int* ierr); void mpi_init_thread_(int* required, int* provided, int* ierr); void mpi_query_thread_(int* provided, int* ierr); void mpi_is_thread_main_(int* flag, int* ierr); void mpi_address_(void* location, MPI_Aint* address, int* ierr); void mpi_get_address_(void* location, MPI_Aint* address, int* ierr); void mpi_type_dup_(int* datatype, int* newdatatype, int* ierr); void mpi_type_set_name_(int* datatype, char* name, int* ierr, int size); void mpi_type_get_name_(int* datatype, char* name, int* len, int* ierr); void mpi_type_get_attr_(int* type, int* type_keyval, void* attribute_val, int* flag, int* ierr); void mpi_type_set_attr_(int* type, int* type_keyval, void* attribute_val, int* ierr); void mpi_type_delete_attr_(int* type, int* type_keyval, int* ierr); void mpi_type_create_keyval_(void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_type_free_keyval_(int* keyval, int* ierr); void mpi_pcontrol_(int* level, int* ierr); void mpi_type_get_extent_(int* datatype, MPI_Aint* lb, MPI_Aint* extent, int* ierr); void mpi_type_get_true_extent_(int* datatype, MPI_Aint* lb, MPI_Aint* extent, int* ierr); void mpi_op_create_(void* function, int* commute, int* op, int* ierr); void mpi_op_free_(int* op, int* ierr); void mpi_op_commutative_(int* op, int* commute, int* ierr); void mpi_group_free_(int* group, int* ierr); void mpi_group_size_(int* group, int* size, int* ierr); void mpi_group_rank_(int* group, int* rank, int* ierr); void mpi_group_translate_ranks_(int* group1, int* n, int* ranks1, int* group2, int* ranks2, int* ierr); void mpi_group_compare_(int* group1, int* group2, int* result, int* ierr); void mpi_group_union_(int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_intersection_(int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_difference_(int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_excl_(int* group, int* n, int* ranks, int* newgroup, int* ierr); void mpi_group_range_incl_(int* group, int* n, int ranges[][3], int* newgroup, int* ierr); void mpi_group_range_excl_(int* group, int* n, int ranges[][3], int* newgroup, int* ierr); void mpi_comm_get_attr_(int* comm, int* comm_keyval, void* attribute_val, int* flag, int* ierr); void mpi_comm_set_attr_(int* comm, int* comm_keyval, void* attribute_val, int* ierr); void mpi_comm_delete_attr_(int* comm, int* comm_keyval, int* ierr); void mpi_comm_create_keyval_(void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_comm_free_keyval_(int* keyval, int* ierr); void mpi_comm_get_name_(int* comm, char* name, int* len, int* ierr); void mpi_comm_compare_(int* comm1, int* comm2, int* result, int* ierr); void mpi_comm_disconnect_(int* comm, int* ierr); void mpi_request_free_(int* request, int* ierr); void mpi_sendrecv_replace_(void* buf, int* count, int* datatype, int* dst, int* sendtag, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr); void mpi_testany_(int* count, int* requests, int* index, int* flag, MPI_Status* status, int* ierr); void mpi_waitsome_(int* incount, int* requests, int* outcount, int* indices, MPI_Status* status, int* ierr); void mpi_reduce_local_(void* inbuf, void* inoutbuf, int* count, int* datatype, int* op, int* ierr); void mpi_reduce_scatter_block_(void* sendbuf, void* recvbuf, int* recvcount, int* datatype, int* op, int* comm, int* ierr); void mpi_pack_size_(int* incount, int* datatype, int* comm, int* size, int* ierr); void mpi_cart_coords_(int* comm, int* rank, int* maxdims, int* coords, int* ierr); void mpi_cart_create_(int* comm_old, int* ndims, int* dims, int* periods, int* reorder, int* comm_cart, int* ierr); void mpi_cart_get_(int* comm, int* maxdims, int* dims, int* periods, int* coords, int* ierr); void mpi_cart_map_(int* comm_old, int* ndims, int* dims, int* periods, int* newrank, int* ierr); void mpi_cart_rank_(int* comm, int* coords, int* rank, int* ierr); void mpi_cart_shift_(int* comm, int* direction, int* displ, int* source, int* dest, int* ierr); void mpi_cart_sub_(int* comm, int* remain_dims, int* comm_new, int* ierr); void mpi_cartdim_get_(int* comm, int* ndims, int* ierr); void mpi_graph_create_(int* comm_old, int* nnodes, int* index, int* edges, int* reorder, int* comm_graph, int* ierr); void mpi_graph_get_(int* comm, int* maxindex, int* maxedges, int* index, int* edges, int* ierr); void mpi_graph_map_(int* comm_old, int* nnodes, int* index, int* edges, int* newrank, int* ierr); void mpi_graph_neighbors_(int* comm, int* rank, int* maxneighbors, int* neighbors, int* ierr); void mpi_graph_neighbors_count_(int* comm, int* rank, int* nneighbors, int* ierr); void mpi_graphdims_get_(int* comm, int* nnodes, int* nedges, int* ierr); void mpi_topo_test_(int* comm, int* top_type, int* ierr); void mpi_error_class_(int* errorcode, int* errorclass, int* ierr); void mpi_errhandler_create_(void* function, void* errhandler, int* ierr); void mpi_errhandler_free_(void* errhandler, int* ierr); void mpi_errhandler_get_(int* comm, void* errhandler, int* ierr); void mpi_errhandler_set_(int* comm, void* errhandler, int* ierr); void mpi_comm_set_errhandler_(int* comm, void* errhandler, int* ierr); void mpi_comm_get_errhandler_(int* comm, void* errhandler, int* ierr); void mpi_type_contiguous_(int* count, int* old_type, int* newtype, int* ierr); void mpi_cancel_(int* request, int* ierr); void mpi_buffer_attach_(void* buffer, int* size, int* ierr); void mpi_buffer_detach_(void* buffer, int* size, int* ierr); void mpi_testsome_(int* incount, int* requests, int* outcount, int* indices, MPI_Status* statuses, int* ierr); void mpi_comm_test_inter_(int* comm, int* flag, int* ierr); void mpi_unpack_(void* inbuf, int* insize, int* position, void* outbuf, int* outcount, int* type, int* comm, int* ierr); void mpi_pack_external_size_(char* datarep, int* incount, int* datatype, MPI_Aint* size, int* ierr); void mpi_pack_external_(char* datarep, void* inbuf, int* incount, int* datatype, void* outbuf, MPI_Aint* outcount, MPI_Aint* position, int* ierr); void mpi_unpack_external_(char* datarep, void* inbuf, MPI_Aint* insize, MPI_Aint* position, void* outbuf, int* outcount, int* datatype, int* ierr); void mpi_type_hindexed_(int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr); void mpi_type_create_hindexed_(int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr); void mpi_type_create_hindexed_block_(int* count, int* blocklength, MPI_Aint* indices, int* old_type, int* newtype, int* ierr); void mpi_type_indexed_(int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr); void mpi_type_create_indexed_(int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr); void mpi_type_create_indexed_block_(int* count, int* blocklength, int* indices, int* old_type, int* newtype, int* ierr); void mpi_type_struct_(int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr); void mpi_type_create_struct_(int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr); void mpi_ssend_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr); void mpi_ssend_init_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr); void mpi_intercomm_create_(int* local_comm, int* local_leader, int* peer_comm, int* remote_leader, int* tag, int* comm_out, int* ierr); void mpi_intercomm_merge_(int* comm, int* high, int* comm_out, int* ierr); void mpi_bsend_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr); void mpi_bsend_init_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr); void mpi_ibsend_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr); void mpi_comm_remote_group_(int* comm, int* group, int* ierr); void mpi_comm_remote_size_(int* comm, int* size, int* ierr); void mpi_issend_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr); void mpi_probe_(int* source, int* tag, int* comm, MPI_Status* status, int* ierr); void mpi_attr_delete_(int* comm, int* keyval, int* ierr); void mpi_attr_put_(int* comm, int* keyval, void* attr_value, int* ierr); void mpi_rsend_init_(void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr); void mpi_keyval_create_(void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_keyval_free_(int* keyval, int* ierr); void mpi_test_cancelled_(MPI_Status* status, int* flag, int* ierr); void mpi_pack_(void* inbuf, int* incount, int* type, void* outbuf, int* outcount, int* position, int* comm, int* ierr); void mpi_get_elements_(MPI_Status* status, int* datatype, int* elements, int* ierr); void mpi_dims_create_(int* nnodes, int* ndims, int* dims, int* ierr); void mpi_iprobe_(int* source, int* tag, int* comm, int* flag, MPI_Status* status, int* ierr); void mpi_type_get_envelope_(int* datatype, int* num_integers, int* num_addresses, int* num_datatypes, int* combiner, int* ierr); void mpi_type_get_contents_(int* datatype, int* max_integers, int* max_addresses, int* max_datatypes, int* array_of_integers, MPI_Aint* array_of_addresses, int* array_of_datatypes, int* ierr); void mpi_type_create_darray_(int* size, int* rank, int* ndims, int* array_of_gsizes, int* array_of_distribs, int* array_of_dargs, int* array_of_psizes, int* order, int* oldtype, int* newtype, int* ierr); void mpi_type_create_resized_(int* oldtype, MPI_Aint* lb, MPI_Aint* extent, int* newtype, int* ierr); void mpi_type_create_subarray_(int* ndims, int* array_of_sizes, int* array_of_subsizes, int* array_of_starts, int* order, int* oldtype, int* newtype, int* ierr); void mpi_type_match_size_(int* typeclass, int* size, int* datatype, int* ierr); void mpi_alltoallw_(void* sendbuf, int* sendcnts, int* sdispls, int* sendtypes, void* recvbuf, int* recvcnts, int* rdispls, int* recvtypes, int* comm, int* ierr); void mpi_exscan_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_comm_set_name_(int* comm, char* name, int* ierr, int size); void mpi_comm_dup_with_info_(int* comm, int* info, int* newcomm, int* ierr); void mpi_comm_split_type_(int* comm, int* split_type, int* key, int* info, int* newcomm, int* ierr); void mpi_comm_set_info_(int* comm, int* info, int* ierr); void mpi_comm_get_info_(int* comm, int* info, int* ierr); void mpi_info_get_(int* info, char* key, int* valuelen, char* value, int* flag, int* ierr, unsigned int keylen); void mpi_comm_create_errhandler_(void* function, void* errhandler, int* ierr); void mpi_add_error_class_(int* errorclass, int* ierr); void mpi_add_error_code_(int* errorclass, int* errorcode, int* ierr); void mpi_add_error_string_(int* errorcode, char* string, int* ierr); void mpi_comm_call_errhandler_(int* comm, int* errorcode, int* ierr); void mpi_info_dup_(int* info, int* newinfo, int* ierr); void mpi_info_get_valuelen_(int* info, char* key, int* valuelen, int* flag, int* ierr, unsigned int keylen); void mpi_info_delete_(int* info, char* key, int* ierr, unsigned int keylen); void mpi_info_get_nkeys_(int* info, int* nkeys, int* ierr); void mpi_info_get_nthkey_(int* info, int* n, char* key, int* ierr, unsigned int keylen); void mpi_get_version_(int* version, int* subversion, int* ierr); void mpi_get_library_version_(char* version, int* len, int* ierr); void mpi_request_get_status_(int* request, int* flag, MPI_Status* status, int* ierr); void mpi_grequest_start_(void* query_fn, void* free_fn, void* cancel_fn, void* extra_state, int* request, int* ierr); void mpi_grequest_complete_(int* request, int* ierr); void mpi_status_set_cancelled_(MPI_Status* status, int* flag, int* ierr); void mpi_status_set_elements_(MPI_Status* status, int* datatype, int* count, int* ierr); void mpi_comm_connect_(char* port_name, int* info, int* root, int* comm, int* newcomm, int* ierr); void mpi_publish_name_(char* service_name, int* info, char* port_name, int* ierr); void mpi_unpublish_name_(char* service_name, int* info, char* port_name, int* ierr); void mpi_lookup_name_(char* service_name, int* info, char* port_name, int* ierr); void mpi_comm_join_(int* fd, int* intercomm, int* ierr); void mpi_open_port_(int* info, char* port_name, int* ierr); void mpi_close_port_(char* port_name, int* ierr); void mpi_comm_accept_(char* port_name, int* info, int* root, int* comm, int* newcomm, int* ierr); void mpi_comm_spawn_(char* command, char* argv, int* maxprocs, int* info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr); void mpi_comm_spawn_multiple_(int* count, char* array_of_commands, char** array_of_argv, int* array_of_maxprocs, int* array_of_info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr); void mpi_comm_get_parent_(int* parent, int* ierr); void mpi_file_close_(int* file, int* ierr); void mpi_file_delete_(char* filename, int* info, int* ierr); void mpi_file_open_(int* comm, char* filename, int* amode, int* info, int* fh, int* ierr); void mpi_file_set_view_(int* fh, long long int* offset, int* etype, int* filetype, char* datarep, int* info, int* ierr); void mpi_file_read_(int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr); void mpi_file_write_(int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr); // TODO, make this static and expose it more cleanly struct s_smpi_privatization_region_t { void* address; int file_descriptor; }; typedef s_smpi_privatization_region_t* smpi_privatization_region_t; extern XBT_PRIVATE int smpi_loaded_page; extern XBT_PRIVATE int smpi_universe_size; XBT_PRIVATE smpi_privatization_region_t smpi_init_global_memory_segment_process(); } /** * Get the address of the beginning of the memory page where addr is located. * Note that we use an integer division here, so (a/b)*b is not a, unless a%b == 0 * * This is used when privatizing. */ #define TOPAGE(addr) (void*)(((unsigned long)(addr) / xbt_pagesize) * xbt_pagesize) #if HAVE_PAPI typedef std::vector> papi_counter_t; XBT_PRIVATE papi_counter_t& smpi_process_papi_counters(); XBT_PRIVATE int smpi_process_papi_event_set(); #endif extern std::unordered_map location2speedup; /** @brief Returns the last call location (filename, linenumber). Process-specific. */ extern "C" { XBT_PUBLIC(smpi_trace_call_location_t*) smpi_process_get_call_location(); XBT_PUBLIC(smpi_trace_call_location_t*) smpi_trace_get_call_location(); } enum smpi_priv_strategies { SMPI_PRIVATIZE_NONE = 0, SMPI_PRIVATIZE_MMAP = 1, SMPI_PRIVATIZE_DLOPEN = 2, SMPI_PRIVATIZE_DEFAULT = SMPI_PRIVATIZE_DLOPEN }; extern XBT_PRIVATE int smpi_privatize_global_variables; #endif SimGrid-3.18/src/smpi/include/smpi_utils.hpp0000644000175000017500000000123313217757321021401 0ustar mquinsonmquinson/* Copyright (c) 2016-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_UTILS_HPP #define SMPI_UTILS_HPP #include "xbt/base.h" #include #include #include extern "C" { // Methods used to parse and store the values for timing injections in smpi struct s_smpi_factor_t { size_t factor = 0; std::vector values; }; typedef s_smpi_factor_t* smpi_os_factor_t; } XBT_PUBLIC(std::vector) parse_factor(std::string smpi_coef_string); #endif SimGrid-3.18/src/smpi/include/smpi_status.hpp0000644000175000017500000000123413217757320021564 0ustar mquinsonmquinson/* Copyright (c) 2009-2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_STATUS_HPP #define SMPI_STATUS_HPP #include "smpi/smpi.h" namespace simgrid{ namespace smpi{ //Status has to stay in C, as its fields are public. //So status class only defines static methods to handle the C struct. class Status{ public: static void empty(MPI_Status * status); static int get_count(MPI_Status * status, MPI_Datatype datatype); }; } } #endif SimGrid-3.18/src/smpi/include/SmpiHost.hpp0000644000175000017500000000166613217757320020770 0ustar mquinsonmquinson/* Copyright (c) 2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_HOST_HPP_ #define SMPI_HOST_HPP_ #include "smpi_utils.hpp" #include "simgrid/s4u/Host.hpp" #include #include #include #include namespace simgrid { namespace smpi { class SmpiHost { private: std::vector orecv_parsed_values; std::vector osend_parsed_values; std::vector oisend_parsed_values; simgrid::s4u::Host *host = nullptr; public: static simgrid::xbt::Extension EXTENSION_ID; explicit SmpiHost(simgrid::s4u::Host *ptr); ~SmpiHost(); double orecv(size_t size); double osend(size_t size); double oisend(size_t size); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_f2c.hpp0000644000175000017500000000275113217757320020720 0ustar mquinsonmquinson/* Handle Fortan - C conversion for MPI Types*/ /* Copyright (c) 2010, 2013-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_F2C_HPP_INCLUDED #define SMPI_F2C_HPP_INCLUDED #include #define KEY_SIZE (sizeof(int) * 2 + 1) namespace simgrid{ namespace smpi{ class F2C { private: // We use a single lookup table for every type. // Beware of collisions if id in mpif.h is not unique static std::unordered_map* f2c_lookup_; static int f2c_id_; protected: static std::unordered_map* f2c_lookup(); static void set_f2c_lookup(std::unordered_map* map); static int f2c_id(); static void f2c_id_increment(); public: static char* get_key(char* key, int id); static char* get_key_id(char* key, int id); static void delete_lookup(); static std::unordered_map* lookup(); virtual ~F2C() = default; //Override these to handle specific values. virtual int add_f(); static void free_f(int id); virtual int c2f(); // This method should be overridden in all subclasses to avoid casting the result when calling it. // For the default one, the MPI_*_NULL returned is assumed to be NULL. static F2C* f2c(int id); }; } } #endif SimGrid-3.18/src/smpi/include/smpi_comm.hpp0000644000175000017500000000443713217757320021204 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_COMM_HPP_INCLUDED #define SMPI_COMM_HPP_INCLUDED #include #include "smpi_keyvals.hpp" #include "smpi_group.hpp" #include "smpi_topo.hpp" namespace simgrid{ namespace smpi{ class Comm : public F2C, public Keyval{ private: MPI_Group group_; SMPI_Topo_type topoType_; MPI_Topology topo_; // to be replaced by an union int refcount_; MPI_Comm leaders_comm_;//inter-node communicator MPI_Comm intra_comm_;//intra-node communicator . For MPI_COMM_WORLD this can't be used, as var is global. //use an intracomm stored in the process data instead int* leaders_map_; //who is the leader of each process int is_uniform_; int* non_uniform_map_; //set if smp nodes have a different number of processes allocated int is_blocked_;// are ranks allocated on the same smp node contiguous ? std::list rma_wins_; // attached windows for synchronization. public: static std::unordered_map keyvals_; static int keyval_id_; Comm() = default; Comm(MPI_Group group, MPI_Topology topo); int dup(MPI_Comm* newcomm); MPI_Group group(); MPI_Topology topo(); int size(); int rank(); void get_name (char* name, int* len); void set_leaders_comm(MPI_Comm leaders); void set_intra_comm(MPI_Comm leaders); int* get_non_uniform_map(); int* get_leaders_map(); MPI_Comm get_leaders_comm(); MPI_Comm get_intra_comm(); int is_uniform(); int is_blocked(); MPI_Comm split(int color, int key); void cleanup_smp(); void ref(); static void unref(MPI_Comm comm); static void destroy(MPI_Comm comm); void init_smp(); int add_f() override; static void free_f(int id); static Comm* f2c(int); static int keyval_create(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state); static int keyval_free(int* keyval); static void keyval_cleanup(); void add_rma_win(MPI_Win win); void remove_rma_win(MPI_Win win); void finish_rma_calls(); }; } } #endif SimGrid-3.18/src/smpi/mpi/0000755000175000017500000000000013217757320015642 5ustar mquinsonmquinsonSimGrid-3.18/src/smpi/mpi/smpi_win.cpp0000644000175000017500000005205313217757320020200 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_win.hpp" #include "private.hpp" #include "smpi_coll.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_info.hpp" #include "smpi_keyvals.hpp" #include "smpi_process.hpp" #include "smpi_request.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_rma, smpi, "Logging specific to SMPI (RMA operations)"); namespace simgrid{ namespace smpi{ std::unordered_map Win::keyvals_; int Win::keyval_id_=0; Win::Win(void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, int allocated, int dynamic): base_(base), size_(size), disp_unit_(disp_unit), assert_(0), info_(info), comm_(comm), allocated_(allocated), dynamic_(dynamic){ int comm_size = comm->size(); rank_ = comm->rank(); XBT_DEBUG("Creating window"); if(info!=MPI_INFO_NULL) info->ref(); name_ = nullptr; opened_ = 0; group_ = MPI_GROUP_NULL; requests_ = new std::vector(); mut_=xbt_mutex_init(); lock_mut_=xbt_mutex_init(); atomic_mut_=xbt_mutex_init(); connected_wins_ = new MPI_Win[comm_size]; connected_wins_[rank_] = this; count_ = 0; if(rank_==0){ bar_ = MSG_barrier_init(comm_size); } mode_=0; comm->add_rma_win(this); Colls::allgather(&(connected_wins_[rank_]), sizeof(MPI_Win), MPI_BYTE, connected_wins_, sizeof(MPI_Win), MPI_BYTE, comm); Colls::bcast(&(bar_), sizeof(msg_bar_t), MPI_BYTE, 0, comm); Colls::barrier(comm); } Win::~Win(){ //As per the standard, perform a barrier to ensure every async comm is finished MSG_barrier_wait(bar_); int finished = finish_comms(); XBT_DEBUG("Win destructor - Finished %d RMA calls", finished); delete requests_; delete[] connected_wins_; if (name_ != nullptr){ xbt_free(name_); } if(info_!=MPI_INFO_NULL){ MPI_Info_free(&info_); } comm_->remove_rma_win(this); Colls::barrier(comm_); int rank=comm_->rank(); if(rank == 0) MSG_barrier_destroy(bar_); xbt_mutex_destroy(mut_); xbt_mutex_destroy(lock_mut_); xbt_mutex_destroy(atomic_mut_); if(allocated_ !=0) xbt_free(base_); cleanup_attr(); } int Win::attach (void *base, MPI_Aint size){ if (not(base_ == MPI_BOTTOM || base_ == 0)) return MPI_ERR_ARG; base_=0;//actually the address will be given in the RMA calls, as being the disp. size_+=size; return MPI_SUCCESS; } int Win::detach (void *base){ base_=MPI_BOTTOM; size_=-1; return MPI_SUCCESS; } void Win::get_name(char* name, int* length){ if(name_==nullptr){ *length=0; name=nullptr; return; } *length = strlen(name_); strncpy(name, name_, *length+1); } void Win::get_group(MPI_Group* group){ if(comm_ != MPI_COMM_NULL){ *group = comm_->group(); } else { *group = MPI_GROUP_NULL; } } MPI_Info Win::info(){ if(info_== MPI_INFO_NULL) info_ = new Info(); info_->ref(); return info_; } int Win::rank(){ return rank_; } MPI_Aint Win::size(){ return size_; } void* Win::base(){ return base_; } int Win::disp_unit(){ return disp_unit_; } int Win::dynamic(){ return dynamic_; } void Win::set_info(MPI_Info info){ if(info_!= MPI_INFO_NULL) info->ref(); info_=info; } void Win::set_name(char* name){ name_ = xbt_strdup(name); } int Win::fence(int assert) { XBT_DEBUG("Entering fence"); if (opened_ == 0) opened_=1; if (assert != MPI_MODE_NOPRECEDE) { // This is not the first fence => finalize what came before MSG_barrier_wait(bar_); xbt_mutex_acquire(mut_); // This (simulated) mutex ensures that no process pushes to the vector of requests during the waitall. // Without this, the vector could get redimensionned when another process pushes. // This would result in the array used by Request::waitall() to be invalidated. // Another solution would be to copy the data and cleanup the vector *before* Request::waitall std::vector *reqs = requests_; int size = static_cast(reqs->size()); // start all requests that have been prepared by another process if (size > 0) { MPI_Request* treqs = &(*reqs)[0]; Request::waitall(size, treqs, MPI_STATUSES_IGNORE); } count_=0; xbt_mutex_release(mut_); } if(assert==MPI_MODE_NOSUCCEED)//there should be no ops after this one, tell we are closed. opened_=0; assert_ = assert; MSG_barrier_wait(bar_); XBT_DEBUG("Leaving fence"); return MPI_SUCCESS; } int Win::put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Request* request) { //get receiver pointer MPI_Win recv_win = connected_wins_[target_rank]; if(opened_==0){//check that post/start has been done // no fence or start .. lock ok ? int locked=0; for (auto const& it : recv_win->lockers_) if (it == comm_->rank()) locked = 1; if(locked != 1) return MPI_ERR_WIN; } if(target_count*target_datatype->get_extent()>recv_win->size_) return MPI_ERR_ARG; void* recv_addr = static_cast ( static_cast(recv_win->base_) + target_disp * recv_win->disp_unit_); XBT_DEBUG("Entering MPI_Put to %d", target_rank); if(target_rank != comm_->rank()){ //prepare send_request MPI_Request sreq = Request::rma_send_init(origin_addr, origin_count, origin_datatype, smpi_process()->index(), comm_->group()->index(target_rank), SMPI_RMA_TAG+1, comm_, MPI_OP_NULL); //prepare receiver request MPI_Request rreq = Request::rma_recv_init(recv_addr, target_count, target_datatype, smpi_process()->index(), comm_->group()->index(target_rank), SMPI_RMA_TAG+1, recv_win->comm_, MPI_OP_NULL); //start send sreq->start(); if(request!=nullptr){ *request=sreq; }else{ xbt_mutex_acquire(mut_); requests_->push_back(sreq); xbt_mutex_release(mut_); } //push request to receiver's win xbt_mutex_acquire(recv_win->mut_); recv_win->requests_->push_back(rreq); rreq->start(); xbt_mutex_release(recv_win->mut_); }else{ Datatype::copy(origin_addr, origin_count, origin_datatype, recv_addr, target_count, target_datatype); if(request!=nullptr) *request = MPI_REQUEST_NULL; } return MPI_SUCCESS; } int Win::get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Request* request) { //get sender pointer MPI_Win send_win = connected_wins_[target_rank]; if(opened_==0){//check that post/start has been done // no fence or start .. lock ok ? int locked=0; for (auto const& it : send_win->lockers_) if (it == comm_->rank()) locked = 1; if(locked != 1) return MPI_ERR_WIN; } if(target_count*target_datatype->get_extent()>send_win->size_) return MPI_ERR_ARG; void* send_addr = static_cast(static_cast(send_win->base_) + target_disp * send_win->disp_unit_); XBT_DEBUG("Entering MPI_Get from %d", target_rank); if(target_rank != comm_->rank()){ //prepare send_request MPI_Request sreq = Request::rma_send_init(send_addr, target_count, target_datatype, comm_->group()->index(target_rank), smpi_process()->index(), SMPI_RMA_TAG+2, send_win->comm_, MPI_OP_NULL); //prepare receiver request MPI_Request rreq = Request::rma_recv_init(origin_addr, origin_count, origin_datatype, comm_->group()->index(target_rank), smpi_process()->index(), SMPI_RMA_TAG+2, comm_, MPI_OP_NULL); //start the send, with another process than us as sender. sreq->start(); //push request to receiver's win xbt_mutex_acquire(send_win->mut_); send_win->requests_->push_back(sreq); xbt_mutex_release(send_win->mut_); //start recv rreq->start(); if(request!=nullptr){ *request=rreq; }else{ xbt_mutex_acquire(mut_); requests_->push_back(rreq); xbt_mutex_release(mut_); } }else{ Datatype::copy(send_addr, target_count, target_datatype, origin_addr, origin_count, origin_datatype); if(request!=nullptr) *request=MPI_REQUEST_NULL; } return MPI_SUCCESS; } int Win::accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Request* request) { //get receiver pointer MPI_Win recv_win = connected_wins_[target_rank]; if(opened_==0){//check that post/start has been done // no fence or start .. lock ok ? int locked=0; for (auto const& it : recv_win->lockers_) if (it == comm_->rank()) locked = 1; if(locked != 1) return MPI_ERR_WIN; } //FIXME: local version if(target_count*target_datatype->get_extent()>recv_win->size_) return MPI_ERR_ARG; void* recv_addr = static_cast(static_cast(recv_win->base_) + target_disp * recv_win->disp_unit_); XBT_DEBUG("Entering MPI_Accumulate to %d", target_rank); //As the tag will be used for ordering of the operations, substract count from it (to avoid collisions with other SMPI tags, SMPI_RMA_TAG is set below all the other ones we use ) //prepare send_request MPI_Request sreq = Request::rma_send_init(origin_addr, origin_count, origin_datatype, smpi_process()->index(), comm_->group()->index(target_rank), SMPI_RMA_TAG-3-count_, comm_, op); //prepare receiver request MPI_Request rreq = Request::rma_recv_init(recv_addr, target_count, target_datatype, smpi_process()->index(), comm_->group()->index(target_rank), SMPI_RMA_TAG-3-count_, recv_win->comm_, op); count_++; //start send sreq->start(); //push request to receiver's win xbt_mutex_acquire(recv_win->mut_); recv_win->requests_->push_back(rreq); rreq->start(); xbt_mutex_release(recv_win->mut_); if(request!=nullptr){ *request=sreq; }else{ xbt_mutex_acquire(mut_); requests_->push_back(sreq); xbt_mutex_release(mut_); } return MPI_SUCCESS; } int Win::get_accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, void *result_addr, int result_count, MPI_Datatype result_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Request* request){ //get sender pointer MPI_Win send_win = connected_wins_[target_rank]; if(opened_==0){//check that post/start has been done // no fence or start .. lock ok ? int locked=0; for (auto const& it : send_win->lockers_) if (it == comm_->rank()) locked = 1; if(locked != 1) return MPI_ERR_WIN; } if(target_count*target_datatype->get_extent()>send_win->size_) return MPI_ERR_ARG; XBT_DEBUG("Entering MPI_Get_accumulate from %d", target_rank); //need to be sure ops are correctly ordered, so finish request here ? slow. MPI_Request req; xbt_mutex_acquire(send_win->atomic_mut_); get(result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, &req); if (req != MPI_REQUEST_NULL) Request::wait(&req, MPI_STATUS_IGNORE); if(op!=MPI_NO_OP) accumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, &req); if (req != MPI_REQUEST_NULL) Request::wait(&req, MPI_STATUS_IGNORE); xbt_mutex_release(send_win->atomic_mut_); return MPI_SUCCESS; } int Win::compare_and_swap(void *origin_addr, void *compare_addr, void *result_addr, MPI_Datatype datatype, int target_rank, MPI_Aint target_disp){ //get sender pointer MPI_Win send_win = connected_wins_[target_rank]; if(opened_==0){//check that post/start has been done // no fence or start .. lock ok ? int locked=0; for (auto const& it : send_win->lockers_) if (it == comm_->rank()) locked = 1; if(locked != 1) return MPI_ERR_WIN; } XBT_DEBUG("Entering MPI_Compare_and_swap with %d", target_rank); MPI_Request req = MPI_REQUEST_NULL; xbt_mutex_acquire(send_win->atomic_mut_); get(result_addr, 1, datatype, target_rank, target_disp, 1, datatype, &req); if (req != MPI_REQUEST_NULL) Request::wait(&req, MPI_STATUS_IGNORE); if (not memcmp(result_addr, compare_addr, datatype->get_extent())) { put(origin_addr, 1, datatype, target_rank, target_disp, 1, datatype); } xbt_mutex_release(send_win->atomic_mut_); return MPI_SUCCESS; } int Win::start(MPI_Group group, int assert){ /* From MPI forum advices The call to MPI_WIN_COMPLETE does not return until the put call has completed at the origin; and the target window will be accessed by the put operation only after the call to MPI_WIN_START has matched a call to MPI_WIN_POST by the target process. This still leaves much choice to implementors. The call to MPI_WIN_START can block until the matching call to MPI_WIN_POST occurs at all target processes. One can also have implementations where the call to MPI_WIN_START is nonblocking, but the call to MPI_PUT blocks until the matching call to MPI_WIN_POST occurred; or implementations where the first two calls are nonblocking, but the call to MPI_WIN_COMPLETE blocks until the call to MPI_WIN_POST occurred; or even implementations where all three calls can complete before any target process called MPI_WIN_POST --- the data put must be buffered, in this last case, so as to allow the put to complete at the origin ahead of its completion at the target. However, once the call to MPI_WIN_POST is issued, the sequence above must complete, without further dependencies. */ //naive, blocking implementation. int i = 0; int j = 0; int size = group->size(); MPI_Request* reqs = xbt_new0(MPI_Request, size); while (j != size) { int src = group->index(j); if (src != smpi_process()->index() && src != MPI_UNDEFINED) { reqs[i] = Request::irecv_init(nullptr, 0, MPI_CHAR, src, SMPI_RMA_TAG + 4, MPI_COMM_WORLD); i++; } j++; } size=i; Request::startall(size, reqs); Request::waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;iref(); return MPI_SUCCESS; } int Win::post(MPI_Group group, int assert){ //let's make a synchronous send here int i = 0; int j = 0; int size = group->size(); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int dst=group->index(j); if(dst!=smpi_process()->index() && dst!=MPI_UNDEFINED){ reqs[i]=Request::send_init(nullptr, 0, MPI_CHAR, dst, SMPI_RMA_TAG+4, MPI_COMM_WORLD); i++; } j++; } size=i; Request::startall(size, reqs); Request::waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;iref(); return MPI_SUCCESS; } int Win::complete(){ if(opened_==0) xbt_die("Complete called on already opened MPI_Win"); XBT_DEBUG("Entering MPI_Win_Complete"); int i = 0; int j = 0; int size = group_->size(); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int dst=group_->index(j); if(dst!=smpi_process()->index() && dst!=MPI_UNDEFINED){ reqs[i]=Request::send_init(nullptr, 0, MPI_CHAR, dst, SMPI_RMA_TAG+5, MPI_COMM_WORLD); i++; } j++; } size=i; XBT_DEBUG("Win_complete - Sending sync messages to %d processes", size); Request::startall(size, reqs); Request::waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;isize(); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int src=group_->index(j); if(src!=smpi_process()->index() && src!=MPI_UNDEFINED){ reqs[i]=Request::irecv_init(nullptr, 0, MPI_CHAR, src,SMPI_RMA_TAG+5, MPI_COMM_WORLD); i++; } j++; } size=i; XBT_DEBUG("Win_wait - Receiving sync messages from %d processes", size); Request::startall(size, reqs); Request::waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;imode_ != MPI_LOCK_SHARED)|| target_win->mode_ == MPI_LOCK_EXCLUSIVE){ xbt_mutex_acquire(target_win->lock_mut_); target_win->mode_+= lock_type;//add the lock_type to differentiate case when we are switching from EXCLUSIVE to SHARED (no release needed in the unlock) if(lock_type == MPI_LOCK_SHARED){//the window used to be exclusive, it's now shared. xbt_mutex_release(target_win->lock_mut_); } } else if (not(target_win->mode_ == MPI_LOCK_SHARED && lock_type == MPI_LOCK_EXCLUSIVE)) target_win->mode_ += lock_type; // don't set to exclusive if it's already shared target_win->lockers_.push_back(comm_->rank()); int finished = finish_comms(rank); XBT_DEBUG("Win_lock %d - Finished %d RMA calls", rank, finished); finished = target_win->finish_comms(rank_); XBT_DEBUG("Win_lock target %d - Finished %d RMA calls", rank, finished); return MPI_SUCCESS; } int Win::lock_all(int assert){ int i=0; int retval = MPI_SUCCESS; for (i=0; isize();i++){ int ret = this->lock(MPI_LOCK_SHARED, i, assert); if(ret != MPI_SUCCESS) retval = ret; } return retval; } int Win::unlock(int rank){ MPI_Win target_win = connected_wins_[rank]; int target_mode = target_win->mode_; target_win->mode_= 0; target_win->lockers_.remove(comm_->rank()); if (target_mode==MPI_LOCK_EXCLUSIVE){ xbt_mutex_release(target_win->lock_mut_); } int finished = finish_comms(rank); XBT_DEBUG("Win_unlock %d - Finished %d RMA calls", rank, finished); finished = target_win->finish_comms(rank_); XBT_DEBUG("Win_unlock target %d - Finished %d RMA calls", rank, finished); return MPI_SUCCESS; } int Win::unlock_all(){ int i=0; int retval = MPI_SUCCESS; for (i=0; isize();i++){ int ret = this->unlock(i); if(ret != MPI_SUCCESS) retval = ret; } return retval; } int Win::flush(int rank){ MPI_Win target_win = connected_wins_[rank]; int finished = finish_comms(rank); XBT_DEBUG("Win_flush on local %d - Finished %d RMA calls", rank_, finished); finished = target_win->finish_comms(rank_); XBT_DEBUG("Win_flush on remote %d - Finished %d RMA calls", rank, finished); return MPI_SUCCESS; } int Win::flush_local(int rank){ int finished = finish_comms(rank); XBT_DEBUG("Win_flush_local for rank %d - Finished %d RMA calls", rank, finished); return MPI_SUCCESS; } int Win::flush_all(){ int i=0; int finished = 0; finished = finish_comms(); XBT_DEBUG("Win_flush_all on local - Finished %d RMA calls", finished); for (i=0; isize();i++){ finished = connected_wins_[i]->finish_comms(rank_); XBT_DEBUG("Win_flush_all on %d - Finished %d RMA calls", i, finished); } return MPI_SUCCESS; } int Win::flush_local_all(){ int finished = finish_comms(); XBT_DEBUG("Win_flush_local_all - Finished %d RMA calls", finished); return MPI_SUCCESS; } Win* Win::f2c(int id){ return static_cast(F2C::f2c(id)); } int Win::finish_comms(){ xbt_mutex_acquire(mut_); //Finish own requests std::vector *reqqs = requests_; int size = static_cast(reqqs->size()); if (size > 0) { MPI_Request* treqs = &(*reqqs)[0]; Request::waitall(size, treqs, MPI_STATUSES_IGNORE); reqqs->clear(); } xbt_mutex_release(mut_); return size; } int Win::finish_comms(int rank){ xbt_mutex_acquire(mut_); //Finish own requests std::vector *reqqs = requests_; int size = static_cast(reqqs->size()); if (size > 0) { size = 0; std::vector myreqqs; std::vector::iterator iter = reqqs->begin(); while (iter != reqqs->end()){ if(((*iter)!=MPI_REQUEST_NULL) && (((*iter)->src() == rank) || ((*iter)->dst() == rank))){ myreqqs.push_back(*iter); iter = reqqs->erase(iter); size++; } else { ++iter; } } if(size >0){ MPI_Request* treqs = &myreqqs[0]; Request::waitall(size, treqs, MPI_STATUSES_IGNORE); myreqqs.clear(); } } xbt_mutex_release(mut_); return size; } } } SimGrid-3.18/src/smpi/mpi/smpi_f2c.cpp0000644000175000017500000000377313217757320020062 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_f2c.hpp" #include "private.hpp" #include "smpi_process.hpp" #include namespace simgrid{ namespace smpi{ std::unordered_map* F2C::f2c_lookup_ = nullptr; int F2C::f2c_id_ = 0; std::unordered_map* F2C::f2c_lookup() { return f2c_lookup_; } void F2C::set_f2c_lookup(std::unordered_map* map) { f2c_lookup_ = map; } void F2C::f2c_id_increment(){ f2c_id_++; }; int F2C::f2c_id(){ return f2c_id_; }; char* F2C::get_key(char* key, int id) { std::snprintf(key, KEY_SIZE, "%x", static_cast(id)); return key; } char* F2C::get_key_id(char* key, int id) { std::snprintf(key, KEY_SIZE, "%x_%d", static_cast(id), smpi_process()->index()); return key; } void F2C::delete_lookup(){ delete f2c_lookup_; } std::unordered_map* F2C::lookup() { return f2c_lookup_; } void F2C::free_f(int id) { char key[KEY_SIZE]; f2c_lookup_->erase(get_key(key, id)); } int F2C::add_f() { if (f2c_lookup_ == nullptr) f2c_lookup_ = new std::unordered_map; char key[KEY_SIZE]; (*f2c_lookup_)[get_key(key, f2c_id_)] = this; f2c_id_increment(); return f2c_id_-1; } int F2C::c2f() { if (f2c_lookup_ == nullptr) { f2c_lookup_ = new std::unordered_map; } for (auto const& elm : *f2c_lookup_) if (elm.second == this) return std::stoi(elm.first); /* this function wasn't found, add it */ return this->add_f(); } F2C* F2C::f2c(int id) { if (f2c_lookup_ == nullptr) f2c_lookup_ = new std::unordered_map; if(id >= 0){ char key[KEY_SIZE]; auto comm = f2c_lookup_->find(get_key(key, id)); return comm == f2c_lookup_->end() ? nullptr : comm->second; }else return nullptr; } } } SimGrid-3.18/src/smpi/mpi/smpi_datatype.cpp0000644000175000017500000004013413217757320021213 0ustar mquinsonmquinson/* smpi_datatype.cpp -- MPI primitives to handle datatypes */ /* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/modelchecker.h" #include "private.hpp" #include "smpi_datatype_derived.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_datatype, smpi, "Logging specific to SMPI (datatype)"); #define CREATE_MPI_DATATYPE(name, type) \ static simgrid::smpi::Datatype mpi_##name ( \ (char*) # name, \ sizeof(type), /* size */ \ 0, /* lb */ \ sizeof(type), /* ub = lb + size */ \ DT_FLAG_BASIC /* flags */ \ ); \ const MPI_Datatype name = &mpi_##name; #define CREATE_MPI_DATATYPE_NULL(name) \ static simgrid::smpi::Datatype mpi_##name ( \ (char*) # name, \ 0, /* size */ \ 0, /* lb */ \ 0, /* ub = lb + size */ \ DT_FLAG_BASIC /* flags */ \ ); \ const MPI_Datatype name = &mpi_##name; // Predefined data types CREATE_MPI_DATATYPE(MPI_CHAR, char); CREATE_MPI_DATATYPE(MPI_SHORT, short); CREATE_MPI_DATATYPE(MPI_INT, int); CREATE_MPI_DATATYPE(MPI_LONG, long); CREATE_MPI_DATATYPE(MPI_LONG_LONG, long long); CREATE_MPI_DATATYPE(MPI_SIGNED_CHAR, signed char); CREATE_MPI_DATATYPE(MPI_UNSIGNED_CHAR, unsigned char); CREATE_MPI_DATATYPE(MPI_UNSIGNED_SHORT, unsigned short); CREATE_MPI_DATATYPE(MPI_UNSIGNED, unsigned int); CREATE_MPI_DATATYPE(MPI_UNSIGNED_LONG, unsigned long); CREATE_MPI_DATATYPE(MPI_UNSIGNED_LONG_LONG, unsigned long long); CREATE_MPI_DATATYPE(MPI_FLOAT, float); CREATE_MPI_DATATYPE(MPI_DOUBLE, double); CREATE_MPI_DATATYPE(MPI_LONG_DOUBLE, long double); CREATE_MPI_DATATYPE(MPI_WCHAR, wchar_t); CREATE_MPI_DATATYPE(MPI_C_BOOL, bool); CREATE_MPI_DATATYPE(MPI_BYTE, int8_t); CREATE_MPI_DATATYPE(MPI_INT8_T, int8_t); CREATE_MPI_DATATYPE(MPI_INT16_T, int16_t); CREATE_MPI_DATATYPE(MPI_INT32_T, int32_t); CREATE_MPI_DATATYPE(MPI_INT64_T, int64_t); CREATE_MPI_DATATYPE(MPI_UINT8_T, uint8_t); CREATE_MPI_DATATYPE(MPI_UINT16_T, uint16_t); CREATE_MPI_DATATYPE(MPI_UINT32_T, uint32_t); CREATE_MPI_DATATYPE(MPI_UINT64_T, uint64_t); CREATE_MPI_DATATYPE(MPI_C_FLOAT_COMPLEX, float _Complex); CREATE_MPI_DATATYPE(MPI_C_DOUBLE_COMPLEX, double _Complex); CREATE_MPI_DATATYPE(MPI_C_LONG_DOUBLE_COMPLEX, long double _Complex); CREATE_MPI_DATATYPE(MPI_AINT, MPI_Aint); CREATE_MPI_DATATYPE(MPI_OFFSET, MPI_Offset); CREATE_MPI_DATATYPE(MPI_FLOAT_INT, float_int); CREATE_MPI_DATATYPE(MPI_LONG_INT, long_int); CREATE_MPI_DATATYPE(MPI_DOUBLE_INT, double_int); CREATE_MPI_DATATYPE(MPI_SHORT_INT, short_int); CREATE_MPI_DATATYPE(MPI_2INT, int_int); CREATE_MPI_DATATYPE(MPI_2FLOAT, float_float); CREATE_MPI_DATATYPE(MPI_2DOUBLE, double_double); CREATE_MPI_DATATYPE(MPI_2LONG, long_long); CREATE_MPI_DATATYPE(MPI_REAL, float); CREATE_MPI_DATATYPE(MPI_REAL4, float); CREATE_MPI_DATATYPE(MPI_REAL8, float); CREATE_MPI_DATATYPE(MPI_REAL16, double); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX8); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX16); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX32); CREATE_MPI_DATATYPE(MPI_INTEGER1, int); CREATE_MPI_DATATYPE(MPI_INTEGER2, int16_t); CREATE_MPI_DATATYPE(MPI_INTEGER4, int32_t); CREATE_MPI_DATATYPE(MPI_INTEGER8, int64_t); CREATE_MPI_DATATYPE(MPI_INTEGER16, integer128_t); CREATE_MPI_DATATYPE(MPI_LONG_DOUBLE_INT, long_double_int); CREATE_MPI_DATATYPE_NULL(MPI_UB); CREATE_MPI_DATATYPE_NULL(MPI_LB); CREATE_MPI_DATATYPE(MPI_PACKED, char); // Internal use only CREATE_MPI_DATATYPE(MPI_PTR, void*); namespace simgrid{ namespace smpi{ std::unordered_map Datatype::keyvals_; int Datatype::keyval_id_=0; Datatype::Datatype(int size,MPI_Aint lb, MPI_Aint ub, int flags) : name_(nullptr), size_(size), lb_(lb), ub_(ub), flags_(flags), refcount_(1){ #if SIMGRID_HAVE_MC if(MC_is_active()) MC_ignore(&(refcount_), sizeof(refcount_)); #endif } //for predefined types, so in_use = 0. Datatype::Datatype(char* name, int size,MPI_Aint lb, MPI_Aint ub, int flags) : name_(name), size_(size), lb_(lb), ub_(ub), flags_(flags), refcount_(0){ #if SIMGRID_HAVE_MC if(MC_is_active()) MC_ignore(&(refcount_), sizeof(refcount_)); #endif } Datatype::Datatype(Datatype *datatype, int* ret) : name_(nullptr), lb_(datatype->lb_), ub_(datatype->ub_), flags_(datatype->flags_), refcount_(1) { flags_ &= ~DT_FLAG_PREDEFINED; *ret = MPI_SUCCESS; if(datatype->name_) name_ = xbt_strdup(datatype->name_); if (not datatype->attributes()->empty()) { int flag; void* value_out; for(auto it = datatype->attributes()->begin(); it != datatype->attributes()->end(); it++){ smpi_key_elem elem = keyvals_.at((*it).first); if (elem != nullptr && elem->copy_fn.type_copy_fn != MPI_NULL_COPY_FN) { *ret = elem->copy_fn.type_copy_fn(datatype, (*it).first, nullptr, (*it).second, &value_out, &flag); if (*ret != MPI_SUCCESS) { break; } if (flag){ elem->refcount++; attributes()->insert({(*it).first, value_out}); } } } } } Datatype::~Datatype(){ xbt_assert(refcount_ >= 0); if(flags_ & DT_FLAG_PREDEFINED) return; //if still used, mark for deletion if(refcount_!=0){ flags_ |=DT_FLAG_DESTROYED; return; } cleanup_attr(); xbt_free(name_); } void Datatype::ref(){ refcount_++; #if SIMGRID_HAVE_MC if(MC_is_active()) MC_ignore(&(refcount_), sizeof(refcount_)); #endif } void Datatype::unref(MPI_Datatype datatype) { if (datatype->refcount_ > 0) datatype->refcount_--; if (datatype->refcount_ == 0 && not(datatype->flags_ & DT_FLAG_PREDEFINED)) delete datatype; #if SIMGRID_HAVE_MC if(MC_is_active()) MC_ignore(&(datatype->refcount_), sizeof(datatype->refcount_)); #endif } void Datatype::commit() { flags_ |= DT_FLAG_COMMITED; } bool Datatype::is_valid(){ return (flags_ & DT_FLAG_COMMITED); } bool Datatype::is_basic() { return (flags_ & DT_FLAG_BASIC); } bool Datatype::is_replayable() { return ((this==MPI_BYTE)||(this==MPI_DOUBLE)||(this==MPI_INT)|| (this==MPI_CHAR)||(this==MPI_SHORT)||(this==MPI_LONG)||(this==MPI_FLOAT)); } size_t Datatype::size(){ return size_; } int Datatype::flags(){ return flags_; } int Datatype::refcount(){ return refcount_; } void Datatype::addflag(int flag){ flags_ &= flag; } MPI_Aint Datatype::lb(){ return lb_; } MPI_Aint Datatype::ub(){ return ub_; } char* Datatype::name(){ return name_; } int Datatype::extent(MPI_Aint * lb, MPI_Aint * extent){ *lb = lb_; *extent = ub_ - lb_; return MPI_SUCCESS; } MPI_Aint Datatype::get_extent(){ return ub_ - lb_; } void Datatype::get_name(char* name, int* length){ *length = strlen(name_); strncpy(name, name_, *length+1); } void Datatype::set_name(char* name){ if(name_!=nullptr && (flags_ & DT_FLAG_PREDEFINED) == 0) xbt_free(name_); name_ = xbt_strdup(name); } int Datatype::pack(void* inbuf, int incount, void* outbuf, int outcount, int* position,MPI_Comm comm){ if (outcount - *position < incount*static_cast(size_)) return MPI_ERR_BUFFER; Datatype::copy(inbuf, incount, this, static_cast(outbuf) + *position, outcount, MPI_CHAR); *position += incount * size_; return MPI_SUCCESS; } int Datatype::unpack(void* inbuf, int insize, int* position, void* outbuf, int outcount,MPI_Comm comm){ if (outcount*static_cast(size_)> insize) return MPI_ERR_BUFFER; Datatype::copy(static_cast(inbuf) + *position, insize, MPI_CHAR, outbuf, outcount, this); *position += outcount * size_; return MPI_SUCCESS; } int Datatype::copy(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype){ // FIXME Handle the case of a partial shared malloc. if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ smpi_switch_data_segment(smpi_process()->index()); } /* First check if we really have something to do */ if (recvcount > 0 && recvbuf != sendbuf) { sendcount *= sendtype->size(); recvcount *= recvtype->size(); int count = sendcount < recvcount ? sendcount : recvcount; if (not(sendtype->flags() & DT_FLAG_DERIVED) && not(recvtype->flags() & DT_FLAG_DERIVED)) { if (not smpi_process()->replaying()) memcpy(recvbuf, sendbuf, count); } else if (not(sendtype->flags() & DT_FLAG_DERIVED)) { recvtype->unserialize(sendbuf, recvbuf, count / recvtype->size(), MPI_REPLACE); } else if (not(recvtype->flags() & DT_FLAG_DERIVED)) { sendtype->serialize(sendbuf, recvbuf, count / sendtype->size()); }else{ void * buf_tmp = xbt_malloc(count); sendtype->serialize( sendbuf, buf_tmp,count/sendtype->size()); recvtype->unserialize( buf_tmp, recvbuf,count/recvtype->size(), MPI_REPLACE); xbt_free(buf_tmp); } } return sendcount > recvcount ? MPI_ERR_TRUNCATE : MPI_SUCCESS; } //Default serialization method : memcpy. void Datatype::serialize( void* noncontiguous_buf, void *contiguous_buf, int count){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+lb_; memcpy(contiguous_buf_char, noncontiguous_buf_char, count*size_); } void Datatype::unserialize( void* contiguous_buf, void *noncontiguous_buf, int count, MPI_Op op){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+lb_; int n=count; if(op!=MPI_OP_NULL) op->apply( contiguous_buf_char, noncontiguous_buf_char, &n, this); } int Datatype::create_contiguous(int count, MPI_Datatype old_type, MPI_Aint lb, MPI_Datatype* new_type){ if(old_type->flags_ & DT_FLAG_DERIVED){ //handle this case as a hvector with stride equals to the extent of the datatype return create_hvector(count, 1, old_type->get_extent(), old_type, new_type); } if(count>0) *new_type = new Type_Contiguous(count * old_type->size(), lb, lb + count * old_type->size(), DT_FLAG_DERIVED, count, old_type); else *new_type = new Datatype(count * old_type->size(), lb, lb + count * old_type->size(),0); return MPI_SUCCESS; } int Datatype::create_vector(int count, int block_length, int stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval; if (block_length<0) return MPI_ERR_ARG; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=old_type->lb(); ub=((count-1)*stride+block_length-1)*old_type->get_extent()+old_type->ub(); } if(old_type->flags() & DT_FLAG_DERIVED || stride != block_length){ *new_type = new Type_Vector(count * (block_length) * old_type->size(), lb, ub, DT_FLAG_DERIVED, count, block_length, stride, old_type); retval=MPI_SUCCESS; }else{ /* in this situation the data are contiguous thus it's not required to serialize and unserialize it*/ *new_type = new Datatype(count * block_length * old_type->size(), 0, ((count -1) * stride + block_length)* old_type->size(), DT_FLAG_CONTIGUOUS); retval=MPI_SUCCESS; } return retval; } int Datatype::create_hvector(int count, int block_length, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval; if (block_length<0) return MPI_ERR_ARG; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=old_type->lb(); ub=((count-1)*stride)+(block_length-1)*old_type->get_extent()+old_type->ub(); } if(old_type->flags() & DT_FLAG_DERIVED || stride != block_length*old_type->get_extent()){ *new_type = new Type_Hvector(count * (block_length) * old_type->size(), lb, ub, DT_FLAG_DERIVED, count, block_length, stride, old_type); retval=MPI_SUCCESS; }else{ /* in this situation the data are contiguous thus it's not required to serialize and unserialize it*/ *new_type = new Datatype(count * block_length * old_type->size(), 0, count * block_length * old_type->size(), DT_FLAG_CONTIGUOUS); retval=MPI_SUCCESS; } return retval; } int Datatype::create_indexed(int count, int* block_lengths, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type){ int size = 0; bool contiguous=true; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0]*old_type->get_extent(); ub=indices[0]*old_type->get_extent() + block_lengths[0]*old_type->ub(); } for (int i = 0; i < count; i++) { if (block_lengths[i] < 0) return MPI_ERR_ARG; size += block_lengths[i]; if(indices[i]*old_type->get_extent()+old_type->lb()get_extent()+old_type->lb(); if(indices[i]*old_type->get_extent()+block_lengths[i]*old_type->ub()>ub) ub = indices[i]*old_type->get_extent()+block_lengths[i]*old_type->ub(); if ( (i< count -1) && (indices[i]+block_lengths[i] != indices[i+1]) ) contiguous=false; } if(old_type->flags_ & DT_FLAG_DERIVED) contiguous=false; if (not contiguous) { *new_type = new Type_Indexed(size * old_type->size(),lb,ub, DT_FLAG_DERIVED|DT_FLAG_DATA, count, block_lengths, indices, old_type); }else{ Datatype::create_contiguous(size, old_type, lb, new_type); } return MPI_SUCCESS; } int Datatype::create_hindexed(int count, int* block_lengths, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type){ int size = 0; bool contiguous=true; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0] + old_type->lb(); ub=indices[0] + block_lengths[0]*old_type->ub(); } for (int i = 0; i < count; i++) { if (block_lengths[i] < 0) return MPI_ERR_ARG; size += block_lengths[i]; if(indices[i]+old_type->lb()lb(); if(indices[i]+block_lengths[i]*old_type->ub()>ub) ub = indices[i]+block_lengths[i]*old_type->ub(); if ( (i< count -1) && (indices[i]+block_lengths[i]*(static_cast(old_type->size())) != indices[i+1]) ) contiguous=false; } if (old_type->flags_ & DT_FLAG_DERIVED || lb!=0) contiguous=false; if (not contiguous) { *new_type = new Type_Hindexed(size * old_type->size(),lb,ub, DT_FLAG_DERIVED|DT_FLAG_DATA, count, block_lengths, indices, old_type); }else{ Datatype::create_contiguous(size, old_type, lb, new_type); } return MPI_SUCCESS; } int Datatype::create_struct(int count, int* block_lengths, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type){ size_t size = 0; bool contiguous=true; size = 0; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0] + old_types[0]->lb(); ub=indices[0] + block_lengths[0]*old_types[0]->ub(); } bool forced_lb=false; bool forced_ub=false; for (int i = 0; i < count; i++) { if (block_lengths[i]<0) return MPI_ERR_ARG; if (old_types[i]->flags_ & DT_FLAG_DERIVED) contiguous=false; size += block_lengths[i]*old_types[i]->size(); if (old_types[i]==MPI_LB){ lb=indices[i]; forced_lb=true; } if (old_types[i]==MPI_UB){ ub=indices[i]; forced_ub=true; } if (not forced_lb && indices[i] + old_types[i]->lb() < lb) lb = indices[i]; if (not forced_ub && indices[i] + block_lengths[i] * old_types[i]->ub() > ub) ub = indices[i]+block_lengths[i]*old_types[i]->ub(); if ( (i< count -1) && (indices[i]+block_lengths[i]*static_cast(old_types[i]->size()) != indices[i+1]) ) contiguous=false; } if (not contiguous) { *new_type = new Type_Struct(size, lb,ub, DT_FLAG_DERIVED|DT_FLAG_DATA, count, block_lengths, indices, old_types); }else{ Datatype::create_contiguous(size, MPI_CHAR, lb, new_type); } return MPI_SUCCESS; } Datatype* Datatype::f2c(int id){ return static_cast(F2C::f2c(id)); } } } SimGrid-3.18/src/smpi/mpi/smpi_topo.cpp0000644000175000017500000002442113217757320020362 0ustar mquinsonmquinson/* Copyright (c) 2014-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi/smpi.h" #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_topo.hpp" #include "xbt/sysdep.h" #include #include /* static functions */ static int assignnodes(int ndim, int nfactor, int *pfacts,int **pdims); static int getfactors(int num, int *nfators, int **factors); namespace simgrid{ namespace smpi{ Topo_Graph::~Topo_Graph() { delete[] index_; delete[] edges_; } Topo_Dist_Graph::~Topo_Dist_Graph() { delete[] in_; delete[] in_weights_; delete[] out_; delete[] out_weights_; } /******************************************************************************* * Cartesian topologies ******************************************************************************/ Topo_Cart::~Topo_Cart() { delete[] dims_; delete[] periodic_; delete[] position_; } Topo_Cart::Topo_Cart(int ndims) : ndims_(ndims) { dims_ = new int[ndims]; periodic_ = new int[ndims]; position_ = new int[ndims]; } /* reorder is ignored, don't know what would be the consequences of a dumb reordering but neither do I see the point of * reordering*/ Topo_Cart::Topo_Cart(MPI_Comm comm_old, int ndims, int dims[], int periods[], int reorder, MPI_Comm *comm_cart) : Topo_Cart(ndims) { MPI_Group newGroup; MPI_Group oldGroup; int rank = comm_old->rank(); if(ndims != 0) { int newSize = 1; for (int i = 0 ; i < ndims ; i++) { newSize *= dims[i]; } if(rank >= newSize) { *comm_cart = MPI_COMM_NULL; return; } oldGroup = comm_old->group(); newGroup = new Group(newSize); for (int i = 0 ; i < newSize ; i++) { newGroup->set_mapping(oldGroup->index(i), i); } nnodes_ = newSize; // FIXME : code duplication... See coords int nranks = newSize; for (int i=0; igroup()), this); } else { *comm_cart = MPI_COMM_NULL; } } setComm(*comm_cart); } Topo_Cart* Topo_Cart::sub(const int remain_dims[], MPI_Comm *newcomm) { int oldNDims = ndims_; int *newDims = nullptr; int *newPeriodic = nullptr; if (remain_dims == nullptr && oldNDims != 0) { return nullptr; } int newNDims = 0; for (int i = 0 ; i < oldNDims ; i++) { if (remain_dims[i]) newNDims++; } if (newNDims > 0) { newDims = xbt_new(int, newNDims); newPeriodic = xbt_new(int, newNDims); // that should not segfault int j = 0; for (int i = 0 ; j < newNDims ; i++) { if(remain_dims[i]) { newDims[j] =dims_[i]; newPeriodic[j] =periodic_[i]; j++; } } } return new Topo_Cart(getComm(), newNDims, newDims, newPeriodic, 0, newcomm); } int Topo_Cart::coords(int rank, int maxdims, int coords[]) { int nnodes = nnodes_; for (int i = 0; i< ndims_; i++ ) { nnodes = nnodes /dims_[i]; coords[i] = rank / nnodes; rank = rank % nnodes; } return MPI_SUCCESS; } int Topo_Cart::get(int maxdims, int* dims, int* periods, int* coords) { int ndims=ndims_ < maxdims ?ndims_ : maxdims; for(int i = 0 ; i < ndims ; i++) { dims[i] =dims_[i]; periods[i] =periodic_[i]; coords[i] =position_[i]; } return MPI_SUCCESS; } int Topo_Cart::rank(int* coords, int* rank) { int ndims =ndims_; *rank = 0; int multiplier = 1; for (int i=ndims-1; i >=0; i-- ) { int coord = coords[i]; /* The user can give us whatever coordinates he wants. If one of them is out of range, either this dimension is * periodic, and we consider the equivalent coordinate inside the bounds, or it's not and then it's an error */ if (coord >=dims_[i]) { if (periodic_[i] ) { coord = coord %dims_[i]; } else { // Should I do that ? *rank = -1; return MPI_ERR_ARG; } } else if (coord < 0) { if(periodic_[i]) { coord = coord %dims_[i]; if (coord) coord =dims_[i] + coord; } else { *rank = -1; return MPI_ERR_ARG; } } *rank += multiplier * coord; multiplier *=dims_[i]; } return MPI_SUCCESS; } int Topo_Cart::shift(int direction, int disp, int *rank_source, int *rank_dest) { int position[ndims_]; if(ndims_ == 0) { return MPI_ERR_ARG; } if (ndims_ < direction) { return MPI_ERR_DIMS; } this->coords(getComm()->rank(), ndims_, position); position[direction] += disp; if(position[direction] < 0 || position[direction] >=dims_[direction]) { if(periodic_[direction]) { position[direction] %=dims_[direction]; this->rank(position, rank_dest); } else { *rank_dest = MPI_PROC_NULL; } } else { this->rank(position, rank_dest); } position[direction] = position_[direction] - disp; if(position[direction] < 0 || position[direction] >=dims_[direction]) { if(periodic_[direction]) { position[direction] %=dims_[direction]; this->rank(position, rank_source); } else { *rank_source = MPI_PROC_NULL; } } else { this->rank(position, rank_source); } return MPI_SUCCESS; } int Topo_Cart::dim_get(int *ndims) { *ndims =ndims_; return MPI_SUCCESS; } // Everything below has been taken from ompi, but could be easily rewritten (and partially was to follow sonar rules). /* * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2005 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2014 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2012 Los Alamos National Security, LLC. All rights * reserved. * Copyright (c) 2014 Intel, Inc. All rights reserved * $COPYRIGHT$ * * Additional copyrights may follow * * $HEADER$ */ /* * This is a utility function, no need to have anything in the lower layer for this at all */ int Topo_Cart::Dims_create(int nnodes, int ndims, int dims[]) { /* Get # of free-to-be-assigned processes and # of free dimensions */ int freeprocs = nnodes; int freedims = 0; int *p = dims; for (int i = 0; i < ndims; ++i) { if (*p == 0) { ++freedims; } else if ((*p < 0) || ((nnodes % *p) != 0)) { return MPI_ERR_DIMS; } else { freeprocs /= *p; } p++; } if (freedims == 0) { if (freeprocs == 1) { return MPI_SUCCESS; } return MPI_ERR_DIMS; } if (freeprocs == 1) { for (int i = 0; i < ndims; ++i) { if (*dims == 0) { *dims = 1; } dims++; } return MPI_SUCCESS; } /* Factor the number of free processes */ int nfactors; int *factors; int err = getfactors(freeprocs, &nfactors, &factors); if (MPI_SUCCESS != err) return err; /* Assign free processes to free dimensions */ int *procs; err = assignnodes(freedims, nfactors, factors, &procs); if (MPI_SUCCESS != err) return err; /* Return assignment results */ p = procs; for (int i = 0; i < ndims; ++i) { if (*dims == 0) { *dims = *p++; } dims++; } delete[] factors; delete[] procs; /* all done */ return MPI_SUCCESS; } } } /* * assignnodes * * Function: - assign processes to dimensions * - get "best-balanced" grid * - greedy bin-packing algorithm used * - sort dimensions in decreasing order * - dimensions array dynamically allocated * Accepts: - # of dimensions * - # of prime factors * - array of prime factors * - ptr to array of dimensions (returned value) * Returns: - 0 or ERROR */ static int assignnodes(int ndim, int nfactor, int *pfacts, int **pdims) { int *pmin; if (0 >= ndim) { return MPI_ERR_DIMS; } /* Allocate and initialize the bins */ int *bins = new int[ndim]; *pdims = bins; int *p = bins; for (int i = 0 ; i < ndim; ++i) { *p = 1; p++; } /* Loop assigning factors from the highest to the lowest */ for (int j = nfactor - 1; j >= 0; --j) { int f = pfacts[j]; /* Assign a factor to the smallest bin */ pmin = bins; p = pmin + 1; for (int i = 1; i < ndim; ++i) { if (*p < *pmin) { pmin = p; } p++; } *pmin *= f; } /* Sort dimensions in decreasing order (O(n^2) for now) */ pmin = bins; for (int i = 0; i < ndim - 1; ++i) { p = pmin + 1; for (int j = i + 1; j < ndim; ++j) { if (*p > *pmin) { int n = *p; *p = *pmin; *pmin = n; } p++; } pmin++; } return MPI_SUCCESS; } /* * getfactors * * Function: - factorize a number * Accepts: - number * - # prime factors * - array of prime factors * Returns: - MPI_SUCCESS or ERROR */ static int getfactors(int num, int *nfactors, int **factors) { if(num < 2) { (*nfactors) = 0; (*factors) = nullptr; return MPI_SUCCESS; } /* Allocate the array of prime factors which cannot exceed log_2(num) entries */ int sqrtnum = ceil(sqrt(num)); int size = ceil(log(num) / log(2)); *factors = new int[size]; int i = 0; /* determine all occurrences of factor 2 */ while((num % 2) == 0) { num /= 2; (*factors)[i++] = 2; } /* determine all occurrences of uneven prime numbers up to sqrt(num) */ for(int d = 3; (num > 1) && (d < sqrtnum); d += 2) { while((num % d) == 0) { num /= d; (*factors)[i++] = d; } } /* as we looped only up to sqrt(num) one factor > sqrt(num) may be left over */ if(num != 1) { (*factors)[i++] = num; } (*nfactors) = i; return MPI_SUCCESS; } SimGrid-3.18/src/smpi/mpi/smpi_comm.cpp0000644000175000017500000003415713217757320020343 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_comm.hpp" #include "private.hpp" #include "simgrid/s4u/Host.hpp" #include "smpi_coll.hpp" #include "smpi_datatype.hpp" #include "smpi_process.hpp" #include "smpi_request.hpp" #include "smpi_status.hpp" #include "smpi_win.hpp" #include "src/simix/smx_private.hpp" #include #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_comm, smpi, "Logging specific to SMPI (comm)"); simgrid::smpi::Comm mpi_MPI_COMM_UNINITIALIZED; MPI_Comm MPI_COMM_UNINITIALIZED=&mpi_MPI_COMM_UNINITIALIZED; /* Support for cartesian topology was added, but there are 2 other types of topology, graph et dist graph. In order to * support them, we have to add a field SMPI_Topo_type, and replace the MPI_Topology field by an union. */ namespace simgrid{ namespace smpi{ std::unordered_map Comm::keyvals_; int Comm::keyval_id_=0; Comm::Comm(MPI_Group group, MPI_Topology topo) : group_(group), topo_(topo) { refcount_ = 1; topoType_ = MPI_INVALID_TOPO; intra_comm_ = MPI_COMM_NULL; leaders_comm_ = MPI_COMM_NULL; is_uniform_ = 1; non_uniform_map_ = nullptr; leaders_map_ = nullptr; is_blocked_ = 0; } void Comm::destroy(Comm* comm) { if (comm == MPI_COMM_UNINITIALIZED){ Comm::destroy(smpi_process()->comm_world()); return; } delete comm->topo_; // there's no use count on topos Comm::unref(comm); } int Comm::dup(MPI_Comm* newcomm){ if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process()->index()); } MPI_Group cp = new Group(this->group()); (*newcomm) = new Comm(cp, this->topo()); int ret = MPI_SUCCESS; if (not attributes()->empty()) { int flag; void* value_out; for (auto const& it : *attributes()) { smpi_key_elem elem = keyvals_.at(it.first); if (elem != nullptr && elem->copy_fn.comm_copy_fn != MPI_NULL_COPY_FN) { ret = elem->copy_fn.comm_copy_fn(this, it.first, nullptr, it.second, &value_out, &flag); if (ret != MPI_SUCCESS) { Comm::destroy(*newcomm); *newcomm = MPI_COMM_NULL; return ret; } if (flag){ elem->refcount++; (*newcomm)->attributes()->insert({it.first, value_out}); } } } } return ret; } MPI_Group Comm::group() { if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->group(); return group_; } MPI_Topology Comm::topo() { return topo_; } int Comm::size() { if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->size(); return group_->size(); } int Comm::rank() { if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->rank(); return group_->rank(smpi_process()->index()); } void Comm::get_name (char* name, int* len) { if (this == MPI_COMM_UNINITIALIZED){ smpi_process()->comm_world()->get_name(name, len); return; } if(this == MPI_COMM_WORLD) { strncpy(name, "WORLD",5); *len = 5; } else { *len = snprintf(name, MPI_MAX_NAME_STRING, "%p", this); } } void Comm::set_leaders_comm(MPI_Comm leaders){ if (this == MPI_COMM_UNINITIALIZED){ smpi_process()->comm_world()->set_leaders_comm(leaders); return; } leaders_comm_=leaders; } void Comm::set_intra_comm(MPI_Comm leaders){ intra_comm_=leaders; } int* Comm::get_non_uniform_map(){ if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->get_non_uniform_map(); return non_uniform_map_; } int* Comm::get_leaders_map(){ if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->get_leaders_map(); return leaders_map_; } MPI_Comm Comm::get_leaders_comm(){ if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->get_leaders_comm(); return leaders_comm_; } MPI_Comm Comm::get_intra_comm(){ if (this == MPI_COMM_UNINITIALIZED || this==MPI_COMM_WORLD) return smpi_process()->comm_intra(); else return intra_comm_; } int Comm::is_uniform(){ if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->is_uniform(); return is_uniform_; } int Comm::is_blocked(){ if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->is_blocked(); return is_blocked_; } MPI_Comm Comm::split(int color, int key) { if (this == MPI_COMM_UNINITIALIZED) return smpi_process()->comm_world()->split(color, key); int system_tag = 123; int* recvbuf; MPI_Group group_root = nullptr; MPI_Group group_out = nullptr; MPI_Group group = this->group(); int rank = this->rank(); int size = this->size(); /* Gather all colors and keys on rank 0 */ int* sendbuf = xbt_new(int, 2); sendbuf[0] = color; sendbuf[1] = key; if(rank == 0) { recvbuf = xbt_new(int, 2 * size); } else { recvbuf = nullptr; } Coll_gather_default::gather(sendbuf, 2, MPI_INT, recvbuf, 2, MPI_INT, 0, this); xbt_free(sendbuf); /* Do the actual job */ if(rank == 0) { MPI_Group* group_snd = xbt_new(MPI_Group, size); std::vector> rankmap; rankmap.reserve(size); for (int i = 0; i < size; i++) { if (recvbuf[2 * i] != MPI_UNDEFINED) { rankmap.clear(); for (int j = i + 1; j < size; j++) { if(recvbuf[2 * i] == recvbuf[2 * j]) { recvbuf[2 * j] = MPI_UNDEFINED; rankmap.push_back({recvbuf[2 * j + 1], j}); } } /* Add self in the group */ recvbuf[2 * i] = MPI_UNDEFINED; rankmap.push_back({recvbuf[2 * i + 1], i}); std::sort(begin(rankmap), end(rankmap)); group_out = new Group(rankmap.size()); if (i == 0) { group_root = group_out; /* Save root's group */ } for (unsigned j = 0; j < rankmap.size(); j++) { int index = group->index(rankmap[j].second); group_out->set_mapping(index, j); } MPI_Request* requests = xbt_new(MPI_Request, rankmap.size()); int reqs = 0; for (auto const& rank : rankmap) { if (rank.second != 0) { group_snd[reqs]=new Group(group_out); requests[reqs] = Request::isend(&(group_snd[reqs]), 1, MPI_PTR, rank.second, system_tag, this); reqs++; } } if(i != 0 && group_out != MPI_COMM_WORLD->group() && group_out != MPI_GROUP_EMPTY) Group::unref(group_out); Request::waitall(reqs, requests, MPI_STATUS_IGNORE); xbt_free(requests); } } xbt_free(recvbuf); xbt_free(group_snd); group_out = group_root; /* exit with root's group */ } else { if(color != MPI_UNDEFINED) { Request::recv(&group_out, 1, MPI_PTR, 0, system_tag, this, MPI_STATUS_IGNORE); } /* otherwise, exit with group_out == nullptr */ } return group_out!=nullptr ? new Comm(group_out, nullptr) : MPI_COMM_NULL; } void Comm::ref(){ if (this == MPI_COMM_UNINITIALIZED){ smpi_process()->comm_world()->ref(); return; } group_->ref(); refcount_++; } void Comm::cleanup_smp(){ if (intra_comm_ != MPI_COMM_NULL) Comm::unref(intra_comm_); if (leaders_comm_ != MPI_COMM_NULL) Comm::unref(leaders_comm_); if (non_uniform_map_ != nullptr) xbt_free(non_uniform_map_); if (leaders_map_ != nullptr) delete[] leaders_map_; } void Comm::unref(Comm* comm){ if (comm == MPI_COMM_UNINITIALIZED){ Comm::unref(smpi_process()->comm_world()); return; } comm->refcount_--; Group::unref(comm->group_); if(comm->refcount_==0){ comm->cleanup_smp(); comm->cleanup_attr(); delete comm; } } void Comm::init_smp(){ int leader = -1; if (this == MPI_COMM_UNINITIALIZED) smpi_process()->comm_world()->init_smp(); int comm_size = this->size(); // If we are in replay - perform an ugly hack // tell SimGrid we are not in replay for a while, because we need the buffers to be copied for the following calls bool replaying = false; //cache data to set it back again after if(smpi_process()->replaying()){ replaying = true; smpi_process()->set_replaying(false); } if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process()->index()); } //identify neighbours in comm //get the indexes of all processes sharing the same simix host const auto& process_list = sg_host_self()->extension()->process_list; int intra_comm_size = 0; int min_index = INT_MAX; // the minimum index will be the leader for (auto const& actor : process_list) { int index = actor.pid - 1; if (this->group()->rank(index) != MPI_UNDEFINED) { intra_comm_size++; // the process is in the comm if (index < min_index) min_index = index; } } XBT_DEBUG("number of processes deployed on my node : %d", intra_comm_size); MPI_Group group_intra = new Group(intra_comm_size); int i = 0; for (auto const& actor : process_list) { int index = actor.pid - 1; if(this->group()->rank(index)!=MPI_UNDEFINED){ group_intra->set_mapping(index, i); i++; } } MPI_Comm comm_intra = new Comm(group_intra, nullptr); leader=min_index; int* leaders_map = new int[comm_size]; int* leader_list = new int[comm_size]; std::fill_n(leaders_map, comm_size, 0); std::fill_n(leader_list, comm_size, -1); Coll_allgather_mpich::allgather(&leader, 1, MPI_INT , leaders_map, 1, MPI_INT, this); if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process()->index()); } if(leaders_map_==nullptr){ leaders_map_= leaders_map; }else{ delete[] leaders_map; } int j=0; int leader_group_size = 0; for(i=0; iset_mapping(leader_list[i], i); leader_comm = new Comm(leaders_group, nullptr); this->set_leaders_comm(leader_comm); this->set_intra_comm(comm_intra); // create intracommunicator }else{ for (i=0; i< leader_group_size;i++) leaders_group->set_mapping(leader_list[i], i); if(this->get_leaders_comm()==MPI_COMM_NULL){ leader_comm = new Comm(leaders_group, nullptr); this->set_leaders_comm(leader_comm); }else{ leader_comm=this->get_leaders_comm(); Group::unref(leaders_group); } smpi_process()->set_comm_intra(comm_intra); } // Are the nodes uniform ? = same number of process/node int my_local_size=comm_intra->size(); if(comm_intra->rank()==0) { int is_uniform = 1; int* non_uniform_map = xbt_new0(int,leader_group_size); Coll_allgather_mpich::allgather(&my_local_size, 1, MPI_INT, non_uniform_map, 1, MPI_INT, leader_comm); for(i=0; i < leader_group_size; i++) { if(non_uniform_map[0] != non_uniform_map[i]) { is_uniform = 0; break; } } if(is_uniform==0 && this->is_uniform()!=0){ non_uniform_map_ = non_uniform_map; }else{ xbt_free(non_uniform_map); } is_uniform_=is_uniform; } Coll_bcast_mpich::bcast(&(is_uniform_),1, MPI_INT, 0, comm_intra ); if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process()->index()); } // Are the ranks blocked ? = allocated contiguously on the SMP nodes int is_blocked=1; int prev=this->group()->rank(comm_intra->group()->index(0)); for (i = 1; i < my_local_size; i++) { int that = this->group()->rank(comm_intra->group()->index(i)); if (that != prev + 1) { is_blocked = 0; break; } prev = that; } int global_blocked; Coll_allreduce_default::allreduce(&is_blocked, &(global_blocked), 1, MPI_INT, MPI_LAND, this); if(MPI_COMM_WORLD==MPI_COMM_UNINITIALIZED || this==MPI_COMM_WORLD){ if(this->rank()==0){ is_blocked_ = global_blocked; } }else{ is_blocked_=global_blocked; } delete[] leader_list; if(replaying) smpi_process()->set_replaying(true); } MPI_Comm Comm::f2c(int id) { if(id == -2) { return MPI_COMM_SELF; } else if(id==0){ return MPI_COMM_WORLD; } else if(F2C::f2c_lookup() != nullptr && id >= 0) { char key[KEY_SIZE]; const auto& lookup = F2C::f2c_lookup(); auto comm = lookup->find(get_key_id(key, id)); return comm == lookup->end() ? MPI_COMM_NULL : static_cast(comm->second); } else { return MPI_COMM_NULL; } } void Comm::free_f(int id) { char key[KEY_SIZE]; F2C::f2c_lookup()->erase(id == 0 ? get_key(key, id) : get_key_id(key, id)); } int Comm::add_f() { if(F2C::f2c_lookup()==nullptr){ F2C::set_f2c_lookup(new std::unordered_map); } char key[KEY_SIZE]; (*(F2C::f2c_lookup()))[this == MPI_COMM_WORLD ? get_key(key, F2C::f2c_id()) : get_key_id(key, F2C::f2c_id())] = this; f2c_id_increment(); return F2C::f2c_id()-1; } void Comm::add_rma_win(MPI_Win win){ rma_wins_.push_back(win); } void Comm::remove_rma_win(MPI_Win win){ rma_wins_.remove(win); } void Comm::finish_rma_calls(){ for (auto const& it : rma_wins_) { if(it->rank()==this->rank()){//is it ours (for MPI_COMM_WORLD)? int finished = it->finish_comms(); XBT_DEBUG("Barrier for rank %d - Finished %d RMA calls",this->rank(), finished); } } } } } SimGrid-3.18/src/smpi/mpi/smpi_op.cpp0000644000175000017500000002001513217757320020012 0ustar mquinsonmquinson/* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_op, smpi, "Logging specific to SMPI (op)"); #define MAX_OP(a, b) (b) = (a) < (b) ? (b) : (a) #define MIN_OP(a, b) (b) = (a) < (b) ? (a) : (b) #define SUM_OP(a, b) (b) += (a) #define PROD_OP(a, b) (b) *= (a) #define LAND_OP(a, b) (b) = (a) && (b) #define LOR_OP(a, b) (b) = (a) || (b) #define LXOR_OP(a, b) (b) = (not(a) && (b)) || ((a) && not(b)) #define BAND_OP(a, b) (b) &= (a) #define BOR_OP(a, b) (b) |= (a) #define BXOR_OP(a, b) (b) ^= (a) #define MAXLOC_OP(a, b) (b) = (a.value) < (b.value) ? (b) : ((a.value) == (b.value) ? ((a.index) < (b.index) ? (a) : (b)) : (a)) #define MINLOC_OP(a, b) (b) = (a.value) < (b.value) ? (a) : ((a.value) == (b.value) ? ((a.index) < (b.index) ? (a) : (b)) : (b)) #define APPLY_FUNC(a, b, length, type, func) \ { \ int i; \ type* x = (type*)(a); \ type* y = (type*)(b); \ for(i = 0; i < *(length); i++) { \ func(x[i], y[i]); \ } \ } #define APPLY_OP_LOOP(dtype, type, op) \ if (*datatype == dtype) {\ APPLY_FUNC(a, b, length, type, op)\ } else \ #define APPLY_BASIC_OP_LOOP(op)\ APPLY_OP_LOOP(MPI_CHAR, char,op)\ APPLY_OP_LOOP(MPI_SHORT, short,op)\ APPLY_OP_LOOP(MPI_INT, int,op)\ APPLY_OP_LOOP(MPI_LONG, long,op)\ APPLY_OP_LOOP(MPI_LONG_LONG, long long,op)\ APPLY_OP_LOOP(MPI_SIGNED_CHAR, signed char,op)\ APPLY_OP_LOOP(MPI_UNSIGNED_CHAR, unsigned char,op)\ APPLY_OP_LOOP(MPI_UNSIGNED_SHORT, unsigned short,op)\ APPLY_OP_LOOP(MPI_UNSIGNED, unsigned int,op)\ APPLY_OP_LOOP(MPI_UNSIGNED_LONG, unsigned long,op)\ APPLY_OP_LOOP(MPI_UNSIGNED_LONG_LONG, unsigned long long,op)\ APPLY_OP_LOOP(MPI_WCHAR, wchar_t,op)\ APPLY_OP_LOOP(MPI_BYTE, int8_t,op)\ APPLY_OP_LOOP(MPI_INT8_T, int8_t,op)\ APPLY_OP_LOOP(MPI_INT16_T, int16_t,op)\ APPLY_OP_LOOP(MPI_INT32_T, int32_t,op)\ APPLY_OP_LOOP(MPI_INT64_T, int64_t,op)\ APPLY_OP_LOOP(MPI_UINT8_T, uint8_t,op)\ APPLY_OP_LOOP(MPI_UINT16_T, uint16_t,op)\ APPLY_OP_LOOP(MPI_UINT32_T, uint32_t,op)\ APPLY_OP_LOOP(MPI_UINT64_T, uint64_t,op)\ APPLY_OP_LOOP(MPI_AINT, MPI_Aint,op)\ APPLY_OP_LOOP(MPI_OFFSET, MPI_Offset,op)\ APPLY_OP_LOOP(MPI_INTEGER1, int,op)\ APPLY_OP_LOOP(MPI_INTEGER2, int16_t,op)\ APPLY_OP_LOOP(MPI_INTEGER4, int32_t,op)\ APPLY_OP_LOOP(MPI_INTEGER8, int64_t,op) #define APPLY_BOOL_OP_LOOP(op)\ APPLY_OP_LOOP(MPI_C_BOOL, bool,op) #define APPLY_FLOAT_OP_LOOP(op)\ APPLY_OP_LOOP(MPI_FLOAT, float,op)\ APPLY_OP_LOOP(MPI_DOUBLE, double,op)\ APPLY_OP_LOOP(MPI_LONG_DOUBLE, long double,op)\ APPLY_OP_LOOP(MPI_REAL, float,op)\ APPLY_OP_LOOP(MPI_REAL4, float,op)\ APPLY_OP_LOOP(MPI_REAL8, float,op)\ APPLY_OP_LOOP(MPI_REAL16, double,op) #define APPLY_COMPLEX_OP_LOOP(op)\ APPLY_OP_LOOP(MPI_C_FLOAT_COMPLEX, float _Complex,op)\ APPLY_OP_LOOP(MPI_C_DOUBLE_COMPLEX, double _Complex,op)\ APPLY_OP_LOOP(MPI_C_LONG_DOUBLE_COMPLEX, long double _Complex,op) #define APPLY_PAIR_OP_LOOP(op)\ APPLY_OP_LOOP(MPI_FLOAT_INT, float_int,op)\ APPLY_OP_LOOP(MPI_LONG_INT, long_int,op)\ APPLY_OP_LOOP(MPI_DOUBLE_INT, double_int,op)\ APPLY_OP_LOOP(MPI_SHORT_INT, short_int,op)\ APPLY_OP_LOOP(MPI_2INT, int_int,op)\ APPLY_OP_LOOP(MPI_2FLOAT, float_float,op)\ APPLY_OP_LOOP(MPI_2DOUBLE, double_double,op)\ APPLY_OP_LOOP(MPI_LONG_DOUBLE_INT, long_double_int,op)\ APPLY_OP_LOOP(MPI_2LONG, long_long,op) #define APPLY_END_OP_LOOP(op)\ {\ xbt_die("Failed to apply " #op " to type %s", (*datatype)->name());\ } static void max_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(MAX_OP) APPLY_FLOAT_OP_LOOP(MAX_OP) APPLY_END_OP_LOOP(MAX_OP) } static void min_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(MIN_OP) APPLY_FLOAT_OP_LOOP(MIN_OP) APPLY_END_OP_LOOP(MIN_OP) } static void sum_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(SUM_OP) APPLY_FLOAT_OP_LOOP(SUM_OP) APPLY_COMPLEX_OP_LOOP(SUM_OP) APPLY_END_OP_LOOP(SUM_OP) } static void prod_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(PROD_OP) APPLY_FLOAT_OP_LOOP(PROD_OP) APPLY_COMPLEX_OP_LOOP(PROD_OP) APPLY_END_OP_LOOP(PROD_OP) } static void land_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(LAND_OP) APPLY_BOOL_OP_LOOP(LAND_OP) APPLY_END_OP_LOOP(LAND_OP) } static void lor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(LOR_OP) APPLY_BOOL_OP_LOOP(LOR_OP) APPLY_END_OP_LOOP(LOR_OP) } static void lxor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(LXOR_OP) APPLY_BOOL_OP_LOOP(LXOR_OP) APPLY_END_OP_LOOP(LXOR_OP) } static void band_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(BAND_OP) APPLY_BOOL_OP_LOOP(BAND_OP) APPLY_END_OP_LOOP(BAND_OP) } static void bor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(BOR_OP) APPLY_BOOL_OP_LOOP(BOR_OP) APPLY_END_OP_LOOP(BOR_OP) } static void bxor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_BASIC_OP_LOOP(BXOR_OP) APPLY_BOOL_OP_LOOP(BXOR_OP) APPLY_END_OP_LOOP(BXOR_OP) } static void minloc_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_PAIR_OP_LOOP(MINLOC_OP) APPLY_END_OP_LOOP(MINLOC_OP) } static void maxloc_func(void *a, void *b, int *length, MPI_Datatype * datatype) { APPLY_PAIR_OP_LOOP(MAXLOC_OP) APPLY_END_OP_LOOP(MAXLOC_OP) } static void replace_func(void *a, void *b, int *length, MPI_Datatype * datatype) { memcpy(b, a, *length * (*datatype)->size()); } static void no_func(void *a, void *b, int *length, MPI_Datatype * datatype) { /* obviously a no-op */ } #define CREATE_MPI_OP(name, func) \ static SMPI_Op mpi_##name (&(func) /* func */, true ); \ MPI_Op name = &mpi_##name; CREATE_MPI_OP(MPI_MAX, max_func); CREATE_MPI_OP(MPI_MIN, min_func); CREATE_MPI_OP(MPI_SUM, sum_func); CREATE_MPI_OP(MPI_PROD, prod_func); CREATE_MPI_OP(MPI_LAND, land_func); CREATE_MPI_OP(MPI_LOR, lor_func); CREATE_MPI_OP(MPI_LXOR, lxor_func); CREATE_MPI_OP(MPI_BAND, band_func); CREATE_MPI_OP(MPI_BOR, bor_func); CREATE_MPI_OP(MPI_BXOR, bxor_func); CREATE_MPI_OP(MPI_MAXLOC, maxloc_func); CREATE_MPI_OP(MPI_MINLOC, minloc_func); CREATE_MPI_OP(MPI_REPLACE, replace_func); CREATE_MPI_OP(MPI_NO_OP, no_func); namespace simgrid{ namespace smpi{ Op::Op(MPI_User_function * function, bool commutative) : func_(function), is_commutative_(commutative) { is_fortran_op_ = false; } bool Op::is_commutative() { return is_commutative_; } bool Op::is_fortran_op() { return is_fortran_op_; } void Op::set_fortran_op() { //tell that we were created from fortran, so we need to translate the type to fortran when called is_fortran_op_ = true; } void Op::apply(void *invec, void *inoutvec, int *len, MPI_Datatype datatype) { if(smpi_privatize_global_variables == SMPI_PRIVATIZE_MMAP){//we need to switch as the called function may silently touch global variables XBT_DEBUG("Applying operation, switch to the right data frame "); smpi_switch_data_segment(smpi_process()->index()); } if (not smpi_process()->replaying() && *len > 0) { if (not is_fortran_op_) this->func_(invec, inoutvec, len, &datatype); else{ XBT_DEBUG("Applying operation of length %d from %p and from/to %p", *len, invec, inoutvec); int tmp = datatype->c2f(); /* Unfortunately, the C and Fortran version of the MPI standard do not agree on the type here, thus the reinterpret_cast. */ this->func_(invec, inoutvec, len, reinterpret_cast(&tmp) ); } } } Op* Op::f2c(int id){ return static_cast(F2C::f2c(id)); } } } SimGrid-3.18/src/smpi/mpi/smpi_datatype_derived.cpp0000644000175000017500000002333313217757320022717 0ustar mquinsonmquinson/* smpi_datatype.cpp -- MPI primitives to handle datatypes */ /* Copyright (c) 2009-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_datatype_derived.hpp" #include "smpi_op.hpp" #include namespace simgrid{ namespace smpi{ Type_Contiguous::Type_Contiguous(int size, MPI_Aint lb, MPI_Aint ub, int flags, int block_count, MPI_Datatype old_type) : Datatype(size, lb, ub, flags), block_count_(block_count), old_type_(old_type) { old_type_->ref(); } Type_Contiguous::~Type_Contiguous() { Datatype::unref(old_type_); } void Type_Contiguous::serialize(void* noncontiguous_buf, void* contiguous_buf, int count) { char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+lb(); memcpy(contiguous_buf_char, noncontiguous_buf_char, count * block_count_ * old_type_->size()); } void Type_Contiguous::unserialize(void* contiguous_buf, void* noncontiguous_buf, int count, MPI_Op op) { char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+lb(); int n= count*block_count_; if(op!=MPI_OP_NULL) op->apply( contiguous_buf_char, noncontiguous_buf_char, &n, old_type_); } Type_Hvector::Type_Hvector(int size,MPI_Aint lb, MPI_Aint ub, int flags, int count, int block_length, MPI_Aint stride, MPI_Datatype old_type): Datatype(size, lb, ub, flags), block_count_(count), block_length_(block_length), block_stride_(stride), old_type_(old_type){ old_type->ref(); } Type_Hvector::~Type_Hvector(){ Datatype::unref(old_type_); } void Type_Hvector::serialize( void* noncontiguous_buf, void *contiguous_buf, int count){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf); for (int i = 0; i < block_count_ * count; i++) { if (not(old_type_->flags() & DT_FLAG_DERIVED)) memcpy(contiguous_buf_char, noncontiguous_buf_char, block_length_ * old_type_->size()); else old_type_->serialize( noncontiguous_buf_char, contiguous_buf_char, block_length_); contiguous_buf_char += block_length_*old_type_->size(); if((i+1)%block_count_ ==0) noncontiguous_buf_char += block_length_*old_type_->size(); else noncontiguous_buf_char += block_stride_; } } void Type_Hvector::unserialize( void* contiguous_buf, void *noncontiguous_buf, int count, MPI_Op op){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf); for (int i = 0; i < block_count_ * count; i++) { if (not(old_type_->flags() & DT_FLAG_DERIVED)) { if(op!=MPI_OP_NULL) op->apply( contiguous_buf_char, noncontiguous_buf_char, &block_length_, old_type_); }else old_type_->unserialize( contiguous_buf_char, noncontiguous_buf_char, block_length_, op); contiguous_buf_char += block_length_*old_type_->size(); if((i+1)%block_count_ ==0) noncontiguous_buf_char += block_length_*old_type_->size(); else noncontiguous_buf_char += block_stride_; } } Type_Vector::Type_Vector(int size, MPI_Aint lb, MPI_Aint ub, int flags, int count, int block_length, int stride, MPI_Datatype old_type) : Type_Hvector(size, lb, ub, flags, count, block_length, stride * old_type->get_extent(), old_type) { } Type_Hindexed::Type_Hindexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int count, int* block_lengths, MPI_Aint* block_indices, MPI_Datatype old_type) : Datatype(size, lb, ub, flags), block_count_(count), old_type_(old_type) { old_type_->ref(); block_lengths_ = new int[count]; block_indices_ = new MPI_Aint[count]; for (int i = 0; i < count; i++) { block_lengths_[i] = block_lengths[i]; block_indices_[i] = block_indices[i]; } } Type_Hindexed::Type_Hindexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int count, int* block_lengths, int* block_indices, MPI_Datatype old_type, MPI_Aint factor) : Datatype(size, lb, ub, flags), block_count_(count), old_type_(old_type) { old_type_->ref(); block_lengths_ = new int[count]; block_indices_ = new MPI_Aint[count]; for (int i = 0; i < count; i++) { block_lengths_[i] = block_lengths[i]; block_indices_[i] = block_indices[i] * factor; } } Type_Hindexed::~Type_Hindexed() { Datatype::unref(old_type_); if(refcount()==0){ delete[] block_lengths_; delete[] block_indices_; } } void Type_Hindexed::serialize( void* noncontiguous_buf, void *contiguous_buf, int count){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+ block_indices_[0]; for (int j = 0; j < count; j++) { for (int i = 0; i < block_count_; i++) { if (not(old_type_->flags() & DT_FLAG_DERIVED)) memcpy(contiguous_buf_char, noncontiguous_buf_char, block_lengths_[i] * old_type_->size()); else old_type_->serialize(noncontiguous_buf_char, contiguous_buf_char,block_lengths_[i]); contiguous_buf_char += block_lengths_[i]*old_type_->size(); if (i(noncontiguous_buf) + block_indices_[i+1]; else noncontiguous_buf_char += block_lengths_[i]*old_type_->get_extent(); } noncontiguous_buf=static_cast(noncontiguous_buf_char); } } void Type_Hindexed::unserialize( void* contiguous_buf, void *noncontiguous_buf, int count, MPI_Op op){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+ block_indices_[0]; for (int j = 0; j < count; j++) { for (int i = 0; i < block_count_; i++) { if (not(old_type_->flags() & DT_FLAG_DERIVED)) { if(op!=MPI_OP_NULL) op->apply( contiguous_buf_char, noncontiguous_buf_char, &block_lengths_[i], old_type_); }else old_type_->unserialize( contiguous_buf_char,noncontiguous_buf_char,block_lengths_[i], op); contiguous_buf_char += block_lengths_[i]*old_type_->size(); if (i(noncontiguous_buf) + block_indices_[i+1]; else noncontiguous_buf_char += block_lengths_[i]*old_type_->get_extent(); } noncontiguous_buf=static_cast(noncontiguous_buf_char); } } Type_Indexed::Type_Indexed(int size, MPI_Aint lb, MPI_Aint ub, int flags, int count, int* block_lengths, int* block_indices, MPI_Datatype old_type) : Type_Hindexed(size, lb, ub, flags, count, block_lengths, block_indices, old_type, old_type->get_extent()) { } Type_Struct::Type_Struct(int size,MPI_Aint lb, MPI_Aint ub, int flags, int count, int* block_lengths, MPI_Aint* block_indices, MPI_Datatype* old_types): Datatype(size, lb, ub, flags), block_count_(count), block_lengths_(block_lengths), block_indices_(block_indices), old_types_(old_types){ block_lengths_= new int[count]; block_indices_= new MPI_Aint[count]; old_types_= new MPI_Datatype[count]; for (int i = 0; i < count; i++) { block_lengths_[i]=block_lengths[i]; block_indices_[i]=block_indices[i]; old_types_[i]=old_types[i]; old_types_[i]->ref(); } } Type_Struct::~Type_Struct(){ for (int i = 0; i < block_count_; i++) { Datatype::unref(old_types_[i]); } if(refcount()==0){ delete[] block_lengths_; delete[] block_indices_; delete[] old_types_; } } void Type_Struct::serialize( void* noncontiguous_buf, void *contiguous_buf, int count){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+ block_indices_[0]; for (int j = 0; j < count; j++) { for (int i = 0; i < block_count_; i++) { if (not(old_types_[i]->flags() & DT_FLAG_DERIVED)) memcpy(contiguous_buf_char, noncontiguous_buf_char, block_lengths_[i] * old_types_[i]->size()); else old_types_[i]->serialize( noncontiguous_buf_char,contiguous_buf_char,block_lengths_[i]); contiguous_buf_char += block_lengths_[i]*old_types_[i]->size(); if (i(noncontiguous_buf) + block_indices_[i+1]; else //let's hope this is MPI_UB ? noncontiguous_buf_char += block_lengths_[i]*old_types_[i]->get_extent(); } noncontiguous_buf=static_cast(noncontiguous_buf_char); } } void Type_Struct::unserialize( void* contiguous_buf, void *noncontiguous_buf, int count, MPI_Op op){ char* contiguous_buf_char = static_cast(contiguous_buf); char* noncontiguous_buf_char = static_cast(noncontiguous_buf)+ block_indices_[0]; for (int j = 0; j < count; j++) { for (int i = 0; i < block_count_; i++) { if (not(old_types_[i]->flags() & DT_FLAG_DERIVED)) { if(op!=MPI_OP_NULL) op->apply( contiguous_buf_char, noncontiguous_buf_char, &block_lengths_[i], old_types_[i]); }else old_types_[i]->unserialize( contiguous_buf_char, noncontiguous_buf_char,block_lengths_[i], op); contiguous_buf_char += block_lengths_[i]*old_types_[i]->size(); if (i(noncontiguous_buf) + block_indices_[i+1]; else noncontiguous_buf_char += block_lengths_[i]*old_types_[i]->get_extent(); } noncontiguous_buf=static_cast(noncontiguous_buf_char); } } } } SimGrid-3.18/src/smpi/mpi/smpi_status.cpp0000644000175000017500000000132513217757320020722 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_status.hpp" #include "private.hpp" #include "smpi_datatype.hpp" #include "src/simix/smx_private.hpp" namespace simgrid{ namespace smpi{ void Status::empty(MPI_Status * status) { if(status != MPI_STATUS_IGNORE) { status->MPI_SOURCE = MPI_ANY_SOURCE; status->MPI_TAG = MPI_ANY_TAG; status->MPI_ERROR = MPI_SUCCESS; status->count=0; } } int Status::get_count(MPI_Status * status, MPI_Datatype datatype) { return status->count / datatype->size(); } } } SimGrid-3.18/src/smpi/mpi/smpi_keyvals.cpp0000644000175000017500000000257313217757320021063 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ //#include "private.hpp" #include "smpi_keyvals.hpp" namespace simgrid{ namespace smpi{ std::unordered_map* Keyval::attributes(){ return &attributes_; }; template <> int Keyval::call_deleter(Comm* obj, smpi_key_elem elem, int keyval, void * value, int* flag){ if(elem->delete_fn.comm_delete_fn!=MPI_NULL_DELETE_FN){ int ret = elem->delete_fn.comm_delete_fn(obj, keyval, value, flag); if(ret!=MPI_SUCCESS) return ret; } return MPI_SUCCESS; } template <> int Keyval::call_deleter(Win* obj, smpi_key_elem elem, int keyval, void * value, int* flag){ if(elem->delete_fn.win_delete_fn!=MPI_NULL_DELETE_FN){ int ret = elem->delete_fn.win_delete_fn(obj, keyval, value, flag); if(ret!=MPI_SUCCESS) return ret; } return MPI_SUCCESS; } template <> int Keyval::call_deleter(Datatype* obj, smpi_key_elem elem, int keyval, void * value, int* flag){ if(elem->delete_fn.type_delete_fn!=MPI_NULL_DELETE_FN){ int ret = elem->delete_fn.type_delete_fn(obj, keyval, value, flag); if(ret!=MPI_SUCCESS) return ret; } return MPI_SUCCESS; } } } SimGrid-3.18/src/smpi/mpi/smpi_request.cpp0000644000175000017500000010360013217757320021066 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_request.hpp" #include "SmpiHost.hpp" #include "mc/mc.h" #include "private.hpp" #include "smpi_comm.hpp" #include "smpi_datatype.hpp" #include "smpi_op.hpp" #include "smpi_process.hpp" #include "src/kernel/activity/CommImpl.hpp" #include "src/mc/mc_replay.hpp" #include "src/simix/ActorImpl.hpp" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_request, smpi, "Logging specific to SMPI (reques)"); static simgrid::config::Flag smpi_iprobe_sleep( "smpi/iprobe", "Minimum time to inject inside a call to MPI_Iprobe", 1e-4); static simgrid::config::Flag smpi_test_sleep( "smpi/test", "Minimum time to inject inside a call to MPI_Test", 1e-4); std::vector smpi_ois_values; extern void (*smpi_comm_copy_data_callback) (smx_activity_t, void*, size_t); namespace simgrid{ namespace smpi{ Request::Request(void* buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, unsigned flags) : buf_(buf), old_type_(datatype), src_(src), dst_(dst), tag_(tag), comm_(comm), flags_(flags) { void *old_buf = nullptr; // FIXME Handle the case of a partial shared malloc. if ((((flags & RECV) != 0) && ((flags & ACCUMULATE) != 0)) || (datatype->flags() & DT_FLAG_DERIVED)) { // This part handles the problem of non-contiguous memory old_buf = buf; if (count==0){ buf_ = nullptr; }else { buf_ = xbt_malloc(count*datatype->size()); if ((datatype->flags() & DT_FLAG_DERIVED) && ((flags & SEND) != 0)) { datatype->serialize(old_buf, buf_, count); } } } // This part handles the problem of non-contiguous memory (for the unserialisation at the reception) old_buf_ = old_buf; size_ = datatype->size() * count; datatype->ref(); comm_->ref(); action_ = nullptr; detached_ = 0; detached_sender_ = nullptr; real_src_ = 0; truncated_ = 0; real_size_ = 0; real_tag_ = 0; if (flags & PERSISTENT) refcount_ = 1; else refcount_ = 0; op_ = MPI_REPLACE; } MPI_Comm Request::comm(){ return comm_; } int Request::src(){ return src_; } int Request::dst(){ return dst_; } int Request::tag(){ return tag_; } int Request::flags(){ return flags_; } int Request::detached(){ return detached_; } size_t Request::size(){ return size_; } size_t Request::real_size(){ return real_size_; } void Request::unref(MPI_Request* request) { if((*request) != MPI_REQUEST_NULL){ (*request)->refcount_--; if((*request)->refcount_<0) xbt_die("wrong refcount"); if((*request)->refcount_==0){ Datatype::unref((*request)->old_type_); Comm::unref((*request)->comm_); (*request)->print_request("Destroying"); delete *request; *request = MPI_REQUEST_NULL; }else{ (*request)->print_request("Decrementing"); } }else{ xbt_die("freeing an already free request"); } } int Request::match_recv(void* a, void* b, simgrid::kernel::activity::CommImpl* ignored) { MPI_Request ref = static_cast(a); MPI_Request req = static_cast(b); XBT_DEBUG("Trying to match a recv of src %d against %d, tag %d against %d",ref->src_,req->src_, ref->tag_, req->tag_); xbt_assert(ref, "Cannot match recv against null reference"); xbt_assert(req, "Cannot match recv against null request"); if((ref->src_ == MPI_ANY_SOURCE || req->src_ == ref->src_) && ((ref->tag_ == MPI_ANY_TAG && req->tag_ >=0) || req->tag_ == ref->tag_)){ //we match, we can transfer some values if(ref->src_ == MPI_ANY_SOURCE) ref->real_src_ = req->src_; if(ref->tag_ == MPI_ANY_TAG) ref->real_tag_ = req->tag_; if(ref->real_size_ < req->real_size_) ref->truncated_ = 1; if(req->detached_==1) ref->detached_sender_=req; //tie the sender to the receiver, as it is detached and has to be freed in the receiver XBT_DEBUG("match succeeded"); return 1; }else return 0; } int Request::match_send(void* a, void* b, simgrid::kernel::activity::CommImpl* ignored) { MPI_Request ref = static_cast(a); MPI_Request req = static_cast(b); XBT_DEBUG("Trying to match a send of src %d against %d, tag %d against %d",ref->src_,req->src_, ref->tag_, req->tag_); xbt_assert(ref, "Cannot match send against null reference"); xbt_assert(req, "Cannot match send against null request"); if((req->src_ == MPI_ANY_SOURCE || req->src_ == ref->src_) && ((req->tag_ == MPI_ANY_TAG && ref->tag_ >=0)|| req->tag_ == ref->tag_)){ if(req->src_ == MPI_ANY_SOURCE) req->real_src_ = ref->src_; if(req->tag_ == MPI_ANY_TAG) req->real_tag_ = ref->tag_; if(req->real_size_ < ref->real_size_) req->truncated_ = 1; if(ref->detached_==1) req->detached_sender_=ref; //tie the sender to the receiver, as it is detached and has to be freed in the receiver XBT_DEBUG("match succeeded"); return 1; } else return 0; } void Request::print_request(const char *message) { XBT_VERB("%s request %p [buf = %p, size = %zu, src = %d, dst = %d, tag = %d, flags = %x]", message, this, buf_, size_, src_, dst_, tag_, flags_); } /* factories, to hide the internal flags from the caller */ MPI_Request Request::send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { return new Request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, PERSISTENT | SEND | PREPARED); } MPI_Request Request::ssend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { return new Request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, PERSISTENT | SSEND | SEND | PREPARED); } MPI_Request Request::isend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { return new Request(buf==MPI_BOTTOM ? nullptr : buf , count, datatype, smpi_process()->index(), comm->group()->index(dst), tag,comm, PERSISTENT | ISEND | SEND | PREPARED); } MPI_Request Request::rma_send_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ if(op==MPI_OP_NULL){ request = new Request(buf==MPI_BOTTOM ? nullptr : buf , count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | ISEND | SEND | PREPARED); }else{ request = new Request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | ISEND | SEND | PREPARED | ACCUMULATE); request->op_ = op; } return request; } MPI_Request Request::recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { return new Request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : comm->group()->index(src), smpi_process()->index(), tag, comm, PERSISTENT | RECV | PREPARED); } MPI_Request Request::rma_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ if(op==MPI_OP_NULL){ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | RECV | PREPARED); }else{ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | RECV | PREPARED | ACCUMULATE); request->op_ = op; } return request; } MPI_Request Request::irecv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { return new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : comm->group()->index(src), smpi_process()->index(), tag, comm, PERSISTENT | RECV | PREPARED); } MPI_Request Request::isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, NON_PERSISTENT | ISEND | SEND); request->start(); return request; } MPI_Request Request::issend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, NON_PERSISTENT | ISEND | SSEND | SEND); request->start(); return request; } MPI_Request Request::irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : comm->group()->index(src), smpi_process()->index(), tag, comm, NON_PERSISTENT | RECV); request->start(); return request; } void Request::recv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = irecv(buf, count, datatype, src, tag, comm); wait(&request,status); request = nullptr; } void Request::send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, NON_PERSISTENT | SEND); request->start(); wait(&request, MPI_STATUS_IGNORE); request = nullptr; } void Request::ssend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = new Request(buf == MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process()->index(), comm->group()->index(dst), tag, comm, NON_PERSISTENT | SSEND | SEND); request->start(); wait(&request,MPI_STATUS_IGNORE); request = nullptr; } void Request::sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype,int dst, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status) { MPI_Request requests[2]; MPI_Status stats[2]; int myid=smpi_process()->index(); if ((comm->group()->index(dst) == myid) && (comm->group()->index(src) == myid)){ Datatype::copy(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype); return; } requests[0] = isend_init(sendbuf, sendcount, sendtype, dst, sendtag, comm); requests[1] = irecv_init(recvbuf, recvcount, recvtype, src, recvtag, comm); startall(2, requests); waitall(2, requests, stats); unref(&requests[0]); unref(&requests[1]); if(status != MPI_STATUS_IGNORE) { // Copy receive status *status = stats[1]; } } void Request::start() { smx_mailbox_t mailbox; xbt_assert(action_ == nullptr, "Cannot (re-)start unfinished communication"); flags_ &= ~PREPARED; flags_ &= ~FINISHED; refcount_++; if ((flags_ & RECV) != 0) { this->print_request("New recv"); simgrid::smpi::Process* process = smpi_process_remote(dst_); int async_small_thresh = xbt_cfg_get_int("smpi/async-small-thresh"); xbt_mutex_t mut = process->mailboxes_mutex(); if (async_small_thresh != 0 || (flags_ & RMA) != 0) xbt_mutex_acquire(mut); if (async_small_thresh == 0 && (flags_ & RMA) == 0 ) { mailbox = process->mailbox(); } else if (((flags_ & RMA) != 0) || static_cast(size_) < async_small_thresh) { //We have to check both mailboxes (because SSEND messages are sent to the large mbox). //begin with the more appropriate one : the small one. mailbox = process->mailbox_small(); XBT_DEBUG("Is there a corresponding send already posted in the small mailbox %p (in case of SSEND)?", mailbox); smx_activity_t action = simcall_comm_iprobe(mailbox, 0, &match_recv, static_cast(this)); if (action == nullptr) { mailbox = process->mailbox(); XBT_DEBUG("No, nothing in the small mailbox test the other one : %p", mailbox); action = simcall_comm_iprobe(mailbox, 0, &match_recv, static_cast(this)); if (action == nullptr) { XBT_DEBUG("Still nothing, switch back to the small mailbox : %p", mailbox); mailbox = process->mailbox_small(); } } else { XBT_DEBUG("yes there was something for us in the large mailbox"); } } else { mailbox = process->mailbox_small(); XBT_DEBUG("Is there a corresponding send already posted the small mailbox?"); smx_activity_t action = simcall_comm_iprobe(mailbox, 0, &match_recv, static_cast(this)); if (action == nullptr) { XBT_DEBUG("No, nothing in the permanent receive mailbox"); mailbox = process->mailbox(); } else { XBT_DEBUG("yes there was something for us in the small mailbox"); } } // we make a copy here, as the size is modified by simix, and we may reuse the request in another receive later real_size_=size_; action_ = simcall_comm_irecv( process->process(), mailbox, buf_, &real_size_, &match_recv, process->replaying() ? &smpi_comm_null_copy_buffer_callback : smpi_comm_copy_data_callback, this, -1.0); XBT_DEBUG("recv simcall posted"); if (async_small_thresh != 0 || (flags_ & RMA) != 0 ) xbt_mutex_release(mut); } else { /* the RECV flag was not set, so this is a send */ simgrid::smpi::Process* process = smpi_process_remote(dst_); int rank = src_; if (TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, dst_, tag_, size_); } this->print_request("New send"); void* buf = buf_; if ((flags_ & SSEND) == 0 && ( (flags_ & RMA) != 0 || static_cast(size_) < xbt_cfg_get_int("smpi/send-is-detached-thresh") ) ) { void *oldbuf = nullptr; detached_ = 1; XBT_DEBUG("Send request %p is detached", this); refcount_++; if (not(old_type_->flags() & DT_FLAG_DERIVED)) { oldbuf = buf_; if (not process->replaying() && oldbuf != nullptr && size_ != 0) { if ((smpi_privatize_global_variables != 0) && (static_cast(buf_) >= smpi_data_exe_start) && (static_cast(buf_) < smpi_data_exe_start + smpi_data_exe_size)) { XBT_DEBUG("Privatization : We are sending from a zone inside global memory. Switch data segment "); smpi_switch_data_segment(src_); } buf = xbt_malloc(size_); memcpy(buf,oldbuf,size_); XBT_DEBUG("buf %p copied into %p",oldbuf,buf); } } } //if we are giving back the control to the user without waiting for completion, we have to inject timings double sleeptime = 0.0; if (detached_ != 0 || ((flags_ & (ISEND | SSEND)) != 0)) { // issend should be treated as isend // isend and send timings may be different sleeptime = ((flags_ & ISEND) != 0) ? simgrid::s4u::Actor::self()->getHost()->extension()->oisend(size_) : simgrid::s4u::Actor::self()->getHost()->extension()->osend(size_); } if(sleeptime > 0.0){ simcall_process_sleep(sleeptime); XBT_DEBUG("sending size of %zu : sleep %f ", size_, sleeptime); } int async_small_thresh = xbt_cfg_get_int("smpi/async-small-thresh"); xbt_mutex_t mut=process->mailboxes_mutex(); if (async_small_thresh != 0 || (flags_ & RMA) != 0) xbt_mutex_acquire(mut); if (not(async_small_thresh != 0 || (flags_ & RMA) != 0)) { mailbox = process->mailbox(); } else if (((flags_ & RMA) != 0) || static_cast(size_) < async_small_thresh) { // eager mode mailbox = process->mailbox(); XBT_DEBUG("Is there a corresponding recv already posted in the large mailbox %p?", mailbox); smx_activity_t action = simcall_comm_iprobe(mailbox, 1, &match_send, static_cast(this)); if (action == nullptr) { if ((flags_ & SSEND) == 0){ mailbox = process->mailbox_small(); XBT_DEBUG("No, nothing in the large mailbox, message is to be sent on the small one %p", mailbox); } else { mailbox = process->mailbox_small(); XBT_DEBUG("SSEND : Is there a corresponding recv already posted in the small mailbox %p?", mailbox); action = simcall_comm_iprobe(mailbox, 1, &match_send, static_cast(this)); if (action == nullptr) { XBT_DEBUG("No, we are first, send to large mailbox"); mailbox = process->mailbox(); } } } else { XBT_DEBUG("Yes there was something for us in the large mailbox"); } } else { mailbox = process->mailbox(); XBT_DEBUG("Send request %p is in the large mailbox %p (buf: %p)",mailbox, this,buf_); } // we make a copy here, as the size is modified by simix, and we may reuse the request in another receive later real_size_=size_; action_ = simcall_comm_isend( SIMIX_process_from_PID(src_ + 1), mailbox, size_, -1.0, buf, real_size_, &match_send, &xbt_free_f, // how to free the userdata if a detached send fails not process->replaying() ? smpi_comm_copy_data_callback : &smpi_comm_null_copy_buffer_callback, this, // detach if msg size < eager/rdv switch limit detached_); XBT_DEBUG("send simcall posted"); /* FIXME: detached sends are not traceable (action_ == nullptr) */ if (action_ != nullptr) simcall_set_category(action_, TRACE_internal_smpi_get_category()); if (async_small_thresh != 0 || ((flags_ & RMA)!=0)) xbt_mutex_release(mut); } } void Request::startall(int count, MPI_Request * requests) { if(requests== nullptr) return; for(int i = 0; i < count; i++) { requests[i]->start(); } } int Request::test(MPI_Request * request, MPI_Status * status) { //assume that request is not MPI_REQUEST_NULL (filtered in PMPI_Test or testall before) // to avoid deadlocks if used as a break condition, such as // while (MPI_Test(request, flag, status) && flag) dostuff... // because the time will not normally advance when only calls to MPI_Test are made -> deadlock // multiplier to the sleeptime, to increase speed of execution, each failed test will increase it static int nsleeps = 1; if(smpi_test_sleep > 0) simcall_process_sleep(nsleeps*smpi_test_sleep); Status::empty(status); int flag = 1; if (((*request)->flags_ & PREPARED) == 0) { if ((*request)->action_ != nullptr) flag = simcall_comm_test((*request)->action_); if (flag) { finish_wait(request,status); nsleeps=1;//reset the number of sleeps we will do next time if (*request != MPI_REQUEST_NULL && ((*request)->flags_ & PERSISTENT) == 0) *request = MPI_REQUEST_NULL; } else if (xbt_cfg_get_boolean("smpi/grow-injected-times")){ nsleeps++; } } return flag; } int Request::testsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]) { int count = 0; int count_dead = 0; MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; for (int i = 0; i < incount; i++) { if (requests[i] != MPI_REQUEST_NULL) { if (test(&requests[i], pstat)) { indices[i] = 1; count++; if (status != MPI_STATUSES_IGNORE) status[i] = *pstat; if ((requests[i] != MPI_REQUEST_NULL) && (requests[i]->flags_ & NON_PERSISTENT)) requests[i] = MPI_REQUEST_NULL; } } else { count_dead++; } } if(count_dead==incount) return MPI_UNDEFINED; else return count; } int Request::testany(int count, MPI_Request requests[], int *index, MPI_Status * status) { std::vector comms; comms.reserve(count); int i; int flag = 0; *index = MPI_UNDEFINED; std::vector map; /** Maps all matching comms back to their location in requests **/ for(i = 0; i < count; i++) { if ((requests[i] != MPI_REQUEST_NULL) && requests[i]->action_ && not(requests[i]->flags_ & PREPARED)) { comms.push_back(requests[i]->action_); map.push_back(i); } } if (not map.empty()) { //multiplier to the sleeptime, to increase speed of execution, each failed testany will increase it static int nsleeps = 1; if(smpi_test_sleep > 0) simcall_process_sleep(nsleeps*smpi_test_sleep); i = simcall_comm_testany(comms.data(), comms.size()); // The i-th element in comms matches! if (i != -1) { // -1 is not MPI_UNDEFINED but a SIMIX return code. (nothing matches) *index = map[i]; finish_wait(&requests[*index],status); flag = 1; nsleeps = 1; if (requests[*index] != MPI_REQUEST_NULL && (requests[*index]->flags_ & NON_PERSISTENT)) { requests[*index] = MPI_REQUEST_NULL; } } else { nsleeps++; } } else { //all requests are null or inactive, return true flag = 1; Status::empty(status); } return flag; } int Request::testall(int count, MPI_Request requests[], MPI_Status status[]) { MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; int flag=1; for(int i=0; iflags_ & PREPARED)) { if (test(&requests[i], pstat)!=1){ flag=0; }else{ requests[i]=MPI_REQUEST_NULL; } }else{ Status::empty(pstat); } if(status != MPI_STATUSES_IGNORE) { status[i] = *pstat; } } return flag; } void Request::probe(int source, int tag, MPI_Comm comm, MPI_Status* status){ int flag=0; //FIXME find another way to avoid busy waiting ? // the issue here is that we have to wait on a nonexistent comm while(flag==0){ iprobe(source, tag, comm, &flag, status); XBT_DEBUG("Busy Waiting on probing : %d", flag); } } void Request::iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status){ // to avoid deadlock, we have to sleep some time here, or the timer won't advance and we will only do iprobe simcalls // especially when used as a break condition, such as while (MPI_Iprobe(...)) dostuff... // nsleeps is a multiplier to the sleeptime, to increase speed of execution, each failed iprobe will increase it // This can speed up the execution of certain applications by an order of magnitude, such as HPL static int nsleeps = 1; double speed = simgrid::s4u::Actor::self()->getHost()->getSpeed(); double maxrate = xbt_cfg_get_double("smpi/iprobe-cpu-usage"); MPI_Request request = new Request(nullptr, 0, MPI_CHAR, source == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : comm->group()->index(source), comm->rank(), tag, comm, PERSISTENT | RECV); if (smpi_iprobe_sleep > 0) { smx_activity_t iprobe_sleep = simcall_execution_start( "iprobe", /* flops to executek*/ nsleeps * smpi_iprobe_sleep * speed * maxrate, /* priority */ 1.0, /* performance bound */ maxrate * speed, smpi_process()->process()->host); simcall_execution_wait(iprobe_sleep); } // behave like a receive, but don't do it smx_mailbox_t mailbox; request->print_request("New iprobe"); // We have to test both mailboxes as we don't know if we will receive one one or another if (xbt_cfg_get_int("smpi/async-small-thresh") > 0){ mailbox = smpi_process()->mailbox_small(); XBT_DEBUG("Trying to probe the perm recv mailbox"); request->action_ = simcall_comm_iprobe(mailbox, 0, &match_recv, static_cast(request)); } if (request->action_ == nullptr){ mailbox = smpi_process()->mailbox(); XBT_DEBUG("trying to probe the other mailbox"); request->action_ = simcall_comm_iprobe(mailbox, 0, &match_recv, static_cast(request)); } if (request->action_ != nullptr){ simgrid::kernel::activity::CommImplPtr sync_comm = boost::static_pointer_cast(request->action_); MPI_Request req = static_cast(sync_comm->src_data); *flag = 1; if(status != MPI_STATUS_IGNORE && (req->flags_ & PREPARED) == 0) { status->MPI_SOURCE = comm->group()->rank(req->src_); status->MPI_TAG = req->tag_; status->MPI_ERROR = MPI_SUCCESS; status->count = req->real_size_; } nsleeps = 1;//reset the number of sleeps we will do next time } else { *flag = 0; if (xbt_cfg_get_boolean("smpi/grow-injected-times")) nsleeps++; } unref(&request); } void Request::finish_wait(MPI_Request* request, MPI_Status * status) { MPI_Request req = *request; Status::empty(status); if (not((req->detached_ != 0) && ((req->flags_ & SEND) != 0)) && ((req->flags_ & PREPARED) == 0)) { if(status != MPI_STATUS_IGNORE) { int src = req->src_ == MPI_ANY_SOURCE ? req->real_src_ : req->src_; status->MPI_SOURCE = req->comm_->group()->rank(src); status->MPI_TAG = req->tag_ == MPI_ANY_TAG ? req->real_tag_ : req->tag_; status->MPI_ERROR = req->truncated_ != 0 ? MPI_ERR_TRUNCATE : MPI_SUCCESS; // this handles the case were size in receive differs from size in send status->count = req->real_size_; } req->print_request("Finishing"); MPI_Datatype datatype = req->old_type_; // FIXME Handle the case of a partial shared malloc. if (((req->flags_ & ACCUMULATE) != 0) || (datatype->flags() & DT_FLAG_DERIVED)) { // && (not smpi_is_shared(req->old_buf_))){ if (not smpi_process()->replaying() && smpi_privatize_global_variables != 0 && static_cast(req->old_buf_) >= smpi_data_exe_start && static_cast(req->old_buf_) < smpi_data_exe_start + smpi_data_exe_size) { XBT_VERB("Privatization : We are unserializing to a zone in global memory Switch data segment "); smpi_switch_data_segment(smpi_process()->index()); } if(datatype->flags() & DT_FLAG_DERIVED){ // This part handles the problem of non-contignous memory the unserialization at the reception if((req->flags_ & RECV) && datatype->size()!=0) datatype->unserialize(req->buf_, req->old_buf_, req->real_size_/datatype->size() , req->op_); xbt_free(req->buf_); }else if(req->flags_ & RECV){//apply op on contiguous buffer for accumulate if(datatype->size()!=0){ int n =req->real_size_/datatype->size(); req->op_->apply(req->buf_, req->old_buf_, &n, datatype); } xbt_free(req->buf_); } } } if (TRACE_smpi_view_internals() && ((req->flags_ & RECV) != 0)){ int rank = smpi_process()->index(); int src_traced = (req->src_ == MPI_ANY_SOURCE ? req->real_src_ : req->src_); TRACE_smpi_recv(src_traced, rank,req->tag_); } if(req->detached_sender_ != nullptr){ //integrate pseudo-timing for buffering of small messages, do not bother to execute the simcall if 0 double sleeptime = simgrid::s4u::Actor::self()->getHost()->extension()->orecv(req->real_size()); if(sleeptime > 0.0){ simcall_process_sleep(sleeptime); XBT_DEBUG("receiving size of %zu : sleep %f ", req->real_size_, sleeptime); } unref(&(req->detached_sender_)); } if(req->flags_ & PERSISTENT) req->action_ = nullptr; req->flags_ |= FINISHED; unref(request); } void Request::wait(MPI_Request * request, MPI_Status * status) { (*request)->print_request("Waiting"); if ((*request)->flags_ & PREPARED) { Status::empty(status); return; } if ((*request)->action_ != nullptr) // this is not a detached send simcall_comm_wait((*request)->action_, -1.0); finish_wait(request,status); if (*request != MPI_REQUEST_NULL && (((*request)->flags_ & NON_PERSISTENT)!=0)) *request = MPI_REQUEST_NULL; } int Request::waitany(int count, MPI_Request requests[], MPI_Status * status) { s_xbt_dynar_t comms; // Keep it on stack to save some extra mallocs int index = MPI_UNDEFINED; if(count > 0) { int size = 0; // Wait for a request to complete xbt_dynar_init(&comms, sizeof(smx_activity_t), [](void*ptr){ intrusive_ptr_release(*(simgrid::kernel::activity::ActivityImpl**)ptr); }); int *map = xbt_new(int, count); XBT_DEBUG("Wait for one of %d", count); for(int i = 0; i < count; i++) { if (requests[i] != MPI_REQUEST_NULL && not(requests[i]->flags_ & PREPARED) && not(requests[i]->flags_ & FINISHED)) { if (requests[i]->action_ != nullptr) { XBT_DEBUG("Waiting any %p ", requests[i]); intrusive_ptr_add_ref(requests[i]->action_.get()); xbt_dynar_push_as(&comms, simgrid::kernel::activity::ActivityImpl*, requests[i]->action_.get()); map[size] = i; size++; } else { // This is a finished detached request, let's return this one size = 0; // so we free the dynar but don't do the waitany call index = i; finish_wait(&requests[i], status); // cleanup if refcount = 0 if (requests[i] != MPI_REQUEST_NULL && (requests[i]->flags_ & NON_PERSISTENT)) requests[i] = MPI_REQUEST_NULL; // set to null break; } } } if (size > 0) { XBT_DEBUG("Enter waitany for %lu comms", xbt_dynar_length(&comms)); int i = simcall_comm_waitany(&comms, -1); // not MPI_UNDEFINED, as this is a simix return code if (i != -1) { index = map[i]; //in case of an accumulate, we have to wait the end of all requests to apply the operation, ordered correctly. if ((requests[index] == MPI_REQUEST_NULL) || (not((requests[index]->flags_ & ACCUMULATE) && (requests[index]->flags_ & RECV)))) { finish_wait(&requests[index],status); if (requests[i] != MPI_REQUEST_NULL && (requests[i]->flags_ & NON_PERSISTENT)) requests[index] = MPI_REQUEST_NULL; } } } xbt_dynar_free_data(&comms); xbt_free(map); } if (index==MPI_UNDEFINED) Status::empty(status); return index; } static int sort_accumulates(MPI_Request a, MPI_Request b) { return (a->tag() > b->tag()); } int Request::waitall(int count, MPI_Request requests[], MPI_Status status[]) { std::vector accumulates; int index; MPI_Status stat; MPI_Status *pstat = (status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat); int retvalue = MPI_SUCCESS; //tag invalid requests in the set if (status != MPI_STATUSES_IGNORE) { for (int c = 0; c < count; c++) { if (requests[c] == MPI_REQUEST_NULL || requests[c]->dst_ == MPI_PROC_NULL || (requests[c]->flags_ & PREPARED)) { Status::empty(&status[c]); } else if (requests[c]->src_ == MPI_PROC_NULL) { Status::empty(&status[c]); status[c].MPI_SOURCE = MPI_PROC_NULL; } } } for (int c = 0; c < count; c++) { if (MC_is_active() || MC_record_replay_is_active()) { wait(&requests[c],pstat); index = c; } else { index = waitany(count, (MPI_Request*)requests, pstat); if (index == MPI_UNDEFINED) break; if (requests[index] != MPI_REQUEST_NULL && (requests[index]->flags_ & RECV) && (requests[index]->flags_ & ACCUMULATE)) accumulates.push_back(requests[index]); if (requests[index] != MPI_REQUEST_NULL && (requests[index]->flags_ & NON_PERSISTENT)) requests[index] = MPI_REQUEST_NULL; } if (status != MPI_STATUSES_IGNORE) { status[index] = *pstat; if (status[index].MPI_ERROR == MPI_ERR_TRUNCATE) retvalue = MPI_ERR_IN_STATUS; } } if (not accumulates.empty()) { std::sort(accumulates.begin(), accumulates.end(), sort_accumulates); for (auto& req : accumulates) { finish_wait(&req, status); } } return retvalue; } int Request::waitsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]) { int count = 0; MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; for (int i = 0; i < incount; i++) { int index = waitany(incount, requests, pstat); if(index!=MPI_UNDEFINED){ indices[count] = index; count++; if(status != MPI_STATUSES_IGNORE) { status[index] = *pstat; } if (requests[index] != MPI_REQUEST_NULL && (requests[index]->flags_ & NON_PERSISTENT)) requests[index] = MPI_REQUEST_NULL; }else{ return MPI_UNDEFINED; } } return count; } MPI_Request Request::f2c(int id) { char key[KEY_SIZE]; if(id==MPI_FORTRAN_REQUEST_NULL) return static_cast(MPI_REQUEST_NULL); return static_cast(F2C::f2c_lookup()->at(get_key_id(key, id))); } int Request::add_f() { if (F2C::f2c_lookup() == nullptr) { F2C::set_f2c_lookup(new std::unordered_map); } char key[KEY_SIZE]; (*(F2C::f2c_lookup()))[get_key_id(key, F2C::f2c_id())] = this; F2C::f2c_id_increment(); return F2C::f2c_id()-1; } void Request::free_f(int id) { if (id != MPI_FORTRAN_REQUEST_NULL) { char key[KEY_SIZE]; F2C::f2c_lookup()->erase(get_key_id(key, id)); } } } } SimGrid-3.18/src/smpi/mpi/smpi_group.cpp0000644000175000017500000001761113217757320020540 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_group.hpp" #include "smpi_comm.hpp" #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_group, smpi, "Logging specific to SMPI (group)"); simgrid::smpi::Group mpi_MPI_GROUP_EMPTY; MPI_Group MPI_GROUP_EMPTY=&mpi_MPI_GROUP_EMPTY; namespace simgrid{ namespace smpi{ Group::Group() { size_ = 0; /* size */ refcount_ = 1; /* refcount_: start > 0 so that this group never gets freed */ } Group::Group(int n) : size_(n), rank_to_index_map_(size_, MPI_UNDEFINED) { refcount_ = 1; } Group::Group(MPI_Group origin) { if (origin != MPI_GROUP_NULL && origin != MPI_GROUP_EMPTY) { size_ = origin->size(); refcount_ = 1; rank_to_index_map_ = origin->rank_to_index_map_; index_to_rank_map_ = origin->index_to_rank_map_; } } void Group::set_mapping(int index, int rank) { if (0 <= rank && rank < size_) { rank_to_index_map_[rank] = index; if (index != MPI_UNDEFINED) { if ((unsigned)index >= index_to_rank_map_.size()) index_to_rank_map_.resize(index + 1, MPI_UNDEFINED); index_to_rank_map_[index] = rank; } } } int Group::index(int rank) { int index; if (0 <= rank && rank < size_) index = rank_to_index_map_[rank]; else index = MPI_UNDEFINED; return index; } int Group::rank(int index) { int rank; if (0 <= index && (unsigned)index < index_to_rank_map_.size()) rank = index_to_rank_map_[index]; else rank = MPI_UNDEFINED; return rank; } void Group::ref() { refcount_++; } void Group::unref(Group* group) { group->refcount_--; if (group->refcount_ <= 0) { delete group; } } int Group::size() { return size_; } int Group::compare(MPI_Group group2) { int result; result = MPI_IDENT; if (size_ != group2->size()) { result = MPI_UNEQUAL; } else { int sz = group2->size(); for (int i = 0; i < sz; i++) { int index = this->index(i); int rank = group2->rank(index); if (rank == MPI_UNDEFINED) { result = MPI_UNEQUAL; break; } if (rank != i) { result = MPI_SIMILAR; } } } return result; } int Group::incl(int n, int* ranks, MPI_Group* newgroup) { int i=0; int index=0; if (n == 0) { *newgroup = MPI_GROUP_EMPTY; } else if (n == size_) { *newgroup = this; if (this != MPI_COMM_WORLD->group() && this != MPI_COMM_SELF->group() && this != MPI_GROUP_EMPTY) this->ref(); } else { *newgroup = new Group(n); for (i = 0; i < n; i++) { index = this->index(ranks[i]); (*newgroup)->set_mapping(index, i); } } return MPI_SUCCESS; } int Group::group_union(MPI_Group group2, MPI_Group* newgroup) { int size1 = size_; int size2 = group2->size(); for (int i = 0; i < size2; i++) { int proc2 = group2->index(i); int proc1 = this->rank(proc2); if (proc1 == MPI_UNDEFINED) { size1++; } } if (size1 == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = new Group(size1); size2 = this->size(); for (int i = 0; i < size2; i++) { int proc1 = this->index(i); (*newgroup)->set_mapping(proc1, i); } for (int i = size2; i < size1; i++) { int proc2 = group2->index(i - size2); (*newgroup)->set_mapping(proc2, i); } } return MPI_SUCCESS; } int Group::intersection(MPI_Group group2, MPI_Group* newgroup) { int size2 = group2->size(); for (int i = 0; i < size2; i++) { int proc2 = group2->index(i); int proc1 = this->rank(proc2); if (proc1 == MPI_UNDEFINED) { size2--; } } if (size2 == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = new Group(size2); int j=0; for (int i = 0; i < group2->size(); i++) { int proc2 = group2->index(i); int proc1 = this->rank(proc2); if (proc1 != MPI_UNDEFINED) { (*newgroup)->set_mapping(proc2, j); j++; } } } return MPI_SUCCESS; } int Group::difference(MPI_Group group2, MPI_Group* newgroup) { int newsize = size_; int size2 = size_; for (int i = 0; i < size2; i++) { int proc1 = this->index(i); int proc2 = group2->rank(proc1); if (proc2 != MPI_UNDEFINED) { newsize--; } } if (newsize == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = new Group(newsize); for (int i = 0; i < size2; i++) { int proc1 = this->index(i); int proc2 = group2->rank(proc1); if (proc2 == MPI_UNDEFINED) { (*newgroup)->set_mapping(proc1, i); } } } return MPI_SUCCESS; } int Group::excl(int n, int *ranks, MPI_Group * newgroup){ int oldsize = size_; int newsize = oldsize - n; *newgroup = new Group(newsize); int* to_exclude = new int[size_]; for (int i = 0; i < oldsize; i++) to_exclude[i]=0; for (int i = 0; i < n; i++) to_exclude[ranks[i]]=1; int j = 0; for (int i = 0; i < oldsize; i++) { if(to_exclude[i]==0){ int index = this->index(i); (*newgroup)->set_mapping(index, j); j++; } } delete[] to_exclude; return MPI_SUCCESS; } static bool is_rank_in_range(int rank, int first, int last) { if (first < last) return rank <= last; else return rank >= last; } int Group::range_incl(int n, int ranges[][3], MPI_Group * newgroup){ int newsize = 0; for (int i = 0; i < n; i++) { for (int rank = ranges[i][0]; /* First */ rank >= 0 && rank < size_; /* Last */ ) { newsize++; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (not is_rank_in_range(rank, ranges[i][0], ranges[i][1])) break; } } *newgroup = new Group(newsize); int j = 0; for (int i = 0; i < n; i++) { for (int rank = ranges[i][0]; /* First */ rank >= 0 && rank < size_; /* Last */ ) { int index = this->index(rank); (*newgroup)->set_mapping(index, j); j++; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (not is_rank_in_range(rank, ranges[i][0], ranges[i][1])) break; } } return MPI_SUCCESS; } int Group::range_excl(int n, int ranges[][3], MPI_Group * newgroup){ int newsize = size_; for (int i = 0; i < n; i++) { for (int rank = ranges[i][0]; /* First */ rank >= 0 && rank < size_; /* Last */ ) { newsize--; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (not is_rank_in_range(rank, ranges[i][0], ranges[i][1])) break; } } if (newsize == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = new Group(newsize); int newrank = 0; int oldrank = 0; while (newrank < newsize) { int add = 1; for (int i = 0; i < n; i++) { for (int rank = ranges[i][0]; rank >= 0 && rank < size_;) { if(rank==oldrank){ add = 0; break; } if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (not is_rank_in_range(rank, ranges[i][0], ranges[i][1])) break; } } if(add==1){ int index = this->index(oldrank); (*newgroup)->set_mapping(index, newrank); newrank++; } oldrank++; } } return MPI_SUCCESS; } MPI_Group Group::f2c(int id) { if(id == -2) { return MPI_GROUP_EMPTY; } else if(F2C::f2c_lookup() != nullptr && id >= 0) { char key[KEY_SIZE]; return static_cast(F2C::f2c_lookup()->at(get_key(key, id))); } else { return static_cast(MPI_GROUP_NULL); } } } } SimGrid-3.18/src/smpi/mpi/smpi_info.cpp0000644000175000017500000000347113217757320020336 0ustar mquinsonmquinson/* Copyright (c) 2007-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi_info.hpp" #include "xbt/ex.hpp" #include "xbt/sysdep.h" namespace simgrid{ namespace smpi{ Info::Info(Info* info) : map_(info->map_) { } void Info::ref(){ refcount_++; } void Info::unref(Info* info){ info->refcount_--; if(info->refcount_==0){ delete info; } } void Info::set(char *key, char *value){ map_[key] = value; } int Info::get(char *key, int valuelen, char *value, int *flag){ *flag=false; auto val = map_.find(key); if (val != map_.end()) { std::string tmpvalue = val->second; memset(value, 0, valuelen); memcpy(value, tmpvalue.c_str(), (tmpvalue.length() + 1 < static_cast(valuelen)) ? tmpvalue.length() + 1 : valuelen); *flag=true; return MPI_SUCCESS; } else { return MPI_ERR_INFO_KEY; } } int Info::remove(char *key){ if (map_.erase(key) == 0) return MPI_ERR_INFO_NOKEY; else return MPI_SUCCESS; } int Info::get_nkeys(int *nkeys){ *nkeys = map_.size(); return MPI_SUCCESS; } int Info::get_nthkey(int n, char *key){ int num=0; for (auto const& elm : map_) { if (num == n) { strncpy(key, elm.first.c_str(), elm.first.length() + 1); return MPI_SUCCESS; } num++; } return MPI_ERR_ARG; } int Info::get_valuelen(char *key, int *valuelen, int *flag){ *flag=false; auto val = map_.find(key); if (val != map_.end()) { *valuelen = val->second.length(); *flag=true; return MPI_SUCCESS; } else { return MPI_ERR_INFO_KEY; } } Info* Info::f2c(int id){ return static_cast(F2C::f2c(id)); } } } SimGrid-3.18/src/s4u/0000755000175000017500000000000013217757316014625 5ustar mquinsonmquinsonSimGrid-3.18/src/s4u/s4u_exec.cpp0000644000175000017500000000411313217757316017047 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "simgrid/s4u/Actor.hpp" #include "simgrid/s4u/Exec.hpp" #include "src/kernel/activity/ExecImpl.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_exec, s4u_activity, "S4U asynchronous executions"); namespace simgrid { namespace s4u { Activity* Exec::start() { pimpl_ = simcall_execution_start(nullptr, flops_amount_, 1 / priority_, 0., host_); state_ = started; return this; } Activity* Exec::wait() { simcall_execution_wait(pimpl_); return this; } Activity* Exec::wait(double timeout) { THROW_UNIMPLEMENTED; return this; } bool Exec::test() { xbt_assert(state_ == inited || state_ == started || state_ == finished); if (state_ == finished) { return true; } if (state_ == inited) { this->start(); } if (simcall_execution_test(pimpl_)) { state_ = finished; return true; } return false; } ExecPtr Exec::setPriority(double priority) { xbt_assert(state_ == inited, "Cannot change the priority of an exec after its start"); priority_ = priority; return this; } ExecPtr Exec::setHost(Host* host) { xbt_assert(state_ == inited, "Cannot change the host of an exec after its start"); host_ = host; return this; } double Exec::getRemains() { return simgrid::simix::kernelImmediate( [this]() { return boost::static_pointer_cast(pimpl_)->remains(); }); } double Exec::getRemainingRatio() { return simgrid::simix::kernelImmediate( [this]() { return boost::static_pointer_cast(pimpl_)->remainingRatio(); }); } void intrusive_ptr_release(simgrid::s4u::Exec* e) { if (e->refcount_.fetch_sub(1, std::memory_order_release) == 1) { std::atomic_thread_fence(std::memory_order_acquire); delete e; } } void intrusive_ptr_add_ref(simgrid::s4u::Exec* e) { e->refcount_.fetch_add(1, std::memory_order_relaxed); } } } SimGrid-3.18/src/s4u/s4u_activity.cpp0000644000175000017500000000136513217757316017765 0ustar mquinsonmquinson/* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/asserts.h" #include "xbt/log.h" #include "simgrid/s4u/Activity.hpp" XBT_LOG_EXTERNAL_CATEGORY(s4u); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_activity,s4u,"S4U activities"); namespace simgrid { namespace s4u { double Activity::getRemains() { return remains_; } Activity* Activity::setRemains(double remains) { xbt_assert(state_ == inited, "Cannot change the remaining amount of work once the Activity is started"); remains_ = remains; return this; } } } SimGrid-3.18/src/s4u/s4u_mailbox.cpp0000644000175000017500000000672513217757316017571 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/Comm.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "src/msg/msg_private.hpp" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_network_private.hpp" #include "xbt/log.h" XBT_LOG_EXTERNAL_CATEGORY(s4u); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_channel,s4u,"S4U Communication Mailboxes"); namespace simgrid { namespace s4u { const simgrid::xbt::string& Mailbox::getName() const { return pimpl_->getName(); } const char* Mailbox::getCname() const { return pimpl_->getCname(); } MailboxPtr Mailbox::byName(const char*name) { kernel::activity::MailboxImpl* mbox = kernel::activity::MailboxImpl::byNameOrNull(name); if (mbox == nullptr) { mbox = simix::kernelImmediate([name] { return kernel::activity::MailboxImpl::byNameOrCreate(name); }); } return MailboxPtr(&mbox->piface_, true); } MailboxPtr Mailbox::byName(std::string name) { return byName(name.c_str()); } bool Mailbox::empty() { return pimpl_->comm_queue.empty(); } bool Mailbox::listen() { return not this->empty() || (pimpl_->permanent_receiver && not pimpl_->done_comm_queue.empty()); } smx_activity_t Mailbox::front() { return pimpl_->comm_queue.empty() ? nullptr : pimpl_->comm_queue.front(); } void Mailbox::setReceiver(ActorPtr actor) { simix::kernelImmediate([this, actor]() { this->pimpl_->setReceiver(actor); }); } /** @brief get the receiver (process associated to the mailbox) */ ActorPtr Mailbox::getReceiver() { if (pimpl_->permanent_receiver == nullptr) return ActorPtr(); return pimpl_->permanent_receiver->iface(); } CommPtr Mailbox::put_init() { CommPtr res = CommPtr(new s4u::Comm()); res->sender_ = SIMIX_process_self(); res->mailbox_ = this; return res; } s4u::CommPtr Mailbox::put_init(void* data, uint64_t simulatedSize) { s4u::CommPtr res = put_init(); res->setRemains(simulatedSize); res->srcBuff_ = data; res->srcBuffSize_ = sizeof(void*); return res; } s4u::CommPtr Mailbox::put_async(void* payload, uint64_t simulatedSize) { xbt_assert(payload != nullptr, "You cannot send nullptr"); s4u::CommPtr res = put_init(payload, simulatedSize); res->start(); return res; } void Mailbox::put(void* payload, uint64_t simulatedSize) { xbt_assert(payload != nullptr, "You cannot send nullptr"); CommPtr c = put_init(); c->setRemains(simulatedSize); c->setSrcData(payload); c->wait(); } /** Blocking send with timeout */ void Mailbox::put(void* payload, uint64_t simulatedSize, double timeout) { xbt_assert(payload != nullptr, "You cannot send nullptr"); CommPtr c = put_init(); c->setRemains(simulatedSize); c->setSrcData(payload); // c->start() is optional. c->wait(timeout); } s4u::CommPtr Mailbox::get_init() { CommPtr res = CommPtr(new s4u::Comm()); res->receiver_ = SIMIX_process_self(); res->mailbox_ = this; return res; } s4u::CommPtr Mailbox::get_async(void** data) { s4u::CommPtr res = get_init(); res->setDstData(data, sizeof(*data)); res->start(); return res; } void* Mailbox::get() { void* res = nullptr; CommPtr c = get_init(); c->setDstData(&res, sizeof(res)); c->wait(); return res; } void* Mailbox::get(double timeout) { void* res = nullptr; CommPtr c = get_init(); c->setDstData(&res, sizeof(res)); c->wait(timeout); return res; } } } SimGrid-3.18/src/s4u/s4u_storage.cpp0000644000175000017500000000376313217757316017601 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/Storage.hpp" #include "simgrid/simix.hpp" #include "src/plugins/file_system/FileSystem.hpp" #include "src/surf/StorageImpl.hpp" #include namespace simgrid { namespace xbt { template class Extendable; } namespace s4u { void getStorageList(std::map* whereTo) { for (auto const& s : *surf::StorageImpl::storagesMap()) whereTo->insert({s.first, &(s.second->piface_)}); // Convert each entry into its interface } Storage* Storage::byName(std::string name) { surf::StorageImpl* res = surf::StorageImpl::byName(name); if (res == nullptr) return nullptr; return &res->piface_; } const std::string& Storage::getName() const { return pimpl_->getName(); } const char* Storage::getCname() const { return pimpl_->getCname(); } const char* Storage::getType() { return pimpl_->typeId_.c_str(); } Host* Storage::getHost() { return attached_to_; } std::map* Storage::getProperties() { return simgrid::simix::kernelImmediate([this] { return pimpl_->getProperties(); }); } const char* Storage::getProperty(std::string key) { return this->pimpl_->getProperty(key); } void Storage::setProperty(std::string key, std::string value) { simgrid::simix::kernelImmediate([this, key, value] { this->pimpl_->setProperty(key, value); }); } sg_size_t Storage::read(sg_size_t size) { return simcall_storage_read(pimpl_, size); } sg_size_t Storage::write(sg_size_t size) { return simcall_storage_write(pimpl_, size); } /************* * Callbacks * *************/ simgrid::xbt::signal Storage::onCreation; simgrid::xbt::signal Storage::onDestruction; } /* namespace s4u */ } /* namespace simgrid */ SimGrid-3.18/src/s4u/s4u_mutex.cpp0000644000175000017500000000207113217757316017266 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/msg/msg_private.hpp" #include "src/simix/smx_synchro_private.hpp" #include "xbt/log.h" #include "simgrid/s4u/Mutex.hpp" namespace simgrid { namespace s4u { /** @brief Blocks the calling actor until the mutex can be obtained */ void Mutex::lock() { simcall_mutex_lock(mutex_); } /** @brief Release the ownership of the mutex, unleashing a blocked actor (if any) * * Will fail if the calling actor does not own the mutex. */ void Mutex::unlock() { simcall_mutex_unlock(mutex_); } /** @brief Acquire the mutex if it's free, and return false (without blocking) if not */ bool Mutex::try_lock() { return simcall_mutex_trylock(mutex_); } /** @brief Create a new mutex * * See @ref s4u_raii. */ MutexPtr Mutex::createMutex() { smx_mutex_t mutex = simcall_mutex_init(); return MutexPtr(&mutex->mutex(), false); } } } SimGrid-3.18/src/s4u/s4u_comm.cpp0000644000175000017500000001201313217757316017054 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/msg/msg_private.hpp" #include "xbt/log.h" #include "simgrid/s4u/Comm.hpp" #include "simgrid/s4u/Mailbox.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_comm,s4u_activity,"S4U asynchronous communications"); namespace simgrid { namespace s4u { Comm::~Comm() { if (state_ == started && not detached_ && (pimpl_ == nullptr || pimpl_->state == SIMIX_RUNNING)) { XBT_INFO("Comm %p freed before its completion. Detached: %d, State: %d", this, detached_, state_); if (pimpl_ != nullptr) XBT_INFO("pimpl_->state: %d", pimpl_->state); else XBT_INFO("pimpl_ is null"); xbt_backtrace_display_current(); } } Activity* Comm::setRate(double rate) { xbt_assert(state_==inited); rate_ = rate; return this; } Activity* Comm::setSrcData(void* buff) { xbt_assert(state_==inited); xbt_assert(dstBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); srcBuff_ = buff; return this; } Activity* Comm::setSrcDataSize(size_t size) { xbt_assert(state_==inited); srcBuffSize_ = size; return this; } Activity* Comm::setSrcData(void* buff, size_t size) { xbt_assert(state_==inited); xbt_assert(dstBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); srcBuff_ = buff; srcBuffSize_ = size; return this; } Activity* Comm::setDstData(void** buff) { xbt_assert(state_==inited); xbt_assert(srcBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); dstBuff_ = buff; return this; } size_t Comm::getDstDataSize(){ xbt_assert(state_==finished); return dstBuffSize_; } Activity* Comm::setDstData(void** buff, size_t size) { xbt_assert(state_==inited); xbt_assert(srcBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); dstBuff_ = buff; dstBuffSize_ = size; return this; } Activity* Comm::start() { xbt_assert(state_ == inited); if (srcBuff_ != nullptr) { // Sender side pimpl_ = simcall_comm_isend(sender_, mailbox_->getImpl(), remains_, rate_, srcBuff_, srcBuffSize_, matchFunction_, cleanFunction_, copyDataFunction_, userData_, detached_); } else if (dstBuff_ != nullptr) { // Receiver side xbt_assert(not detached_, "Receive cannot be detached"); pimpl_ = simcall_comm_irecv(receiver_, mailbox_->getImpl(), dstBuff_, &dstBuffSize_, matchFunction_, copyDataFunction_, userData_, rate_); } else { xbt_die("Cannot start a communication before specifying whether we are the sender or the receiver"); } state_ = started; return this; } /** @brief Block the calling actor until the communication is finished */ Activity* Comm::wait() { return this->wait(-1); } /** @brief Block the calling actor until the communication is finished, or until timeout * * On timeout, an exception is thrown. * * @param timeout the amount of seconds to wait for the comm termination. * Negative values denote infinite wait times. 0 as a timeout returns immediately. */ Activity* Comm::wait(double timeout) { switch (state_) { case finished: return this; case inited: // It's not started yet. Do it in one simcall if (srcBuff_ != nullptr) { simcall_comm_send(sender_, mailbox_->getImpl(), remains_, rate_, srcBuff_, srcBuffSize_, matchFunction_, copyDataFunction_, userData_, timeout); } else { // Receiver simcall_comm_recv(receiver_, mailbox_->getImpl(), dstBuff_, &dstBuffSize_, matchFunction_, copyDataFunction_, userData_, timeout, rate_); } state_ = finished; return this; case started: simcall_comm_wait(pimpl_, timeout); state_ = finished; return this; default: THROW_IMPOSSIBLE; } return this; } Activity* Comm::detach() { xbt_assert(state_ == inited, "You cannot detach communications once they are started."); xbt_assert(srcBuff_ != nullptr && srcBuffSize_ != 0, "You can only detach sends, not recvs"); detached_ = true; return start(); } Activity* Comm::cancel() { simgrid::kernel::activity::CommImplPtr commPimpl = boost::static_pointer_cast(pimpl_); commPimpl->cancel(); return this; } bool Comm::test() { xbt_assert(state_ == inited || state_ == started || state_ == finished); if (state_ == finished) { return true; } if (state_ == inited) { this->start(); } if(simcall_comm_test(pimpl_)){ state_ = finished; return true; } return false; } MailboxPtr Comm::getMailbox() { return mailbox_; } void intrusive_ptr_release(simgrid::s4u::Comm* c) { if (c->refcount_.fetch_sub(1, std::memory_order_release) == 1) { std::atomic_thread_fence(std::memory_order_acquire); delete c; } } void intrusive_ptr_add_ref(simgrid::s4u::Comm* c) { c->refcount_.fetch_add(1, std::memory_order_relaxed); } } } // namespaces SimGrid-3.18/src/s4u/s4u_conditionVariable.cpp0000644000175000017500000000431613217757316021564 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "simgrid/s4u/ConditionVariable.hpp" #include "simgrid/simix.h" #include "src/simix/smx_synchro_private.hpp" namespace simgrid { namespace s4u { ConditionVariablePtr ConditionVariable::createConditionVariable() { smx_cond_t cond = simcall_cond_init(); return ConditionVariablePtr(&cond->cond_, false); } /** * Wait functions */ void ConditionVariable::wait(MutexPtr lock) { simcall_cond_wait(cond_, lock->mutex_); } void ConditionVariable::wait(std::unique_lock& lock) { simcall_cond_wait(cond_, lock.mutex()->mutex_); } std::cv_status s4u::ConditionVariable::wait_for(std::unique_lock& lock, double timeout) { // The simcall uses -1 for "any timeout" but we don't want this: if (timeout < 0) timeout = 0.0; try { simcall_cond_wait_timeout(cond_, lock.mutex()->mutex_, timeout); return std::cv_status::no_timeout; } catch (xbt_ex& e) { // If the exception was a timeout, we have to take the lock again: if (e.category == timeout_error) { try { lock.mutex()->lock(); return std::cv_status::timeout; } catch (...) { std::terminate(); } } // Another exception: should we reaquire the lock? std::terminate(); } catch (...) { std::terminate(); } } std::cv_status ConditionVariable::wait_until(std::unique_lock& lock, double timeout_time) { double now = SIMIX_get_clock(); double timeout; if (timeout_time < now) timeout = 0.0; else timeout = timeout_time - now; return this->wait_for(lock, timeout); } /** * Notify functions */ void ConditionVariable::notify_one() { simcall_cond_signal(cond_); } void ConditionVariable::notify_all() { simcall_cond_broadcast(cond_); } void intrusive_ptr_add_ref(ConditionVariable* cond) { intrusive_ptr_add_ref(cond->cond_); } void intrusive_ptr_release(ConditionVariable* cond) { intrusive_ptr_release(cond->cond_); } } } SimGrid-3.18/src/s4u/s4u_host.cpp0000644000175000017500000001733213217757316017107 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "simgrid/s4u/Engine.hpp" #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/Storage.hpp" #include "simgrid/simix.hpp" #include "src/kernel/routing/NetPoint.hpp" #include "src/msg/msg_private.hpp" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_private.hpp" #include "src/surf/HostImpl.hpp" #include "src/surf/cpu_interface.hpp" #include "xbt/log.h" XBT_LOG_EXTERNAL_CATEGORY(surf_route); int USER_HOST_LEVEL = -1; namespace simgrid { namespace xbt { template class Extendable; } namespace s4u { std::map host_list; // FIXME: move it to Engine simgrid::xbt::signal Host::onCreation; simgrid::xbt::signal Host::onDestruction; simgrid::xbt::signal Host::onStateChange; simgrid::xbt::signal Host::onSpeedChange; Host::Host(const char* name) : name_(name) { xbt_assert(Host::by_name_or_null(name) == nullptr, "Refusing to create a second host named '%s'.", name); host_list[name_] = this; new simgrid::surf::HostImpl(this); } Host::~Host() { xbt_assert(currentlyDestroying_, "Please call h->destroy() instead of manually deleting it."); delete pimpl_; if (pimpl_netpoint != nullptr) // not removed yet by a children class simgrid::s4u::Engine::getInstance()->netpointUnregister(pimpl_netpoint); delete pimpl_cpu; delete mounts; } /** @brief Fire the required callbacks and destroy the object * * Don't delete directly an Host, call h->destroy() instead. * * This is cumbersome but this is the simplest solution to ensure that the * onDestruction() callback receives a valid object (because of the destructor * order in a class hierarchy). */ void Host::destroy() { if (not currentlyDestroying_) { currentlyDestroying_ = true; onDestruction(*this); host_list.erase(name_); delete this; } } Host* Host::by_name(std::string name) { return host_list.at(name); // Will raise a std::out_of_range if the host does not exist } Host* Host::by_name(const char* name) { return host_list.at(std::string(name)); // Will raise a std::out_of_range if the host does not exist } Host* Host::by_name_or_null(const char* name) { return by_name_or_null(std::string(name)); } Host* Host::by_name_or_null(std::string name) { auto host = host_list.find(name); return host == host_list.end() ? nullptr : host->second; } Host *Host::current(){ smx_actor_t smx_proc = SIMIX_process_self(); if (smx_proc == nullptr) xbt_die("Cannot call Host::current() from the maestro context"); return smx_proc->host; } void Host::turnOn() { if (isOff()) { simgrid::simix::kernelImmediate([this] { this->extension()->turnOn(); this->pimpl_cpu->turnOn(); onStateChange(*this); }); } } void Host::turnOff() { if (isOn()) { smx_actor_t self = SIMIX_process_self(); simgrid::simix::kernelImmediate([this, self] { SIMIX_host_off(this, self); onStateChange(*this); }); } } bool Host::isOn() { return this->pimpl_cpu->isOn(); } int Host::getPstatesCount() const { return this->pimpl_cpu->getNbPStates(); } /** * \brief Return the list of actors attached to an host. * * \param whereto a vector in which we should push actors living on that host */ void Host::actorList(std::vector* whereto) { for (auto& actor : this->extension()->process_list) { whereto->push_back(actor.ciface()); } } /** * \brief Find a route toward another host * * \param dest [IN] where to * \param links [OUT] where to store the list of links (must exist, cannot be nullptr). * \param latency [OUT] where to store the latency experienced on the path (or nullptr if not interested) * It is the caller responsibility to initialize latency to 0 (we add to provided route) * \pre links!=nullptr * * walk through the routing components tree and find a route between hosts * by calling each "get_route" function in each routing component. */ void Host::routeTo(Host* dest, std::vector& links, double* latency) { std::vector linkImpls; this->routeTo(dest, linkImpls, latency); for (surf::LinkImpl* const& l : linkImpls) links.push_back(&l->piface_); } /** @brief Just like Host::routeTo, but filling an array of link implementations */ void Host::routeTo(Host* dest, std::vector& links, double* latency) { simgrid::kernel::routing::NetZoneImpl::getGlobalRoute(pimpl_netpoint, dest->pimpl_netpoint, links, latency); if (XBT_LOG_ISENABLED(surf_route, xbt_log_priority_debug)) { XBT_CDEBUG(surf_route, "Route from '%s' to '%s' (latency: %f):", getCname(), dest->getCname(), (latency == nullptr ? -1 : *latency)); for (auto const& link : links) XBT_CDEBUG(surf_route, "Link %s", link->getCname()); } } /** Get the properties assigned to a host */ std::map* Host::getProperties() { return simgrid::simix::kernelImmediate([this] { return this->pimpl_->getProperties(); }); } /** Retrieve the property value (or nullptr if not set) */ const char* Host::getProperty(const char* key) { return this->pimpl_->getProperty(key); } void Host::setProperty(std::string key, std::string value) { simgrid::simix::kernelImmediate([this, key, value] { this->pimpl_->setProperty(key, value); }); } /** Get the processes attached to the host */ void Host::getProcesses(std::vector* list) { for (auto& actor : this->extension()->process_list) { list->push_back(actor.iface()); } } /** @brief Get the peak processor speed (in flops/s), at the specified pstate */ double Host::getPstateSpeed(int pstate_index) { return simgrid::simix::kernelImmediate([this, pstate_index] { return this->pimpl_cpu->getPstateSpeed(pstate_index); }); } /** @brief Get the peak processor speed (in flops/s), at the current pstate */ double Host::getSpeed() { return pimpl_cpu->getSpeed(1.0); } /** @brief Returns the number of core of the processor. */ int Host::getCoreCount() { return pimpl_cpu->coreCount(); } /** @brief Set the pstate at which the host should run */ void Host::setPstate(int pstate_index) { simgrid::simix::kernelImmediate([this, pstate_index] { this->pimpl_cpu->setPState(pstate_index); }); } /** @brief Retrieve the pstate at which the host is currently running */ int Host::getPstate() { return this->pimpl_cpu->getPState(); } /** * \ingroup simix_storage_management * \brief Returns the list of storages attached to an host. * \return a vector containing all storages attached to the host */ void Host::getAttachedStorages(std::vector* storages) { simgrid::simix::kernelImmediate([this, storages] { this->pimpl_->getAttachedStorageList(storages); }); } std::unordered_map const& Host::getMountedStorages() { if (mounts == nullptr) { mounts = new std::unordered_map(); for (auto const& m : this->pimpl_->storage_) { mounts->insert({m.first, &m.second->piface_}); } } return *mounts; } void Host::execute(double flops) { Host* host_list[1] = {this}; double flops_list[1] = {flops}; smx_activity_t s = simcall_execution_parallel_start(nullptr /*name*/, 1, host_list, flops_list, nullptr /*comm_sizes */, -1.0, -1 /*timeout*/); simcall_execution_wait(s); } double Host::getLoad() { return this->pimpl_cpu->getLoad(); } } // namespace simgrid } // namespace s4u SimGrid-3.18/src/s4u/s4u_actor.cpp0000644000175000017500000002077013217757316017242 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "simgrid/s4u/Actor.hpp" #include "simgrid/s4u/Comm.hpp" #include "simgrid/s4u/Exec.hpp" #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "src/kernel/context/Context.hpp" #include "src/simix/smx_private.hpp" #include XBT_LOG_NEW_DEFAULT_CATEGORY(s4u_actor, "S4U actors"); namespace simgrid { namespace s4u { // ***** Actor creation ***** ActorPtr Actor::self() { smx_context_t self_context = SIMIX_context_self(); if (self_context == nullptr) return simgrid::s4u::ActorPtr(); return self_context->process()->iface(); } ActorPtr Actor::createActor(const char* name, s4u::Host* host, std::function code) { simgrid::simix::ActorImpl* actor = simcall_process_create(name, std::move(code), nullptr, host, nullptr); return actor->iface(); } ActorPtr Actor::createActor(const char* name, s4u::Host* host, const char* function, std::vector args) { simgrid::simix::ActorCodeFactory& factory = SIMIX_get_actor_code_factory(function); simgrid::simix::ActorCode code = factory(std::move(args)); simgrid::simix::ActorImpl* actor = simcall_process_create(name, std::move(code), nullptr, host, nullptr); return actor->iface(); } void intrusive_ptr_add_ref(Actor* actor) { intrusive_ptr_add_ref(actor->pimpl_); } void intrusive_ptr_release(Actor* actor) { intrusive_ptr_release(actor->pimpl_); } // ***** Actor methods ***** void Actor::join() { simcall_process_join(this->pimpl_, -1); } void Actor::join(double timeout) { simcall_process_join(this->pimpl_, timeout); } void Actor::setAutoRestart(bool autorestart) { simgrid::simix::kernelImmediate([this, autorestart]() { pimpl_->auto_restart = autorestart; }); } void Actor::onExit(int_f_pvoid_pvoid_t fun, void* data) { simcall_process_on_exit(pimpl_, fun, data); } void Actor::migrate(Host* new_host) { simgrid::simix::kernelImmediate([this, new_host]() { pimpl_->new_host = new_host; }); } s4u::Host* Actor::getHost() { return this->pimpl_->host; } void Actor::daemonize() { simgrid::simix::kernelImmediate([this]() { pimpl_->daemonize(); }); } const simgrid::xbt::string& Actor::getName() const { return this->pimpl_->getName(); } const char* Actor::getCname() const { return this->pimpl_->getCname(); } aid_t Actor::getPid() { return this->pimpl_->pid; } aid_t Actor::getPpid() { return this->pimpl_->ppid; } void Actor::suspend() { simcall_process_suspend(pimpl_); } void Actor::resume() { simgrid::simix::kernelImmediate([this] { pimpl_->resume(); }); } int Actor::isSuspended() { return simgrid::simix::kernelImmediate([this] { return pimpl_->suspended; }); } void Actor::setKillTime(double time) { simcall_process_set_kill_time(pimpl_,time); } /** \brief Get the kill time of an actor(or 0 if unset). */ double Actor::getKillTime() { return SIMIX_timer_get_date(pimpl_->kill_timer); } void Actor::kill(aid_t pid) { smx_actor_t process = SIMIX_process_from_PID(pid); if(process != nullptr) { simgrid::simix::kernelImmediate([process] { SIMIX_process_kill(process, process); }); } else { std::ostringstream oss; oss << "kill: ("<< pid <<") - No such process" << std::endl; throw std::runtime_error(oss.str()); } } smx_actor_t Actor::getImpl() { return pimpl_; } void Actor::kill() { smx_actor_t process = SIMIX_process_self(); simgrid::simix::kernelImmediate( [this, process] { SIMIX_process_kill(pimpl_, (pimpl_ == simix_global->maestro_process) ? pimpl_ : process); }); } // ***** Static functions ***** ActorPtr Actor::byPid(aid_t pid) { smx_actor_t process = SIMIX_process_from_PID(pid); if (process != nullptr) return process->iface(); else return ActorPtr(); } void Actor::killAll() { simcall_process_killall(1); } void Actor::killAll(int resetPid) { simcall_process_killall(resetPid); } std::map* Actor::getProperties() { return simgrid::simix::kernelImmediate([this] { return this->pimpl_->getProperties(); }); } /** Retrieve the property value (or nullptr if not set) */ const char* Actor::getProperty(const char* key) { return simgrid::simix::kernelImmediate([this, key] { return pimpl_->getProperty(key); }); } void Actor::setProperty(const char* key, const char* value) { simgrid::simix::kernelImmediate([this, key, value] { pimpl_->setProperty(key, value); }); } Actor* Actor::restart() { return simgrid::simix::kernelImmediate([this]() { return pimpl_->restart(); }); } // ***** this_actor ***** namespace this_actor { /** Returns true if run from the kernel mode, and false if run from a real actor * * Everything that is run out of any actor (simulation setup before the engine is run, * computing the model evolutions as a result to the actors' action, etc) is run in * kernel mode, just as in any operating systems. * * In SimGrid, the actor in charge of doing the stuff in kernel mode is called Maestro, * because it is the one scheduling when the others should move or wait. */ bool isMaestro() { smx_actor_t process = SIMIX_process_self(); return process == nullptr || process == simix_global->maestro_process; } void sleep_for(double duration) { if (duration > 0) simcall_process_sleep(duration); } void yield() { simgrid::simix::kernelImmediate([] { /* do nothing*/ }); } XBT_PUBLIC(void) sleep_until(double timeout) { double now = SIMIX_get_clock(); if (timeout > now) simcall_process_sleep(timeout - now); } void execute(double flops) { smx_activity_t s = simcall_execution_start(nullptr, flops, 1.0 /*priority*/, 0. /*bound*/, getHost()); simcall_execution_wait(s); } void execute(double flops, double priority) { smx_activity_t s = simcall_execution_start(nullptr, flops, 1 / priority /*priority*/, 0. /*bound*/, getHost()); simcall_execution_wait(s); } void parallel_execute(int host_nb, sg_host_t* host_list, double* flops_amount, double* bytes_amount, double timeout) { smx_activity_t s = simcall_execution_parallel_start(nullptr, host_nb, host_list, flops_amount, bytes_amount, -1, timeout); simcall_execution_wait(s); } void parallel_execute(int host_nb, sg_host_t* host_list, double* flops_amount, double* bytes_amount) { smx_activity_t s = simcall_execution_parallel_start(nullptr, host_nb, host_list, flops_amount, bytes_amount, -1, -1); simcall_execution_wait(s); } ExecPtr exec_init(double flops_amount) { ExecPtr res = ExecPtr(new Exec()); res->host_ = getHost(); res->flops_amount_ = flops_amount; res->setRemains(flops_amount); return res; } ExecPtr exec_async(double flops) { ExecPtr res = exec_init(flops); res->start(); return res; } void* recv(MailboxPtr chan) // deprecated { return chan->get(); } void* recv(MailboxPtr chan, double timeout) // deprecated { return chan->get(timeout); } void send(MailboxPtr chan, void* payload, double simulatedSize) // deprecated { chan->put(payload, simulatedSize); } void send(MailboxPtr chan, void* payload, double simulatedSize, double timeout) // deprecated { chan->put(payload, simulatedSize, timeout); } CommPtr isend(MailboxPtr chan, void* payload, double simulatedSize) // deprecated { return chan->put_async(payload, simulatedSize); } CommPtr irecv(MailboxPtr chan, void** data) // deprecated { return chan->get_async(data); } aid_t getPid() { return SIMIX_process_self()->pid; } aid_t getPpid() { return SIMIX_process_self()->ppid; } std::string getName() { return SIMIX_process_self()->getName(); } const char* getCname() { return SIMIX_process_self()->getCname(); } Host* getHost() { return SIMIX_process_self()->host; } void suspend() { simcall_process_suspend(SIMIX_process_self()); } void resume() { smx_actor_t process = SIMIX_process_self(); simgrid::simix::kernelImmediate([process] { process->resume(); }); } bool isSuspended() { smx_actor_t process = SIMIX_process_self(); return simgrid::simix::kernelImmediate([process] { return process->suspended; }); } void kill() { smx_actor_t process = SIMIX_process_self(); simgrid::simix::kernelImmediate([process] { SIMIX_process_kill(process, process); }); } void onExit(int_f_pvoid_pvoid_t fun, void* data) { simcall_process_on_exit(SIMIX_process_self(), fun, data); } void migrate(Host* new_host) { smx_actor_t process = SIMIX_process_self(); simgrid::simix::kernelImmediate([process, new_host] { process->new_host = new_host; }); } } } } SimGrid-3.18/src/s4u/s4u_netzone.cpp0000644000175000017500000000514513217757316017613 0ustar mquinsonmquinson/* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/NetZone.hpp" #include "simgrid/simix.hpp" #include "src/kernel/routing/NetPoint.hpp" #include "src/surf/network_interface.hpp" // Link FIXME: move to proper header XBT_LOG_NEW_DEFAULT_CATEGORY(s4u_netzone, "S4U Networking Zones"); namespace simgrid { namespace s4u { simgrid::xbt::signal& link_list)> NetZone::onRouteCreation; simgrid::xbt::signal NetZone::onCreation; simgrid::xbt::signal NetZone::onSeal; NetZone::NetZone(NetZone* father, std::string name) : father_(father), name_(name) { children_ = new std::vector(); } void NetZone::seal() { sealed_ = true; } NetZone::~NetZone() { for (auto const& nz : *children_) delete nz; delete children_; } std::unordered_map* NetZone::getProperties() { return simgrid::simix::kernelImmediate([this] { return &properties_; }); } /** Retrieve the property value (or nullptr if not set) */ const char* NetZone::getProperty(const char* key) { return properties_.at(key).c_str(); } void NetZone::setProperty(const char* key, const char* value) { simgrid::simix::kernelImmediate([this,key,value] { properties_[key] = value; }); } std::vector* NetZone::getChildren() { return children_; } const char* NetZone::getCname() const { return name_.c_str(); } NetZone* NetZone::getFather() { return father_; } void NetZone::getHosts(std::vector* whereto) { for (auto const& card : vertices_) { s4u::Host* host = simgrid::s4u::Host::by_name_or_null(card->getName()); if (host != nullptr) whereto->push_back(host); } } int NetZone::addComponent(kernel::routing::NetPoint* elm) { vertices_.push_back(elm); return vertices_.size() - 1; // The rank of the newly created object } void NetZone::addRoute(sg_netpoint_t /*src*/, sg_netpoint_t /*dst*/, sg_netpoint_t /*gw_src*/, sg_netpoint_t /*gw_dst*/, std::vector& /*link_list*/, bool /*symmetrical*/) { xbt_die("NetZone '%s' does not accept new routes (wrong class).", name_.c_str()); } } }; // namespace simgrid::s4u SimGrid-3.18/src/s4u/s4u_engine.cpp0000644000175000017500000001143113217757316017371 0ustar mquinsonmquinson/* s4u::Engine Simulation Engine and global functions. */ /* Copyright (c) 2006-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "instr/instr_interface.h" #include "mc/mc.h" #include "simgrid/s4u/Engine.hpp" #include "simgrid/s4u/Host.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "simgrid/s4u/NetZone.hpp" #include "simgrid/s4u/Storage.hpp" #include "simgrid/simix.h" #include "src/kernel/EngineImpl.hpp" #include "src/kernel/routing/NetPoint.hpp" #include "src/kernel/routing/NetZoneImpl.hpp" #include "src/surf/network_interface.hpp" #include "surf/surf.hpp" // routing_platf. FIXME:KILLME. SOON XBT_LOG_NEW_CATEGORY(s4u,"Log channels of the S4U (Simgrid for you) interface"); namespace simgrid { namespace s4u { xbt::signal onPlatformCreated; xbt::signal onSimulationEnd; xbt::signal onTimeAdvance; xbt::signal onDeadlock; Engine *Engine::instance_ = nullptr; /* That singleton is awful, but I don't see no other solution right now. */ Engine::Engine(int *argc, char **argv) { xbt_assert(s4u::Engine::instance_ == nullptr, "It is currently forbidden to create more than one instance of s4u::Engine"); TRACE_global_init(); SIMIX_global_init(argc, argv); pimpl = new kernel::EngineImpl(); s4u::Engine::instance_ = this; } Engine::~Engine() { delete pimpl; s4u::Engine::instance_ = nullptr; } Engine* Engine::getInstance() { if (s4u::Engine::instance_ == nullptr) return new Engine(0, nullptr); else return s4u::Engine::instance_; } void Engine::shutdown() { delete s4u::Engine::instance_; s4u::Engine::instance_ = nullptr; } double Engine::getClock() { return SIMIX_get_clock(); } void Engine::loadPlatform(const char *platf) { SIMIX_create_environment(platf); } void Engine::registerFunction(const char*name, int (*code)(int,char**)) { SIMIX_function_register(name,code); } void Engine::registerDefault(int (*code)(int,char**)) { SIMIX_function_register_default(code); } void Engine::loadDeployment(const char *deploy) { SIMIX_launch_application(deploy); } // FIXME: The following duplicates the content of s4u::Host extern std::map host_list; /** @brief Returns the amount of hosts in the platform */ size_t Engine::getHostCount() { return host_list.size(); } /** @brief Fills the passed list with all hosts found in the platform */ void Engine::getHostList(std::vector* list) { for (auto const& kv : host_list) list->push_back(kv.second); } /** @brief Returns the amount of links in the platform */ size_t Engine::getLinkCount() { return simgrid::surf::LinkImpl::linksCount(); } /** @brief Fills the passed list with all hosts found in the platform */ void Engine::getLinkList(std::vector* list) { simgrid::surf::LinkImpl::linksList(list); } void Engine::run() { if (MC_is_active()) { MC_run(); } else { SIMIX_run(); } } s4u::NetZone* Engine::getNetRoot() { return pimpl->netRoot_; } static s4u::NetZone* netzoneByNameRecursive(s4u::NetZone* current, const char* name) { if (not strcmp(current->getCname(), name)) return current; for (auto const& elem : *(current->getChildren())) { simgrid::s4u::NetZone* tmp = netzoneByNameRecursive(elem, name); if (tmp != nullptr) { return tmp; } } return nullptr; } /** @brief Retrieve the NetZone of the given name (or nullptr if not found) */ NetZone* Engine::getNetzoneByNameOrNull(const char* name) { return netzoneByNameRecursive(getNetRoot(), name); } /** @brief Retrieve the netpoint of the given name (or nullptr if not found) */ simgrid::kernel::routing::NetPoint* Engine::getNetpointByNameOrNull(std::string name) { auto netp = pimpl->netpoints_.find(name); return netp == pimpl->netpoints_.end() ? nullptr : netp->second; } /** @brief Fill the provided vector with all existing netpoints */ void Engine::getNetpointList(std::vector* list) { for (auto const& kv : pimpl->netpoints_) list->push_back(kv.second); } /** @brief Register a new netpoint to the system */ void Engine::netpointRegister(simgrid::kernel::routing::NetPoint* point) { // simgrid::simix::kernelImmediate([&]{ FIXME: this segfaults in set_thread pimpl->netpoints_[point->getName()] = point; // }); } /** @brief Unregister a given netpoint */ void Engine::netpointUnregister(simgrid::kernel::routing::NetPoint* point) { simgrid::simix::kernelImmediate([this, point] { pimpl->netpoints_.erase(point->getName()); delete point; }); } bool Engine::isInitialized() { return Engine::instance_ != nullptr; } void Engine::setConfig(std::string str) { xbt_cfg_set_parse(str.c_str()); } } } // namespace SimGrid-3.18/src/s4u/s4u_link.cpp0000644000175000017500000000743313217757316017070 0ustar mquinsonmquinson/* Copyright (c) 2013-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "simgrid/s4u/Link.hpp" #include "simgrid/sg_config.h" #include "simgrid/simix.hpp" #include "src/kernel/lmm/maxmin.hpp" #include "src/surf/network_interface.hpp" #include "xbt/log.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_link, s4u, "Logging specific to the S4U links"); /********* * C API * *********/ extern "C" { const char* sg_link_name(sg_link_t link) { return link->getCname(); } sg_link_t sg_link_by_name(const char* name) { return simgrid::s4u::Link::byName(name); } int sg_link_is_shared(sg_link_t link) { return link->sharingPolicy(); } double sg_link_bandwidth(sg_link_t link) { return link->bandwidth(); } double sg_link_latency(sg_link_t link) { return link->latency(); } void* sg_link_data(sg_link_t link) { return link->getData(); } void sg_link_data_set(sg_link_t link, void* data) { link->setData(data); } int sg_link_count() { return simgrid::surf::LinkImpl::linksCount(); } sg_link_t* sg_link_list() { simgrid::surf::LinkImpl** list = simgrid::surf::LinkImpl::linksList(); sg_link_t* res = (sg_link_t*)list; // Use the same memory area int size = sg_link_count(); for (int i = 0; i < size; i++) res[i] = &(list[i]->piface_); // Convert each entry into its interface return res; } void sg_link_exit() { simgrid::surf::LinkImpl::linksExit(); } } /*********** * C++ API * ***********/ namespace simgrid { namespace s4u { Link* Link::byName(const char* name) { surf::LinkImpl* res = surf::LinkImpl::byName(name); if (res == nullptr) return nullptr; return &res->piface_; } const std::string& Link::getName() const { return this->pimpl_->getName(); } const char* Link::getCname() const { return this->pimpl_->getCname(); } const char* Link::name() { return getCname(); } bool Link::isUsed() { return this->pimpl_->isUsed(); } double Link::latency() { return this->pimpl_->latency(); } double Link::bandwidth() { return this->pimpl_->bandwidth(); } int Link::sharingPolicy() { return this->pimpl_->sharingPolicy(); } double Link::getUsage() { return this->pimpl_->constraint()->get_usage(); } void Link::turnOn() { simgrid::simix::kernelImmediate([this]() { this->pimpl_->turnOn(); }); } void Link::turnOff() { simgrid::simix::kernelImmediate([this]() { this->pimpl_->turnOff(); }); } void* Link::getData() { return this->pimpl_->getData(); } void Link::setData(void* d) { simgrid::simix::kernelImmediate([this, d]() { this->pimpl_->setData(d); }); } void Link::setStateTrace(tmgr_trace_t trace) { simgrid::simix::kernelImmediate([this, trace]() { this->pimpl_->setStateTrace(trace); }); } void Link::setBandwidthTrace(tmgr_trace_t trace) { simgrid::simix::kernelImmediate([this, trace]() { this->pimpl_->setBandwidthTrace(trace); }); } void Link::setLatencyTrace(tmgr_trace_t trace) { simgrid::simix::kernelImmediate([this, trace]() { this->pimpl_->setLatencyTrace(trace); }); } const char* Link::getProperty(const char* key) { return this->pimpl_->getProperty(key); } void Link::setProperty(std::string key, std::string value) { simgrid::simix::kernelImmediate([this, key, value] { this->pimpl_->setProperty(key, value); }); } /************* * Callbacks * *************/ simgrid::xbt::signal Link::onCreation; simgrid::xbt::signal Link::onDestruction; simgrid::xbt::signal Link::onStateChange; simgrid::xbt::signal Link::onCommunicate; simgrid::xbt::signal Link::onCommunicationStateChange; } } SimGrid-3.18/src/instr/0000755000175000017500000000000013217757321015245 5ustar mquinsonmquinsonSimGrid-3.18/src/instr/instr_paje_types.hpp0000644000175000017500000000611113217757321021337 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PAJE_TYPES_HPP #define INSTR_PAJE_TYPES_HPP #include "src/instr/instr_private.hpp" #include #include namespace simgrid { namespace instr { class ContainerType; class EventType; class Type { long long int id_; std::string name_; std::string color_; Type* father_; public: std::map children_; Container* issuer_ = nullptr; Type(std::string name, std::string alias, std::string color, Type* father); virtual ~Type(); std::string getName() { return name_; } const char* getCname() { return name_.c_str(); } long long int getId() { return id_; } bool isColored() { return not color_.empty(); } Type* byName(std::string name); ContainerType* getOrCreateContainerType(std::string name); EventType* getOrCreateEventType(std::string name); LinkType* getOrCreateLinkType(std::string name, Type* source, Type* dest); StateType* getOrCreateStateType(std::string name); VariableType* getOrCreateVariableType(std::string name, std::string color); void setCallingContainer(Container* container) { issuer_ = container; } void logDefinition(e_event_type event_type); void logDefinition(Type* source, Type* dest); }; class ContainerType : public Type { public: explicit ContainerType(std::string name) : Type(name, name, "", nullptr){}; ContainerType(std::string name, Type* father); }; class VariableType : public Type { std::vector events_; public: VariableType(std::string name, std::string color, Type* father); ~VariableType(); void setEvent(double timestamp, double value); void addEvent(double timestamp, double value); void subEvent(double timestamp, double value); }; class ValueType : public Type { public: std::map values_; ValueType(std::string name, std::string alias, Type* father) : Type(name, alias, "", father){}; ValueType(std::string name, Type* father) : Type(name, name, "", father){}; virtual ~ValueType(); void addEntityValue(std::string name, std::string color); void addEntityValue(std::string name); EntityValue* getEntityValue(std::string name); }; class LinkType : public ValueType { public: LinkType(std::string name, std::string alias, Type* father); void startEvent(Container* startContainer, std::string value, std::string key); void startEvent(Container* startContainer, std::string value, std::string key, int size); void endEvent(Container* endContainer, std::string value, std::string key); }; class EventType : public ValueType { public: EventType(std::string name, Type* father); }; class StateType : public ValueType { std::vector events_; public: StateType(std::string name, Type* father); ~StateType(); void setEvent(std::string value_name); void pushEvent(std::string value_name); void pushEvent(std::string value_name, TIData* extra); void popEvent(); }; } } #endif SimGrid-3.18/src/instr/instr_paje_types.cpp0000644000175000017500000001711213217757321021335 0ustar mquinsonmquinson/* Copyright (c) 2012, 2014-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_types, instr, "Paje tracing event system (types)"); extern FILE* tracing_file; namespace simgrid { namespace instr { Type::Type(std::string name, std::string alias, std::string color, Type* father) : id_(instr_new_paje_id()), name_(name), color_(color), father_(father) { if (name.empty() || alias.empty()) THROWF(tracing_error, 0, "can't create a new type with no name or alias"); if (father != nullptr){ father->children_.insert({alias, this}); XBT_DEBUG("new type %s, child of %s", name_.c_str(), father->getCname()); } } Type::~Type() { for (auto elm : children_) delete elm.second; } ValueType::~ValueType() { for (auto elm : values_) delete elm.second; } ContainerType::ContainerType(std::string name, Type* father) : Type(name, name, "", father) { XBT_DEBUG("ContainerType %s(%lld), child of %s(%lld)", getCname(), getId(), father->getCname(), father->getId()); logDefinition(PAJE_DefineContainerType); } EventType::EventType(std::string name, Type* father) : ValueType(name, father) { XBT_DEBUG("EventType %s(%lld), child of %s(%lld)", getCname(), getId(), father->getCname(), father->getId()); logDefinition(PAJE_DefineEventType); } StateType::StateType(std::string name, Type* father) : ValueType(name, father) { XBT_DEBUG("StateType %s(%lld), child of %s(%lld)", getCname(), getId(), father->getCname(), father->getId()); logDefinition(PAJE_DefineStateType); } StateType::~StateType() { events_.clear(); } void StateType::setEvent(std::string value_name) { events_.push_back(new StateEvent(issuer_, this, PAJE_SetState, getEntityValue(value_name))); } void StateType::pushEvent(std::string value_name, TIData* extra) { events_.push_back(new StateEvent(issuer_, this, PAJE_PushState, getEntityValue(value_name), extra)); } void StateType::pushEvent(std::string value_name) { events_.push_back(new StateEvent(issuer_, this, PAJE_PushState, getEntityValue(value_name))); } void StateType::popEvent() { events_.push_back(new StateEvent(issuer_, this, PAJE_PopState, nullptr)); } VariableType::VariableType(std::string name, std::string color, Type* father) : Type(name, name, color, father) { XBT_DEBUG("VariableType %s(%lld), child of %s(%lld)", getCname(), getId(), father->getCname(), father->getId()); logDefinition(PAJE_DefineVariableType); } VariableType::~VariableType() { events_.clear(); } void VariableType::setEvent(double timestamp, double value) { events_.push_back(new VariableEvent(timestamp, issuer_, this, PAJE_SetVariable, value)); } void VariableType::addEvent(double timestamp, double value) { events_.push_back(new VariableEvent(timestamp, issuer_, this, PAJE_AddVariable, value)); } void VariableType::subEvent(double timestamp, double value) { events_.push_back(new VariableEvent(timestamp, issuer_, this, PAJE_SubVariable, value)); } LinkType::LinkType(std::string name, std::string alias, Type* father) : ValueType(name, alias, father) { } void LinkType::startEvent(container_t startContainer, std::string value, std::string key) { startEvent(startContainer, value, key, -1); } void LinkType::startEvent(container_t startContainer, std::string value, std::string key, int size) { new LinkEvent(issuer_, this, PAJE_StartLink, startContainer, value, key, size); } void LinkType::endEvent(container_t endContainer, std::string value, std::string key) { new LinkEvent(issuer_, this, PAJE_EndLink, endContainer, value, key); } void Type::logDefinition(e_event_type event_type) { if (instr_fmt_type != instr_fmt_paje) return; std::stringstream stream; XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, event_type, TRACE_precision(), 0.); stream << std::fixed << std::setprecision(TRACE_precision()) << event_type << " " << getId(); stream << " " << father_->getId() << " " << getName(); if (isColored()) stream << " \"" << color_ << "\""; XBT_DEBUG("Dump %s", stream.str().c_str()); stream << std::endl; fprintf(tracing_file, "%s", stream.str().c_str()); } void Type::logDefinition(simgrid::instr::Type* source, simgrid::instr::Type* dest) { if (instr_fmt_type != instr_fmt_paje) return; std::stringstream stream; XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, PAJE_DefineLinkType, TRACE_precision(), 0.); stream << std::fixed << std::setprecision(TRACE_precision()) << PAJE_DefineLinkType << " " << getId(); stream << " " << father_->getId() << " " << source->getId() << " " << dest->getId() << " " << getName(); XBT_DEBUG("Dump %s", stream.str().c_str()); stream << std::endl; fprintf(tracing_file, "%s", stream.str().c_str()); } Type* Type::byName(std::string name) { Type* ret = nullptr; for (auto elm : children_) { if (elm.second->name_ == name) { if (ret != nullptr) { THROWF (tracing_error, 0, "there are two children types with the same name?"); } else { ret = elm.second; } } } if (ret == nullptr) THROWF(tracing_error, 2, "type with name (%s) not found in father type (%s)", name.c_str(), getCname()); return ret; } void ValueType::addEntityValue(std::string name) { addEntityValue(name, ""); } void ValueType::addEntityValue(std::string name, std::string color) { if (name.empty()) THROWF(tracing_error, 0, "can't get a value with no name"); auto it = values_.find(name); if (it == values_.end()) { EntityValue* new_val = new EntityValue(name, color, this); values_.insert({name, new_val}); XBT_DEBUG("new value %s, child of %s", name.c_str(), getCname()); new_val->print(); } } EntityValue* ValueType::getEntityValue(std::string name) { auto ret = values_.find(name); if (ret == values_.end()) { THROWF(tracing_error, 2, "value with name (%s) not found in father type (%s)", name.c_str(), getCname()); } return ret->second; } ContainerType* Type::getOrCreateContainerType(std::string name) { auto cont = children_.find(name); return cont == children_.end() ? new ContainerType(name, this) : static_cast(cont->second); } EventType* Type::getOrCreateEventType(std::string name) { auto cont = children_.find(name); return cont == children_.end() ? new EventType(name, this) : static_cast(cont->second); } StateType* Type::getOrCreateStateType(std::string name) { auto cont = children_.find(name); return cont == children_.end() ? new StateType(name, this) : static_cast(cont->second); } VariableType* Type::getOrCreateVariableType(std::string name, std::string color) { auto cont = children_.find(name); std::string mycolor = color.empty() ? "1 1 1" : color; return cont == children_.end() ? new VariableType(name, mycolor, this) : static_cast(cont->second); } LinkType* Type::getOrCreateLinkType(std::string name, Type* source, Type* dest) { std::string alias = name + "-" + std::to_string(source->id_) + "-" + std::to_string(dest->id_); auto it = children_.find(alias); if (it == children_.end()) { LinkType* ret = new LinkType(name, alias, this); XBT_DEBUG("LinkType %s(%lld), child of %s(%lld) %s(%lld)->%s(%lld)", ret->getCname(), ret->getId(), getCname(), getId(), source->getCname(), source->getId(), dest->getCname(), dest->getId()); ret->logDefinition(source, dest); return ret; } else return static_cast(it->second); } } } SimGrid-3.18/src/instr/instr_paje_containers.hpp0000644000175000017500000000320713217757321022343 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PAJE_CONTAINERS_HPP #define INSTR_PAJE_CONTAINERS_HPP #include "src/instr/instr_private.hpp" #include namespace simgrid { namespace instr { class Type; class LinkType; class StateType; class VariableType; class Container { long long int id_; std::string name_; /* Unique name of this container */ public: Container(std::string name, std::string type_name, Container* father); virtual ~Container(); Type* type_; /* Type of this container */ Container* father_; std::map children_; sg_netpoint_t netpoint_ = nullptr; static Container* byNameOrNull(std::string name); static Container* byName(std::string name); std::string getName() { return name_; } const char* getCname() { return name_.c_str(); } long long int getId() { return id_; } void removeFromParent(); void logCreation(); void logDestruction(); StateType* getState(std::string name); LinkType* getLink(std::string name); VariableType* getVariable(std::string name); static Container* getRoot(); }; class NetZoneContainer : public Container { public: NetZoneContainer(std::string name, unsigned int level, NetZoneContainer* father); }; class RouterContainer : public Container { public: RouterContainer(std::string name, Container* father); }; class HostContainer : public Container { public: HostContainer(simgrid::s4u::Host& host, NetZoneContainer* father); }; } } #endif SimGrid-3.18/src/instr/instr_paje_events.hpp0000644000175000017500000000500013217757321021473 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PAJE_EVENTS_HPP #define INSTR_PAJE_EVENTS_HPP #include "src/instr/instr_private.hpp" #include namespace simgrid { namespace instr { class EntityValue; class TIData; enum e_event_type : unsigned int { PAJE_DefineContainerType, PAJE_DefineVariableType, PAJE_DefineStateType, PAJE_DefineEventType, PAJE_DefineLinkType, PAJE_DefineEntityValue, PAJE_CreateContainer, PAJE_DestroyContainer, PAJE_SetVariable, PAJE_AddVariable, PAJE_SubVariable, PAJE_SetState, PAJE_PushState, PAJE_PopState, PAJE_ResetState, PAJE_StartLink, PAJE_EndLink, PAJE_NewEvent }; class PajeEvent { Container* container_; Type* type_; protected: Type* getType() { return type_; } Container* getContainer() { return container_; } public: double timestamp_; e_event_type eventType_; PajeEvent(Container* container, Type* type, double timestamp, e_event_type eventType) : container_(container), type_(type), timestamp_(timestamp), eventType_(eventType){}; virtual ~PajeEvent() = default; virtual void print() = 0; void insertIntoBuffer(); }; class VariableEvent : public PajeEvent { double value; public: VariableEvent(double timestamp, Container* container, Type* type, e_event_type event_type, double value); void print() override; }; class StateEvent : public PajeEvent { EntityValue* value; std::string filename; int linenumber = 0; TIData* extra_ = nullptr; public: StateEvent(Container* container, Type* type, e_event_type event_type, EntityValue* value); StateEvent(Container* container, Type* type, e_event_type event_type, EntityValue* value, TIData* extra); void print() override; }; class LinkEvent : public PajeEvent { Container* endpoint_; std::string value_; std::string key_; int size_ = -1; public: LinkEvent(Container* container, Type* type, e_event_type event_type, Container* sourceContainer, std::string value, std::string key); LinkEvent(Container* container, Type* type, e_event_type event_type, Container* sourceContainer, std::string value, std::string key, int size); void print() override; }; class NewEvent : public PajeEvent { EntityValue* val; public: NewEvent(double timestamp, Container* container, Type* type, EntityValue* val); void print() override; }; } } #endif SimGrid-3.18/src/instr/instr_paje_containers.cpp0000644000175000017500000001703313217757321022340 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/Engine.hpp" #include "simgrid/s4u/Host.hpp" #include "src/instr/instr_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_containers, instr, "Paje tracing event system (containers)"); extern FILE* tracing_file; extern std::map tracing_files; // TI specific double prefix = 0.0; // TI specific static container_t rootContainer = nullptr; /* the root container */ static std::map allContainers; /* all created containers indexed by name */ std::set trivaNodeTypes; /* all host types defined */ std::set trivaEdgeTypes; /* all link types defined */ long long int instr_new_paje_id () { static long long int type_id = 0; return type_id++; } namespace simgrid { namespace instr { container_t Container::getRoot() { return rootContainer; } NetZoneContainer::NetZoneContainer(std::string name, unsigned int level, NetZoneContainer* father) : Container::Container(name, "", father) { netpoint_ = simgrid::s4u::Engine::getInstance()->getNetpointByNameOrNull(name); xbt_assert(netpoint_, "Element '%s' not found", name.c_str()); if (father_) { type_ = father_->type_->getOrCreateContainerType(std::string("L") + std::to_string(level)); father_->children_.insert({getName(), this}); logCreation(); } else { type_ = new ContainerType("0"); rootContainer = this; } } RouterContainer::RouterContainer(std::string name, Container* father) : Container::Container(name, "ROUTER", father) { xbt_assert(father, "Only the Root container has no father"); netpoint_ = simgrid::s4u::Engine::getInstance()->getNetpointByNameOrNull(name); xbt_assert(netpoint_, "Element '%s' not found", name.c_str()); trivaNodeTypes.insert(type_->getName()); } HostContainer::HostContainer(simgrid::s4u::Host& host, NetZoneContainer* father) : Container::Container(host.getCname(), "HOST", father) { xbt_assert(father, "Only the Root container has no father"); netpoint_ = host.pimpl_netpoint; xbt_assert(netpoint_, "Element '%s' not found", host.getCname()); trivaNodeTypes.insert(type_->getName()); } Container::Container(std::string name, std::string type_name, Container* father) : name_(name), father_(father) { static long long int container_id = 0; id_ = container_id; // id (or alias) of the container container_id++; if (father_) { XBT_DEBUG("new container %s, child of %s", name.c_str(), father->name_.c_str()); if (not type_name.empty()) { type_ = father_->type_->getOrCreateContainerType(type_name); father_->children_.insert({name_, this}); logCreation(); } } //register all kinds by name if (not allContainers.emplace(name_, this).second) THROWF(tracing_error, 1, "container %s already present in allContainers data structure", name_.c_str()); XBT_DEBUG("Add container name '%s'", name_.c_str()); //register NODE types for triva configuration if (type_name == "LINK") trivaNodeTypes.insert(type_->getName()); } Container::~Container() { XBT_DEBUG("destroy container %s", name_.c_str()); // Begin with destroying my own children for (auto child : children_) delete child.second; // obligation to dump previous events because they might reference the container that is about to be destroyed TRACE_last_timestamp_to_dump = SIMIX_get_clock(); TRACE_paje_dump_buffer(true); // trace my destruction, but not if user requests so or if the container is root if (not TRACE_disable_destroy() && this != Container::getRoot()) logDestruction(); // remove me from the allContainers data structure allContainers.erase(name_); } Container* Container::byNameOrNull(std::string name) { auto cont = allContainers.find(name); return cont == allContainers.end() ? nullptr : cont->second; } Container* Container::byName(std::string name) { Container* ret = Container::byNameOrNull(name); if (ret == nullptr) THROWF(tracing_error, 1, "container with name %s not found", name.c_str()); return ret; } void Container::removeFromParent() { if (father_) { XBT_DEBUG("removeChildContainer (%s) FromContainer (%s) ", getCname(), father_->getCname()); father_->children_.erase(name_); } } void Container::logCreation() { double timestamp = SIMIX_get_clock(); std::stringstream stream; XBT_DEBUG("%s: event_type=%u, timestamp=%f", __FUNCTION__, PAJE_CreateContainer, timestamp); if (instr_fmt_type == instr_fmt_paje) { stream << std::fixed << std::setprecision(TRACE_precision()) << PAJE_CreateContainer << " "; /* prevent 0.0000 in the trace - this was the behavior before the transition to c++ */ if (timestamp < 1e-12) stream << 0; else stream << timestamp; stream << " " << id_ << " " << type_->getId() << " " << father_->id_ << " \"" << name_ << "\""; XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } else if (instr_fmt_type == instr_fmt_TI) { // if we are in the mode with only one file static FILE* ti_unique_file = nullptr; if (tracing_files.empty()) { // generate unique run id with time prefix = xbt_os_time(); } if (not xbt_cfg_get_boolean("tracing/smpi/format/ti-one-file") || ti_unique_file == nullptr) { std::string folder_name = TRACE_get_filename() + "_files"; std::string filename = folder_name + "/" + std::to_string(prefix) + "_" + name_ + ".txt"; #ifdef WIN32 _mkdir(folder_name.c_str()); #else mkdir(folder_name.c_str(), S_IRWXU | S_IRWXG | S_IRWXO); #endif ti_unique_file = fopen(filename.c_str(), "w"); xbt_assert(ti_unique_file, "Tracefile %s could not be opened for writing: %s", filename.c_str(), strerror(errno)); fprintf(tracing_file, "%s\n", filename.c_str()); } tracing_files.insert({this, ti_unique_file}); } else { THROW_IMPOSSIBLE; } } void Container::logDestruction() { std::stringstream stream; double timestamp = SIMIX_get_clock(); XBT_DEBUG("%s: event_type=%u, timestamp=%f", __FUNCTION__, PAJE_DestroyContainer, timestamp); if (instr_fmt_type == instr_fmt_paje) { stream << std::fixed << std::setprecision(TRACE_precision()) << PAJE_DestroyContainer << " "; /* prevent 0.0000 in the trace - this was the behavior before the transition to c++ */ if (timestamp < 1e-12) stream << 0 << " " << type_->getId() << " " << id_; else stream << timestamp << " " << type_->getId() << " " << id_; XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } else if (instr_fmt_type == instr_fmt_TI) { if (not xbt_cfg_get_boolean("tracing/smpi/format/ti-one-file") || tracing_files.size() == 1) { fclose(tracing_files.at(this)); } tracing_files.erase(this); } else { THROW_IMPOSSIBLE; } } StateType* Container::getState(std::string name) { StateType* ret = dynamic_cast(type_->byName(name)); ret->setCallingContainer(this); return ret; } LinkType* Container::getLink(std::string name) { LinkType* ret = dynamic_cast(type_->byName(name)); ret->setCallingContainer(this); return ret; } VariableType* Container::getVariable(std::string name) { VariableType* ret = dynamic_cast(type_->byName(name)); ret->setCallingContainer(this); return ret; } } } SimGrid-3.18/src/instr/instr_smpi.hpp0000644000175000017500000000352013217757321020145 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_SMPI_HPP #define INSTR_SMPI_HPP #include "smpi/smpi.h" #include "src/instr/instr_private.hpp" #include XBT_PRIVATE container_t smpi_container(int rank); extern "C" { XBT_PRIVATE void TRACE_internal_smpi_set_category(const char* category); XBT_PRIVATE const char* TRACE_internal_smpi_get_category(); XBT_PRIVATE void TRACE_smpi_computing_init(int rank); XBT_PRIVATE void TRACE_smpi_computing_out(int rank); XBT_PRIVATE void TRACE_smpi_computing_in(int rank, double amount); XBT_PRIVATE void TRACE_smpi_sleeping_init(int rank); XBT_PRIVATE void TRACE_smpi_sleeping_out(int rank); XBT_PRIVATE void TRACE_smpi_sleeping_in(int rank, double duration); XBT_PRIVATE void TRACE_smpi_testing_out(int rank); XBT_PRIVATE void TRACE_smpi_testing_in(int rank); XBT_PRIVATE void TRACE_smpi_alloc(); XBT_PRIVATE void TRACE_smpi_release(); XBT_PRIVATE void TRACE_smpi_comm_in(int rank, const char* operation, simgrid::instr::TIData* extra); XBT_PRIVATE void TRACE_smpi_comm_out(int rank); XBT_PRIVATE void TRACE_smpi_send(int rank, int src, int dst, int tag, int size); XBT_PRIVATE void TRACE_smpi_recv(int src, int dst, int tag); XBT_PRIVATE void TRACE_smpi_init(int rank); XBT_PRIVATE void TRACE_smpi_finalize(int rank); XBT_PRIVATE const char* encode_datatype(MPI_Datatype datatype); class smpi_trace_call_location_t { public: std::string filename; int linenumber; std::string previous_filename; int previous_linenumber; std::string get_composed_key() { return previous_filename + ':' + std::to_string(previous_linenumber) + ':' + filename + ':' + std::to_string(linenumber); } }; } #endif SimGrid-3.18/src/instr/instr_paje_events.cpp0000644000175000017500000001416613217757321021503 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.hpp" #include "src/instr/instr_smpi.hpp" #include "src/smpi/include/private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_paje_events, instr, "Paje tracing event system (events)"); extern FILE* tracing_file; std::map tracing_files; // TI specific namespace simgrid { namespace instr { NewEvent::NewEvent(double timestamp, container_t container, Type* type, EntityValue* val) : simgrid::instr::PajeEvent::PajeEvent(container, type, timestamp, PAJE_NewEvent), val(val) { XBT_DEBUG("%s: event_type=%u, timestamp=%f", __FUNCTION__, eventType_, timestamp_); insertIntoBuffer(); } LinkEvent::LinkEvent(container_t container, Type* type, e_event_type event_type, container_t endpoint, std::string value, std::string key) : LinkEvent(container, type, event_type, endpoint, value, key, -1) { } LinkEvent::LinkEvent(container_t container, Type* type, e_event_type event_type, container_t endpoint, std::string value, std::string key, int size) : PajeEvent(container, type, SIMIX_get_clock(), event_type) , endpoint_(endpoint) , value_(value) , key_(key) , size_(size) { XBT_DEBUG("%s: event_type=%u, timestamp=%f, value:%s", __FUNCTION__, eventType_, timestamp_, value_.c_str()); insertIntoBuffer(); } VariableEvent::VariableEvent(double timestamp, Container* container, Type* type, e_event_type event_type, double value) : PajeEvent::PajeEvent(container, type, timestamp, event_type), value(value) { XBT_DEBUG("%s: event_type=%u, timestamp=%f", __FUNCTION__, eventType_, timestamp_); insertIntoBuffer(); } StateEvent::StateEvent(Container* container, Type* type, e_event_type event_type, EntityValue* value) : StateEvent(container, type, event_type, value, nullptr) { } StateEvent::StateEvent(Container* container, Type* type, e_event_type event_type, EntityValue* value, TIData* extra) : PajeEvent::PajeEvent(container, type, SIMIX_get_clock(), event_type), value(value), extra_(extra) { #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { smpi_trace_call_location_t* loc = smpi_trace_get_call_location(); filename = loc->filename; linenumber = loc->linenumber; } #endif XBT_DEBUG("%s: event_type=%u, timestamp=%f", __FUNCTION__, eventType_, timestamp_); insertIntoBuffer(); }; void NewEvent::print() { std::stringstream stream; stream << std::fixed << std::setprecision(TRACE_precision()); XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, eventType_, TRACE_precision(), timestamp_); if (instr_fmt_type != instr_fmt_paje) return; if (timestamp_ < 1e-12) stream << eventType_ << " " << 0 << " "; else stream << eventType_ << " " << timestamp_ << " "; stream << getType()->getId() << " " << getContainer()->getId() << " " << val->getId(); XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } void LinkEvent::print() { std::stringstream stream; stream << std::fixed << std::setprecision(TRACE_precision()); XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, eventType_, TRACE_precision(), timestamp_); if (instr_fmt_type != instr_fmt_paje) return; if (timestamp_ < 1e-12) stream << eventType_ << " " << 0 << " " << getType()->getId() << " " << getContainer()->getId(); else stream << eventType_ << " " << timestamp_ << " " << getType()->getId() << " " << getContainer()->getId(); stream << " " << value_ << " " << endpoint_->getId() << " " << key_; if (TRACE_display_sizes()) { stream << " " << size_; } XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } void VariableEvent::print() { std::stringstream stream; stream << std::fixed << std::setprecision(TRACE_precision()); XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, eventType_, TRACE_precision(), timestamp_); if (instr_fmt_type != instr_fmt_paje) return; if (timestamp_ < 1e-12) stream << eventType_ << " " << 0 << " "; else stream << eventType_ << " " << timestamp_ << " "; stream << getType()->getId() << " " << getContainer()->getId() << " " << value; XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } void StateEvent::print() { std::stringstream stream; stream << std::fixed << std::setprecision(TRACE_precision()); XBT_DEBUG("%s: event_type=%u, timestamp=%.*f", __FUNCTION__, eventType_, TRACE_precision(), timestamp_); if (instr_fmt_type == instr_fmt_paje) { if (timestamp_ < 1e-12) stream << eventType_ << " " << 0 << " " << getType()->getId() << " " << getContainer()->getId(); else stream << eventType_ << " " << timestamp_ << " " << getType()->getId() << " " << getContainer()->getId(); if (value != nullptr) // PAJE_PopState Event does not need to have a value stream << " " << value->getId(); if (TRACE_display_sizes()) stream << " " << ((extra_ != nullptr) ? extra_->display_size() : 0); #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { stream << " \"" << filename << "\" " << linenumber; } #endif XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } else if (instr_fmt_type == instr_fmt_TI) { if (extra_ == nullptr) return; /* Unimplemented calls are: WAITANY, SENDRECV, SCAN, EXSCAN, SSEND, and ISSEND. */ // FIXME: dirty extract "rank-" from the name, as we want the bare process id here if (getContainer()->getName().find("rank-") != 0) stream << getContainer()->getName() << " " << extra_->print(); else stream << getContainer()->getName().erase(0, 5) << " " << extra_->print(); fprintf(tracing_files.at(getContainer()), "%s\n", stream.str().c_str()); } else { THROW_IMPOSSIBLE; } delete extra_; } } } SimGrid-3.18/src/instr/instr_config.cpp0000644000175000017500000004723113217757321020444 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "include/xbt/config.hpp" #include "src/instr/instr_private.hpp" #include "surf/surf.hpp" #include #include XBT_LOG_NEW_CATEGORY(instr, "Logging the behavior of the tracing system (used for Visualization/Analysis of simulations)"); XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_config, instr, "Configuration"); #define OPT_TRACING_BASIC "tracing/basic" #define OPT_TRACING_BUFFER "tracing/buffer" #define OPT_TRACING_CATEGORIZED "tracing/categorized" #define OPT_TRACING_COMMENT_FILE "tracing/comment-file" #define OPT_TRACING_COMMENT "tracing/comment" #define OPT_TRACING_DISABLE_DESTROY "tracing/disable-destroy" #define OPT_TRACING_DISABLE_LINK "tracing/disable-link" #define OPT_TRACING_DISABLE_POWER "tracing/disable-power" #define OPT_TRACING_DISPLAY_SIZES "tracing/smpi/display-sizes" #define OPT_TRACING_FILENAME "tracing/filename" #define OPT_TRACING_FORMAT_TI_ONEFILE "tracing/smpi/format/ti-one-file" #define OPT_TRACING_FORMAT "tracing/smpi/format" #define OPT_TRACING_MSG_PROCESS "tracing/msg/process" #define OPT_TRACING_MSG_VM "tracing/msg/vm" #define OPT_TRACING_ONELINK_ONLY "tracing/onelink-only" #define OPT_TRACING_PLATFORM "tracing/platform" #define OPT_TRACING_PRECISION "tracing/precision" #define OPT_TRACING_SMPI_COMPUTING "tracing/smpi/computing" #define OPT_TRACING_SMPI_GROUP "tracing/smpi/group" #define OPT_TRACING_SMPI_INTERNALS "tracing/smpi/internals" #define OPT_TRACING_SMPI_SLEEPING "tracing/smpi/sleeping" #define OPT_TRACING_SMPI "tracing/smpi" #define OPT_TRACING_TOPOLOGY "tracing/platform/topology" #define OPT_TRACING "tracing" #define OPT_TRACING_UNCATEGORIZED "tracing/uncategorized" static bool trace_enabled = false; static bool trace_platform; static bool trace_platform_topology; static bool trace_smpi_enabled; static bool trace_smpi_grouped; static bool trace_smpi_computing; static bool trace_smpi_sleeping; static bool trace_view_internals; static bool trace_categorized; static bool trace_uncategorized; static bool trace_msg_process_enabled; static bool trace_msg_vm_enabled; static bool trace_buffer; static bool trace_onelink_only; static bool trace_disable_destroy; static bool trace_basic; static bool trace_display_sizes = false; static bool trace_disable_link; static bool trace_disable_power; static int trace_precision; static bool trace_configured = false; static bool trace_active = false; instr_fmt_type_t instr_fmt_type = instr_fmt_paje; static void TRACE_getopts() { trace_enabled = xbt_cfg_get_boolean(OPT_TRACING); trace_platform = xbt_cfg_get_boolean(OPT_TRACING_PLATFORM); trace_platform_topology = xbt_cfg_get_boolean(OPT_TRACING_TOPOLOGY); trace_smpi_enabled = xbt_cfg_get_boolean(OPT_TRACING_SMPI); trace_smpi_grouped = xbt_cfg_get_boolean(OPT_TRACING_SMPI_GROUP); trace_smpi_computing = xbt_cfg_get_boolean(OPT_TRACING_SMPI_COMPUTING); trace_smpi_sleeping = xbt_cfg_get_boolean(OPT_TRACING_SMPI_SLEEPING); trace_view_internals = xbt_cfg_get_boolean(OPT_TRACING_SMPI_INTERNALS); trace_categorized = xbt_cfg_get_boolean(OPT_TRACING_CATEGORIZED); trace_uncategorized = xbt_cfg_get_boolean(OPT_TRACING_UNCATEGORIZED); trace_msg_process_enabled = xbt_cfg_get_boolean(OPT_TRACING_MSG_PROCESS); trace_msg_vm_enabled = xbt_cfg_get_boolean(OPT_TRACING_MSG_VM); trace_buffer = xbt_cfg_get_boolean(OPT_TRACING_BUFFER); trace_onelink_only = xbt_cfg_get_boolean(OPT_TRACING_ONELINK_ONLY); trace_disable_destroy = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_DESTROY); trace_basic = xbt_cfg_get_boolean(OPT_TRACING_BASIC); trace_display_sizes = xbt_cfg_get_boolean(OPT_TRACING_DISPLAY_SIZES); trace_disable_link = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_LINK); trace_disable_power = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_POWER); trace_precision = xbt_cfg_get_int(OPT_TRACING_PRECISION); } int TRACE_start() { if (TRACE_is_configured()) TRACE_getopts(); // tracing system must be: // - enabled (with --cfg=tracing:yes) // - already configured (TRACE_global_init already called) if (TRACE_is_enabled()) { XBT_DEBUG("Tracing starts"); /* init the tracing module to generate the right output */ /* open the trace file(s) */ std::string format = xbt_cfg_get_string(OPT_TRACING_FORMAT); XBT_DEBUG("Tracing format %s\n", format.c_str()); if (format == "Paje") { TRACE_paje_start(); } else if (format == "TI") { instr_fmt_type = instr_fmt_TI; TRACE_TI_start(); }else{ xbt_die("Unknown trace format :%s ", format.c_str()); } /* activate trace */ if (trace_active) { THROWF(tracing_error, 0, "Tracing is already active"); } trace_active = true; XBT_DEBUG("Tracing is on"); } return 0; } int TRACE_end() { int retval; if (not trace_active) { retval = 1; } else { retval = 0; /* dump trace buffer */ TRACE_last_timestamp_to_dump = surf_get_clock(); TRACE_paje_dump_buffer(true); simgrid::instr::Type* root_type = simgrid::instr::Container::getRoot()->type_; /* destroy all data structures of tracing (and free) */ delete simgrid::instr::Container::getRoot(); delete root_type; /* close the trace files */ std::string format = xbt_cfg_get_string(OPT_TRACING_FORMAT); XBT_DEBUG("Tracing format %s\n", format.c_str()); if (format == "Paje") { TRACE_paje_end(); } else if (format == "TI") { TRACE_TI_end(); }else{ xbt_die("Unknown trace format :%s ", format.c_str()); } /* de-activate trace */ trace_active = false; XBT_DEBUG("Tracing is off"); XBT_DEBUG("Tracing system is shutdown"); } return retval; } bool TRACE_needs_platform () { return TRACE_msg_process_is_enabled() || TRACE_msg_vm_is_enabled() || TRACE_categorized() || TRACE_uncategorized() || TRACE_platform () || (TRACE_smpi_is_enabled() && TRACE_smpi_is_grouped()); } bool TRACE_is_enabled() { return trace_enabled; } bool TRACE_platform() { return trace_platform; } bool TRACE_platform_topology() { return trace_platform_topology; } bool TRACE_is_configured() { return trace_configured; } bool TRACE_smpi_is_enabled() { return (trace_smpi_enabled || TRACE_smpi_is_grouped()) && TRACE_is_enabled(); } bool TRACE_smpi_is_grouped() { return trace_smpi_grouped; } bool TRACE_smpi_is_computing() { return trace_smpi_computing; } bool TRACE_smpi_is_sleeping() { return trace_smpi_sleeping; } bool TRACE_smpi_view_internals() { return trace_view_internals; } bool TRACE_categorized () { return trace_categorized; } bool TRACE_uncategorized () { return trace_uncategorized; } bool TRACE_msg_process_is_enabled() { return trace_msg_process_enabled && TRACE_is_enabled(); } bool TRACE_msg_vm_is_enabled() { return trace_msg_vm_enabled && TRACE_is_enabled(); } bool TRACE_disable_link() { return trace_disable_link && TRACE_is_enabled(); } bool TRACE_disable_speed() { return trace_disable_power && TRACE_is_enabled(); } bool TRACE_buffer () { return trace_buffer && TRACE_is_enabled(); } bool TRACE_onelink_only () { return trace_onelink_only && TRACE_is_enabled(); } bool TRACE_disable_destroy () { return trace_disable_destroy && TRACE_is_enabled(); } bool TRACE_basic () { return trace_basic && TRACE_is_enabled(); } bool TRACE_display_sizes () { return trace_display_sizes && trace_smpi_enabled && TRACE_is_enabled(); } std::string TRACE_get_comment() { return xbt_cfg_get_string(OPT_TRACING_COMMENT); } std::string TRACE_get_comment_file() { return xbt_cfg_get_string(OPT_TRACING_COMMENT_FILE); } int TRACE_precision () { return xbt_cfg_get_int(OPT_TRACING_PRECISION); } std::string TRACE_get_filename() { return xbt_cfg_get_string(OPT_TRACING_FILENAME); } void TRACE_global_init() { static bool is_initialised = false; if (is_initialised) return; is_initialised = true; /* name of the tracefile */ xbt_cfg_register_string (OPT_TRACING_FILENAME, "simgrid.trace", nullptr, "Trace file created by the instrumented SimGrid."); xbt_cfg_register_boolean(OPT_TRACING, "no", nullptr, "Enable Tracing."); xbt_cfg_register_boolean(OPT_TRACING_PLATFORM, "no", nullptr, "Register the platform in the trace as a hierarchy."); xbt_cfg_register_boolean(OPT_TRACING_TOPOLOGY, "yes", nullptr, "Register the platform topology in the trace as a graph."); xbt_cfg_register_boolean(OPT_TRACING_SMPI, "no", nullptr, "Tracing of the SMPI interface."); xbt_cfg_register_boolean(OPT_TRACING_SMPI_GROUP,"no", nullptr, "Group MPI processes by host."); xbt_cfg_register_boolean(OPT_TRACING_SMPI_COMPUTING, "no", nullptr, "Generate states for timing out of SMPI parts of the application"); xbt_cfg_register_boolean(OPT_TRACING_SMPI_SLEEPING, "no", nullptr, "Generate states for timing out of SMPI parts of the application"); xbt_cfg_register_boolean(OPT_TRACING_SMPI_INTERNALS, "no", nullptr, "View internal messages sent by Collective communications in SMPI"); xbt_cfg_register_boolean(OPT_TRACING_CATEGORIZED, "no", nullptr, "Tracing categorized resource utilization of hosts and links."); xbt_cfg_register_boolean(OPT_TRACING_UNCATEGORIZED, "no", nullptr, "Tracing uncategorized resource utilization of hosts and links."); xbt_cfg_register_boolean(OPT_TRACING_MSG_PROCESS, "no", nullptr, "Tracing of MSG process behavior."); xbt_cfg_register_boolean(OPT_TRACING_MSG_VM, "no", nullptr, "Tracing of MSG process behavior."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_LINK, "no", nullptr, "Do not trace link bandwidth and latency."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_POWER, "no", nullptr, "Do not trace host power."); xbt_cfg_register_boolean(OPT_TRACING_BUFFER, "yes", nullptr, "Buffer trace events to put them in temporal order."); xbt_cfg_register_boolean(OPT_TRACING_ONELINK_ONLY, "no", nullptr, "Use only routes with one link to trace platform."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_DESTROY, "no", nullptr, "Disable platform containers destruction."); xbt_cfg_register_boolean(OPT_TRACING_BASIC, "no", nullptr, "Avoid extended events (impoverished trace file)."); xbt_cfg_register_boolean(OPT_TRACING_DISPLAY_SIZES, "no", nullptr, "(smpi only) Extended events with message size information"); xbt_cfg_register_string(OPT_TRACING_FORMAT, "Paje", nullptr, "(smpi only) Switch the output format of Tracing"); xbt_cfg_register_boolean(OPT_TRACING_FORMAT_TI_ONEFILE, "no", nullptr, "(smpi only) For replay format only : output to one file only"); xbt_cfg_register_string(OPT_TRACING_COMMENT, "", nullptr, "Comment to be added on the top of the trace file."); xbt_cfg_register_string(OPT_TRACING_COMMENT_FILE, "", nullptr, "The contents of the file are added to the top of the trace file as comment."); xbt_cfg_register_int(OPT_TRACING_PRECISION, 6, nullptr, "Numerical precision used when timestamping events " "(expressed in number of digits after decimal point)"); xbt_cfg_register_alias(OPT_TRACING_COMMENT_FILE,"tracing/comment_file"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_DESTROY, "tracing/disable_destroy"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_LINK, "tracing/disable_link"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_POWER, "tracing/disable_power"); xbt_cfg_register_alias(OPT_TRACING_DISPLAY_SIZES, "tracing/smpi/display_sizes"); xbt_cfg_register_alias(OPT_TRACING_FORMAT_TI_ONEFILE, "tracing/smpi/format/ti_one_file"); xbt_cfg_register_alias(OPT_TRACING_ONELINK_ONLY, "tracing/onelink_only"); /* instrumentation can be considered configured now */ trace_configured = true; } static void print_line (const char *option, const char *desc, const char *longdesc, int detailed) { std::string str = std::string("--cfg=") + option + " "; int len = str.size(); printf("%s%*.*s %s\n", str.c_str(), 30 - len, 30 - len, "", desc); if (longdesc != nullptr && detailed){ printf ("%s\n\n", longdesc); } } void TRACE_help (int detailed) { printf("Description of the tracing options accepted by this simulator:\n\n"); print_line (OPT_TRACING, "Enable the tracing system", " It activates the tracing system and register the simulation platform\n" " in the trace file. You have to enable this option to others take effect.", detailed); print_line (OPT_TRACING_CATEGORIZED, "Trace categorized resource utilization", " It activates the categorized resource utilization tracing. It should\n" " be enabled if tracing categories are used by this simulator.", detailed); print_line (OPT_TRACING_UNCATEGORIZED, "Trace uncategorized resource utilization", " It activates the uncategorized resource utilization tracing. Use it if\n" " this simulator do not use tracing categories and resource use have to be\n" " traced.", detailed); print_line(OPT_TRACING_FILENAME, "Filename to register traces", " A file with this name will be created to register the simulation. The file\n" " is in the Paje format and can be analyzed using Paje, and PajeNG visualization\n" " tools. More information can be found in these webpages:\n" " http://github.com/schnorr/pajeng/\n" " http://paje.sourceforge.net/", detailed); print_line (OPT_TRACING_SMPI, "Trace the MPI Interface (SMPI)", " This option only has effect if this simulator is SMPI-based. Traces the MPI\n" " interface and generates a trace that can be analyzed using Gantt-like\n" " visualizations. Every MPI function (implemented by SMPI) is transformed in a\n" " state, and point-to-point communications can be analyzed with arrows.", detailed); print_line (OPT_TRACING_SMPI_GROUP, "Group MPI processes by host (SMPI)", " This option only has effect if this simulator is SMPI-based. The processes\n" " are grouped by the hosts where they were executed.", detailed); print_line (OPT_TRACING_SMPI_COMPUTING, "Generates a \" Computing \" State", " This option aims at tracing computations in the application, outside SMPI\n" " to allow further study of simulated or real computation time", detailed); print_line (OPT_TRACING_SMPI_SLEEPING, "Generates a \" Sleeping \" State", " This option aims at tracing sleeps in the application, outside SMPI\n" " to allow further study of simulated or real sleep time", detailed); print_line (OPT_TRACING_SMPI_INTERNALS, "Generates tracing events corresponding", " to point-to-point messages sent by collective communications", detailed); print_line (OPT_TRACING_MSG_PROCESS, "Trace processes behavior (MSG)", " This option only has effect if this simulator is MSG-based. It traces the\n" " behavior of all categorized MSG processes, grouping them by hosts. This option\n" " can be used to track process location if this simulator has process migration.", detailed); print_line (OPT_TRACING_BUFFER, "Buffer events to put them in temporal order", " This option put some events in a time-ordered buffer using the insertion\n" " sort algorithm. The process of acquiring and releasing locks to access this\n" " buffer and the cost of the sorting algorithm make this process slow. The\n" " simulator performance can be severely impacted if this option is activated,\n" " but you are sure to get a trace file with events sorted.", detailed); print_line (OPT_TRACING_ONELINK_ONLY, "Consider only one link routes to trace platform", " This option changes the way SimGrid register its platform on the trace file.\n" " Normally, the tracing considers all routes (no matter their size) on the\n" " platform file to re-create the resource topology. If this option is activated,\n" " only the routes with one link are used to register the topology within an AS.\n" " Routes among AS continue to be traced as usual.", detailed); print_line (OPT_TRACING_DISABLE_DESTROY, "Disable platform containers destruction", " Disable the destruction of containers at the end of simulation. This can be\n" " used with simulators that have a different notion of time (different from\n" " the simulated time).", detailed); print_line (OPT_TRACING_BASIC, "Avoid extended events (impoverished trace file).", " Some visualization tools are not able to parse correctly the Paje file format.\n" " Use this option if you are using one of these tools to visualize the simulation\n" " trace. Keep in mind that the trace might be incomplete, without all the\n" " information that would be registered otherwise.", detailed); print_line (OPT_TRACING_DISPLAY_SIZES, "Only works for SMPI now. Add message size information", " Message size (in bytes) is added to links, and to states. For collectives,\n" " the displayed value is the more relevant to the collective (total sent by\n" " the process, usually)", detailed); print_line (OPT_TRACING_FORMAT, "Only works for SMPI now. Switch output format", " Default format is Paje. Time independent traces are also supported,\n" " to output traces that can later be used by the trace replay tool", detailed); print_line (OPT_TRACING_FORMAT_TI_ONEFILE, "Only works for SMPI now, and TI output format", " By default, each process outputs to a separate file, inside a filename_files folder\n" " By setting this option to yes, all processes will output to only one file\n" " This is meant to avoid opening thousands of files with large simulations", detailed); print_line (OPT_TRACING_COMMENT, "Comment to be added on the top of the trace file.", " Use this to add a comment line to the top of the trace file.", detailed); print_line (OPT_TRACING_COMMENT_FILE, "File contents added to trace file as comment.", " Use this to add the contents of a file to the top of the trace file as comment.", detailed); print_line (OPT_TRACING_TOPOLOGY, "Register the platform topology as a graph", " This option (enabled by default) can be used to disable the tracing of\n" " the platform topology in the trace file. Sometimes, such task is really\n" " time consuming, since it must get the route from each host to other hosts\n" " within the same Autonomous System (AS).", detailed); } static void output_types (const char *name, xbt_dynar_t types, FILE *file) { unsigned int i; fprintf (file, " %s = (", name); for (i = xbt_dynar_length(types); i > 0; i--) { char *type = *(static_cast(xbt_dynar_get_ptr(types, i - 1))); fprintf (file, "\"%s\"", type); if (i - 1 > 0){ fprintf (file, ","); }else{ fprintf (file, ");\n"); } } xbt_dynar_free (&types); } static int previous_trace_state = -1; void instr_pause_tracing () { previous_trace_state = trace_enabled; if (not TRACE_is_enabled()) { XBT_DEBUG ("Tracing is already paused, therefore do nothing."); }else{ XBT_DEBUG ("Tracing is being paused."); } trace_enabled = false; XBT_DEBUG ("Tracing is paused."); } void instr_resume_tracing () { if (TRACE_is_enabled()){ XBT_DEBUG ("Tracing is already running while trying to resume, therefore do nothing."); }else{ XBT_DEBUG ("Tracing is being resumed."); } if (previous_trace_state != -1){ trace_enabled = previous_trace_state; }else{ trace_enabled = true; } XBT_DEBUG ("Tracing is resumed."); previous_trace_state = -1; } SimGrid-3.18/src/instr/instr_interface.cpp0000644000175000017500000012345013217757321021135 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid_config.h" #include "src/instr/instr_private.hpp" #include "src/kernel/routing/NetPoint.hpp" #include "src/surf/network_interface.hpp" #include "src/surf/surf_private.hpp" #include "surf/surf.hpp" #include enum InstrUserVariable { INSTR_US_DECLARE, INSTR_US_SET, INSTR_US_ADD, INSTR_US_SUB }; XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_api, instr, "API"); std::set created_categories; std::set declared_marks; std::set user_host_variables; std::set user_vm_variables; std::set user_link_variables; extern std::set trivaNodeTypes; extern std::set trivaEdgeTypes; static xbt_dynar_t instr_set_to_dynar(std::set* filter) { if (not TRACE_is_enabled() || not TRACE_needs_platform()) return nullptr; xbt_dynar_t ret = xbt_dynar_new (sizeof(char*), &xbt_free_ref); for (auto const& name : *filter) xbt_dynar_push_as(ret, char*, xbt_strdup(name.c_str())); return ret; } /** \ingroup TRACE_category * \brief Declare a new category with a random color. * * This function should be used to define a user category. The category can be used to differentiate the tasks that * are created during the simulation (for example, tasks from server1, server2, or request tasks, computation tasks, * communication tasks). All resource utilization (host power and link bandwidth) will be classified according to the * task category. Tasks that do not belong to a category are not traced. The color for the category that is being * declared is random. This function has no effect if a category with the same name has been already declared. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \param category The name of the new tracing category to be created. * * \see TRACE_category_with_color, MSG_task_set_category, SD_task_set_category */ void TRACE_category(const char *category) { TRACE_category_with_color (category, nullptr); } /** \ingroup TRACE_category * \brief Declare a new category with a color. * * Same as #TRACE_category, but let user specify a color encoded as a RGB-like string with three floats from 0 to 1. * So, to specify a red color, pass "1 0 0" as color parameter. A light-gray color can be specified using "0.7 0.7 0.7" * as color. This function has no effect if a category with the same name has been already declared. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \param category The name of the new tracing category to be created. * \param color The color of the category (see \ref outcomes_vizu to * know how to correctly specify the color) * * \see MSG_task_set_category, SD_task_set_category */ void TRACE_category_with_color (const char *category, const char *color) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with categories */ if (not TRACE_is_enabled() || not TRACE_needs_platform()) return; if (not(TRACE_categorized() && category != nullptr)) return; //check if category is already created if (created_categories.find(category) != created_categories.end()) return; else created_categories.insert(category); //define final_color std::string final_color; if (not color) { //generate a random color double red = drand48(); double green = drand48(); double blue = drand48(); final_color = std::to_string(red) + " " + std::to_string(green) + " " + std::to_string(blue); }else{ final_color = std::string(color); } XBT_DEBUG("CAT,declare %s, \"%s\" \"%s\"", category, color, final_color.c_str()); //define the type of this category on top of hosts and links instr_new_variable_type (category, final_color); } /** \ingroup TRACE_category * \brief Get declared categories * * This function should be used to get categories that were already declared with #TRACE_category or with * #TRACE_category_with_color. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \return A dynar with the declared categories, must be freed with xbt_dynar_free. * * \see MSG_task_set_category, SD_task_set_category */ xbt_dynar_t TRACE_get_categories () { if (not TRACE_is_enabled() || not TRACE_categorized()) return nullptr; return instr_set_to_dynar(&created_categories); } /** \ingroup TRACE_mark * \brief Declare a new type for tracing mark. * * This function declares a new Paje event type in the trace file that can be used by simulators to declare * application-level marks. This function is independent of which API is used in SimGrid. * * \param mark_type The name of the new type. * * \see TRACE_mark */ void TRACE_declare_mark(const char *mark_type) { /* safe switchs. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (not TRACE_is_enabled() || not TRACE_needs_platform()) return; if (not mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); //check if mark_type is already declared if (declared_marks.find(mark_type) != declared_marks.end()) { THROWF (tracing_error, 1, "mark_type with name (%s) is already declared", mark_type); } XBT_DEBUG("MARK,declare %s", mark_type); simgrid::instr::Container::getRoot()->type_->getOrCreateEventType(mark_type); declared_marks.insert(mark_type); } /** \ingroup TRACE_mark * \brief Declare a new colored value for a previously declared mark type. * * This function declares a new colored value for a Paje event type in the trace file that can be used by simulators to * declare application-level marks. This function is independent of which API is used in SimGrid. The color needs to be * a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. If a nullptr color is provided, the color used will * be white ("1 1 1"). * * \param mark_type The name of the new type. * \param mark_value The name of the new value for this type. * \param mark_color The color of the new value for this type. * * \see TRACE_mark */ void TRACE_declare_mark_value_with_color (const char *mark_type, const char *mark_value, const char *mark_color) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (not TRACE_is_enabled() || not TRACE_needs_platform()) return; if (not mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); if (not mark_value) THROWF (tracing_error, 1, "mark_value is nullptr"); simgrid::instr::EventType* type = static_cast(simgrid::instr::Container::getRoot()->type_->byName(mark_type)); if (not type) { THROWF (tracing_error, 1, "mark_type with name (%s) is not declared", mark_type); } else { if (not mark_color) mark_color = "1.0 1.0 1.0" /*white*/; XBT_DEBUG("MARK,declare_value %s %s %s", mark_type, mark_value, mark_color); type->addEntityValue(mark_value, mark_color); } } /** \ingroup TRACE_mark * \brief Declare a new value for a previously declared mark type. * * This function declares a new value for a Paje event type in the trace file that can be used by simulators to declare * application-level marks. This function is independent of which API is used in SimGrid. Calling this function is the * same as calling \ref TRACE_declare_mark_value_with_color with a nullptr color. * * \param mark_type The name of the new type. * \param mark_value The name of the new value for this type. * * \see TRACE_mark */ void TRACE_declare_mark_value (const char *mark_type, const char *mark_value) { TRACE_declare_mark_value_with_color (mark_type, mark_value, nullptr); } /** * \ingroup TRACE_mark * \brief Create a new instance of a tracing mark type. * * This function creates a mark in the trace file. The first parameter had to be previously declared using * #TRACE_declare_mark, the second is the identifier for this mark instance. We recommend that the mark_value is a * unique value for the whole simulation. Nevertheless, this is not a strong requirement: the trace will be valid even * if there are multiple mark identifiers for the same trace. * * \param mark_type The name of the type for which the new instance will belong. * \param mark_value The name of the new instance mark. * * \see TRACE_declare_mark */ void TRACE_mark(const char *mark_type, const char *mark_value) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (not TRACE_is_enabled() || not TRACE_needs_platform()) return; if (not mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); if (not mark_value) THROWF (tracing_error, 1, "mark_value is nullptr"); //check if mark_type is already declared simgrid::instr::EventType* type = static_cast(simgrid::instr::Container::getRoot()->type_->byName(mark_type)); if (not type) { THROWF (tracing_error, 1, "mark_type with name (%s) is not declared", mark_type); } else { XBT_DEBUG("MARK %s %s", mark_type, mark_value); new simgrid::instr::NewEvent(MSG_get_clock(), simgrid::instr::Container::getRoot(), type, type->getEntityValue(mark_value)); } } /** \ingroup TRACE_mark * \brief Get declared marks * * This function should be used to get marks that were already declared with #TRACE_declare_mark. * * \return A dynar with the declared marks, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_marks () { if (not TRACE_is_enabled()) return nullptr; return instr_set_to_dynar(&declared_marks); } static void instr_user_variable(double time, const char* resource, const char* variable_name, const char* father_type, double value, InstrUserVariable what, const char* color, std::set* filter) { /* safe switches. tracing has to be activated and if platform is not traced, we don't allow user variables */ if (not TRACE_is_enabled() || not TRACE_needs_platform()) return; //check if variable is already declared auto created = filter->find(variable_name); if (what == INSTR_US_DECLARE){ if (created == filter->end()) { // not declared yet filter->insert(variable_name); instr_new_user_variable_type(father_type, variable_name, color == nullptr ? "" : color); } }else{ if (created != filter->end()) { // declared, let's work simgrid::instr::VariableType* variable = simgrid::instr::Container::byName(resource)->getVariable(variable_name); switch (what){ case INSTR_US_SET: variable->setEvent(time, value); break; case INSTR_US_ADD: variable->addEvent(time, value); break; case INSTR_US_SUB: variable->subEvent(time, value); break; default: THROW_IMPOSSIBLE; break; } } } } static void instr_user_srcdst_variable(double time, const char *src, const char *dst, const char *variable, const char *father_type, double value, InstrUserVariable what) { sg_netpoint_t src_elm = sg_netpoint_by_name_or_null(src); if (not src_elm) xbt_die("Element '%s' not found!",src); sg_netpoint_t dst_elm = sg_netpoint_by_name_or_null(dst); if (not dst_elm) xbt_die("Element '%s' not found!",dst); std::vector route; simgrid::kernel::routing::NetZoneImpl::getGlobalRoute(src_elm, dst_elm, route, nullptr); for (auto const& link : route) instr_user_variable(time, link->getCname(), variable, father_type, value, what, nullptr, &user_link_variables); } /** \ingroup TRACE_API * \brief Creates a file with the topology of the platform file used for the simulator. * * The graph topology will have the following properties: all hosts, links and routers of the platform file are mapped * to graph nodes; routes are mapped to edges. * The platform's AS are not represented in the output. * * \param filename The name of the file that will hold the graph. * * \return 1 of successful, 0 otherwise. */ int TRACE_platform_graph_export_graphviz (const char *filename) { /* returns 1 if successful, 0 otherwise */ if (not TRACE_is_enabled()) return 0; xbt_graph_t g = instr_routing_platform_graph(); if (g == nullptr) return 0; instr_routing_platform_graph_export_graphviz (g, filename); xbt_graph_free_graph(g, xbt_free_f, xbt_free_f, nullptr); return 1; } /* * Derived functions that use instr_user_variable and TRACE_user_srcdst_variable. They were previously defined as * pre-processors directives, but were transformed into functions so the user can track them using gdb. */ /* for VM variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to VMs. * * Declare a user variable that will be associated to VMs. A user vm variable can be used to trace user variables * such as the number of tasks in a VM, the number of clients in an application (for VMs), and so on. The color * associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_vm_variable_declare_with_color */ void TRACE_vm_variable_declare (const char *variable) { instr_user_variable(0, nullptr, variable, "MSG_VM", 0, INSTR_US_DECLARE, nullptr, &user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to VMs with a color. * * Same as #TRACE_vm_variable_declare, but associated a color to the newly created user host variable. The color needs * to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_vm_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable(0, nullptr, variable, "MSG_VM", 0, INSTR_US_DECLARE, color, &user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host. * * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_add, TRACE_vm_variable_sub */ void TRACE_vm_variable_set (const char *vm, const char *variable, double value) { TRACE_vm_variable_set_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a VM. * * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set, TRACE_vm_variable_sub */ void TRACE_vm_variable_add (const char *vm, const char *variable, double value) { TRACE_vm_variable_add_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a VM. * * \param vm The name of the vm to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set, TRACE_vm_variable_add */ void TRACE_vm_variable_sub (const char *vm, const char *variable, double value) { TRACE_vm_variable_sub_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_add_with_time, TRACE_vm_variable_sub_with_time */ void TRACE_vm_variable_set_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_SET, nullptr, &user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set_with_time, TRACE_vm_variable_sub_with_time */ void TRACE_vm_variable_add_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_ADD, nullptr, &user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set_with_time, TRACE_vm_variable_add_with_time */ void TRACE_vm_variable_sub_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_SUB, nullptr, &user_vm_variables); } /* for host variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to hosts. * * Declare a user variable that will be associated to hosts. * A user host variable can be used to trace user variables such as the number of tasks in a server, the number of * clients in an application (for hosts), and so on. The color associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_host_variable_declare_with_color */ void TRACE_host_variable_declare (const char *variable) { instr_user_variable(0, nullptr, variable, "HOST", 0, INSTR_US_DECLARE, nullptr, &user_host_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to hosts with a color. * * Same as #TRACE_host_variable_declare, but associated a color to the newly created user host variable. The color * needs to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_host_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable(0, nullptr, variable, "HOST", 0, INSTR_US_DECLARE, color, &user_host_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_add, TRACE_host_variable_sub */ void TRACE_host_variable_set (const char *host, const char *variable, double value) { TRACE_host_variable_set_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set, TRACE_host_variable_sub */ void TRACE_host_variable_add (const char *host, const char *variable, double value) { TRACE_host_variable_add_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set, TRACE_host_variable_add */ void TRACE_host_variable_sub (const char *host, const char *variable, double value) { TRACE_host_variable_sub_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_add_with_time, TRACE_host_variable_sub_with_time */ void TRACE_host_variable_set_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_SET, nullptr, &user_host_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set_with_time, TRACE_host_variable_sub_with_time */ void TRACE_host_variable_add_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_ADD, nullptr, &user_host_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set_with_time, TRACE_host_variable_add_with_time */ void TRACE_host_variable_sub_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_SUB, nullptr, &user_host_variables); } /** \ingroup TRACE_user_variables * \brief Get declared user host variables * * This function should be used to get host variables that were already declared with #TRACE_host_variable_declare or * with #TRACE_host_variable_declare_with_color. * * \return A dynar with the declared host variables, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_host_variables () { return instr_set_to_dynar(&user_host_variables); } /* for link variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to links. * * Declare a user variable that will be associated to links. * A user link variable can be used, for example, to trace user variables such as the number of messages being * transferred through network links. The color associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_link_variable_declare_with_color */ void TRACE_link_variable_declare (const char *variable) { instr_user_variable(0, nullptr, variable, "LINK", 0, INSTR_US_DECLARE, nullptr, &user_link_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to links with a color. * * Same as #TRACE_link_variable_declare, but associated a color to the newly created user link variable. The color * needs to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_link_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable(0, nullptr, variable, "LINK", 0, INSTR_US_DECLARE, color, &user_link_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_add, TRACE_link_variable_sub */ void TRACE_link_variable_set (const char *link, const char *variable, double value) { TRACE_link_variable_set_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set, TRACE_link_variable_sub */ void TRACE_link_variable_add (const char *link, const char *variable, double value) { TRACE_link_variable_add_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set, TRACE_link_variable_add */ void TRACE_link_variable_sub (const char *link, const char *variable, double value) { TRACE_link_variable_sub_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_add_with_time, TRACE_link_variable_sub_with_time */ void TRACE_link_variable_set_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable(time, link, variable, "LINK", value, INSTR_US_SET, nullptr, &user_link_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set_with_time, TRACE_link_variable_sub_with_time */ void TRACE_link_variable_add_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable(time, link, variable, "LINK", value, INSTR_US_ADD, nullptr, &user_link_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set_with_time, TRACE_link_variable_add_with_time */ void TRACE_link_variable_sub_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable(time, link, variable, "LINK", value, INSTR_US_SUB, nullptr, &user_link_variables); } /* for link variables, but with src and dst used for get_route */ /** \ingroup TRACE_user_variables * \brief Set the value of the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_set, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the variable * set to the provided value. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_add, TRACE_link_srcdst_variable_sub */ void TRACE_link_srcdst_variable_set (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_set_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_add, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the value * passed as parameter added to the current value of the variable name to be considered. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set, TRACE_link_srcdst_variable_sub */ void TRACE_link_srcdst_variable_add (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_add_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_sub, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the value * passed as parameter subtracted from the current value of the variable name to be considered. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set, TRACE_link_srcdst_variable_add */ void TRACE_link_srcdst_variable_sub (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_sub_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of the variable present in the links connecting source and destination at a given timestamp. * * Same as #TRACE_link_srcdst_variable_set, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_add_with_time, TRACE_link_srcdst_variable_sub_with_time */ void TRACE_link_srcdst_variable_set_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_SET); } /** \ingroup TRACE_user_variables * \brief Add a value to the variable present in the links connecting source and destination at a given timestamp. * * Same as #TRACE_link_srcdst_variable_add, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set_with_time, TRACE_link_srcdst_variable_sub_with_time */ void TRACE_link_srcdst_variable_add_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_ADD); } /** \ingroup TRACE_user_variables * \brief Subtract a value from the variable present in the links connecting source and dest. at a given timestamp. * * Same as #TRACE_link_srcdst_variable_sub, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set_with_time, TRACE_link_srcdst_variable_add_with_time */ void TRACE_link_srcdst_variable_sub_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_SUB); } /** \ingroup TRACE_user_variables * \brief Get declared user link variables * * This function should be used to get link variables that were already declared with #TRACE_link_variable_declare or * with #TRACE_link_variable_declare_with_color. * * \return A dynar with the declared link variables, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_link_variables () { return instr_set_to_dynar(&user_link_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user state associated to hosts. * * Declare a user state that will be associated to hosts. * A user host state can be used to trace application states. * * \param state The name of the new state to be declared. * * \see TRACE_host_state_declare_value */ void TRACE_host_state_declare (const char *state) { instr_new_user_state_type("HOST", state); } /** \ingroup TRACE_user_variables * \brief Declare a new value for a user state associated to hosts. * * Declare a value for a state. The color needs to be a string with 3 numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param state The name of the new state to be declared. * \param value The name of the value * \param color The color of the value * * \see TRACE_host_state_declare */ void TRACE_host_state_declare_value (const char *state, const char *value, const char *color) { instr_new_value_for_user_state_type (state, value, color); } /** \ingroup TRACE_user_variables * \brief Set the user state to the given value. * * Change a user state previously declared to the given value. * * \param host The name of the host to be considered. * \param state The name of the state previously declared. * \param value The new value of the state. * * \see TRACE_host_state_declare, TRACE_host_push_state, TRACE_host_pop_state, TRACE_host_reset_state */ void TRACE_host_set_state(const char* host, const char* state_name, const char* value_name) { simgrid::instr::StateType* state = simgrid::instr::Container::byName(host)->getState(state_name); state->addEntityValue(value_name); state->setEvent(value_name); } /** \ingroup TRACE_user_variables * \brief Push a new value for a state of a given host. * * Change a user state previously declared by pushing the new value to the state. * * \param host The name of the host to be considered. * \param state The name of the state previously declared. * \param value The value to be pushed. * * \see TRACE_host_state_declare, TRACE_host_set_state, TRACE_host_pop_state, TRACE_host_reset_state */ void TRACE_host_push_state(const char* host, const char* state_name, const char* value_name) { simgrid::instr::Container::byName(host)->getState(state_name)->pushEvent(value_name); } /** \ingroup TRACE_user_variables * \brief Pop the last value of a state of a given host. * * Change a user state previously declared by removing the last value of the state. * * \param host The name of the host to be considered. * \param state The name of the state to be popped. * * \see TRACE_host_state_declare, TRACE_host_set_state, TRACE_host_push_state, TRACE_host_reset_state */ void TRACE_host_pop_state(const char* host, const char* state_name) { simgrid::instr::Container::byName(host)->getState(state_name)->popEvent(); } /** \ingroup TRACE_API * \brief Get Paje container types that can be mapped to the nodes of a graph. * * This function can be used to create a user made graph configuration file for Triva. Normally, it is used with the * functions defined in \ref TRACE_user_variables. * * \return A dynar with the types, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_node_types () { return instr_set_to_dynar(&trivaNodeTypes); } /** \ingroup TRACE_API * \brief Get Paje container types that can be mapped to the edges of a graph. * * This function can be used to create a user made graph configuration file for Triva. Normally, it is used with the * functions defined in \ref TRACE_user_variables. * * \return A dynar with the types, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_edge_types () { return instr_set_to_dynar(&trivaEdgeTypes); } SimGrid-3.18/src/instr/instr_paje_trace.cpp0000644000175000017500000001212113217757321021262 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/sg_config.h" #include "src/instr/instr_private.hpp" #include "src/instr/instr_smpi.hpp" #include "src/smpi/include/private.hpp" #include "typeinfo" #include "xbt/virtu.h" /* sg_cmdline */ #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_paje_trace, instr, "tracing event system"); static std::stringstream stream; FILE *tracing_file = nullptr; std::vector buffer; void buffer_debug(std::vector* buf); void dump_comment(std::string comment) { if (comment.empty()) return; fprintf(tracing_file, "# %s\n", comment.c_str()); } void dump_comment_file(std::string filename) { if (filename.empty()) return; std::ifstream* fs = new std::ifstream(); fs->open(filename.c_str(), std::ifstream::in); if (fs->fail()) { THROWF(system_error, 1, "Comment file %s could not be opened for reading.", filename.c_str()); } while (not fs->eof()) { std::string line; fprintf (tracing_file, "# "); std::getline(*fs, line); fprintf(tracing_file, "%s", line.c_str()); } fs->close(); } double TRACE_last_timestamp_to_dump = 0; //dumps the trace file until the timestamp TRACE_last_timestamp_to_dump void TRACE_paje_dump_buffer(bool force) { if (not TRACE_is_enabled()) return; XBT_DEBUG("%s: dump until %f. starts", __FUNCTION__, TRACE_last_timestamp_to_dump); if (force){ for (auto const& event : buffer) { event->print(); delete event; } buffer.clear(); }else{ std::vector::iterator i = buffer.begin(); for (auto const& event : buffer) { double head_timestamp = event->timestamp_; if (head_timestamp > TRACE_last_timestamp_to_dump) break; event->print(); delete event; ++i; } buffer.erase(buffer.begin(), i); } XBT_DEBUG("%s: ends", __FUNCTION__); } void buffer_debug(std::vector* buf) { if (not XBT_LOG_ISENABLED(instr_paje_trace, xbt_log_priority_debug)) return; XBT_DEBUG(">>>>>> Dump the state of the buffer. %zu events", buf->size()); for (auto const& event : *buf) { event->print(); XBT_DEBUG("%p %s", event, stream.str().c_str()); stream.str(""); stream.clear(); } XBT_DEBUG("<<<<<<"); } /* internal do the instrumentation module */ void simgrid::instr::PajeEvent::insertIntoBuffer() { if (not TRACE_buffer()) { print(); delete this; return; } buffer_debug(&buffer); XBT_DEBUG("%s: insert event_type=%u, timestamp=%f, buffersize=%zu)", __FUNCTION__, eventType_, timestamp_, buffer.size()); std::vector::reverse_iterator i; for (i = buffer.rbegin(); i != buffer.rend(); ++i) { simgrid::instr::PajeEvent* e1 = *i; XBT_DEBUG("compare to %p is of type %u; timestamp:%f", e1, e1->eventType_, e1->timestamp_); if (e1->timestamp_ <= timestamp_) break; } if (i == buffer.rend()) XBT_DEBUG("%s: inserted at beginning", __FUNCTION__); else if (i == buffer.rbegin()) XBT_DEBUG("%s: inserted at end", __FUNCTION__); else XBT_DEBUG("%s: inserted at pos= %zd from its end", __FUNCTION__, std::distance(buffer.rbegin(), i)); buffer.insert(i.base(), this); buffer_debug(&buffer); } void TRACE_paje_start() { std::string filename = TRACE_get_filename(); tracing_file = fopen(filename.c_str(), "w"); if (tracing_file == nullptr){ THROWF(system_error, 1, "Tracefile %s could not be opened for writing.", filename.c_str()); } XBT_DEBUG("Filename %s is open for writing", filename.c_str()); /* output generator version */ fprintf (tracing_file, "#This file was generated using SimGrid-%d.%d.%d\n", SIMGRID_VERSION_MAJOR, SIMGRID_VERSION_MINOR, SIMGRID_VERSION_PATCH); fprintf (tracing_file, "#["); unsigned int cpt; char *str; xbt_dynar_foreach (xbt_cmdline, cpt, str){ fprintf(tracing_file, "%s ",str); } fprintf (tracing_file, "]\n"); /* output one line comment */ dump_comment (TRACE_get_comment()); /* output comment file */ dump_comment_file (TRACE_get_comment_file()); /* output header */ TRACE_header(TRACE_basic(),TRACE_display_sizes()); } void TRACE_paje_end() { fclose(tracing_file); XBT_DEBUG("Filename %s is closed", TRACE_get_filename().c_str()); } void TRACE_TI_start() { std::string filename = TRACE_get_filename(); tracing_file = fopen(filename.c_str(), "w"); if (tracing_file == nullptr) { THROWF(system_error, 1, "Tracefile %s could not be opened for writing.", filename.c_str()); } XBT_DEBUG("Filename %s is open for writing", filename.c_str()); /* output one line comment */ dump_comment(TRACE_get_comment()); /* output comment file */ dump_comment_file(TRACE_get_comment_file()); } void TRACE_TI_end() { fclose(tracing_file); XBT_DEBUG("Filename %s is closed", TRACE_get_filename().c_str()); } SimGrid-3.18/src/instr/instr_paje_header.cpp0000644000175000017500000002321213217757321021417 0ustar mquinsonmquinson/* Copyright (c) 2010, 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/sg_config.h" #include "src/instr/instr_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_paje_header, instr, "Paje tracing event system (header)"); extern FILE *tracing_file; static void TRACE_header_PajeDefineContainerType(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineContainerType %u\n", simgrid::instr::PAJE_DefineContainerType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineVariableType(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineVariableType %u\n", simgrid::instr::PAJE_DefineVariableType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%% Color color\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineStateType(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineStateType %u\n", simgrid::instr::PAJE_DefineStateType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineEventType(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineEventType %u\n", simgrid::instr::PAJE_DefineEventType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineLinkType(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineLinkType %u\n", simgrid::instr::PAJE_DefineLinkType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); fprintf(tracing_file, "%% SourceContainerType string\n"); fprintf(tracing_file, "%% DestContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% StartContainerType string\n"); fprintf(tracing_file, "%% EndContainerType string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineEntityValue(bool basic) { fprintf(tracing_file, "%%EventDef PajeDefineEntityValue %u\n", simgrid::instr::PAJE_DefineEntityValue); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% EntityType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%% Color color\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeCreateContainer() { fprintf(tracing_file, "%%EventDef PajeCreateContainer %u\n", simgrid::instr::PAJE_CreateContainer); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Alias string\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDestroyContainer() { fprintf(tracing_file, "%%EventDef PajeDestroyContainer %u\n", simgrid::instr::PAJE_DestroyContainer); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSetVariable() { fprintf(tracing_file, "%%EventDef PajeSetVariable %u\n", simgrid::instr::PAJE_SetVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeAddVariable() { fprintf(tracing_file, "%%EventDef PajeAddVariable %u\n", simgrid::instr::PAJE_AddVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSubVariable() { fprintf(tracing_file, "%%EventDef PajeSubVariable %u\n", simgrid::instr::PAJE_SubVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSetState() { fprintf(tracing_file, "%%EventDef PajeSetState %u\n", simgrid::instr::PAJE_SetState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajePushState(int size) { fprintf(tracing_file, "%%EventDef PajePushState %u\n", simgrid::instr::PAJE_PushState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (size) fprintf(tracing_file, "%% Size int\n"); #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { /** * paje currently (May 2016) uses "Filename" and "Linenumber" as * reserved words. We cannot use them... */ fprintf(tracing_file, "%% Fname string\n"); fprintf(tracing_file, "%% Lnumber int\n"); } #endif fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajePopState() { fprintf(tracing_file, "%%EventDef PajePopState %u\n", simgrid::instr::PAJE_PopState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeResetState(bool basic) { if (basic) return; fprintf(tracing_file, "%%EventDef PajeResetState %u\n", simgrid::instr::PAJE_ResetState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeStartLink(bool basic, int size) { fprintf(tracing_file, "%%EventDef PajeStartLink %u\n", simgrid::instr::PAJE_StartLink); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (basic){ fprintf(tracing_file, "%% SourceContainer string\n"); }else{ fprintf(tracing_file, "%% StartContainer string\n"); } fprintf(tracing_file, "%% Key string\n"); if (size) fprintf(tracing_file, "%% Size int\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeEndLink(bool basic) { fprintf(tracing_file, "%%EventDef PajeEndLink %u\n", simgrid::instr::PAJE_EndLink); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (basic){ fprintf(tracing_file, "%% DestContainer string\n"); }else{ fprintf(tracing_file, "%% EndContainer string\n"); } fprintf(tracing_file, "%% Key string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeNewEvent() { fprintf(tracing_file, "%%EventDef PajeNewEvent %u\n", simgrid::instr::PAJE_NewEvent); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } void TRACE_header(bool basic, int size) { XBT_DEBUG ("Define paje header"); TRACE_header_PajeDefineContainerType(basic); TRACE_header_PajeDefineVariableType(basic); TRACE_header_PajeDefineStateType(basic); TRACE_header_PajeDefineEventType(basic); TRACE_header_PajeDefineLinkType(basic); TRACE_header_PajeDefineEntityValue(basic); TRACE_header_PajeCreateContainer(); TRACE_header_PajeDestroyContainer(); TRACE_header_PajeSetVariable(); TRACE_header_PajeAddVariable(); TRACE_header_PajeSubVariable(); TRACE_header_PajeSetState(); TRACE_header_PajePushState(size); TRACE_header_PajePopState(); TRACE_header_PajeResetState(basic); TRACE_header_PajeStartLink (basic, size); TRACE_header_PajeEndLink(basic); TRACE_header_PajeNewEvent(); } SimGrid-3.18/src/instr/instr_private.hpp0000644000175000017500000002230013217757321020644 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PRIVATE_HPP #define INSTR_PRIVATE_HPP #include #include "instr/instr_interface.h" #include "simgrid/instr.h" #include "simgrid_config.h" #include "src/instr/instr_paje_containers.hpp" #include "src/instr/instr_paje_events.hpp" #include "src/instr/instr_paje_types.hpp" #include "src/instr/instr_paje_values.hpp" #include "src/internal_config.h" #include "xbt/graph.h" #include /** std::setprecision **/ #include #include #include #include #include #ifdef WIN32 #include // _mkdir /* Need to define function drand48 for Windows */ /* FIXME: use _drand48() defined in src/surf/random_mgr.c instead */ #define drand48() (rand() / (RAND_MAX + 1.0)) #endif typedef simgrid::instr::Container* container_t; namespace simgrid { namespace instr { class TIData { std::string name_; double amount_ = 0; public: int endpoint = 0; int send_size = 0; std::vector* sendcounts = nullptr; int recv_size = 0; std::vector* recvcounts = nullptr; std::string send_type = ""; std::string recv_type = ""; // NoOpTI: init, finalize, test, wait, barrier explicit TIData(std::string name) : name_(name){}; // CPuTI: compute, sleep (+ waitAny and waitAll out of laziness) explicit TIData(std::string name, double amount) : name_(name), amount_(amount){}; // Pt2PtTI: send, isend, sssend, issend, recv, irecv explicit TIData(std::string name, int endpoint, int size, std::string datatype) : name_(name), endpoint(endpoint), send_size(size), send_type(datatype){}; // CollTI: bcast, reduce, allReduce, gather, scatter, allGather, allToAll explicit TIData(std::string name, int root, double amount, int send_size, int recv_size, std::string send_type, std::string recv_type) : name_(name) , amount_(amount) , endpoint(root) , send_size(send_size) , recv_size(recv_size) , send_type(send_type) , recv_type(recv_type){}; // VarCollTI: gatherV, scatterV, allGatherV, allToAllV (+ reduceScatter out of laziness) explicit TIData(std::string name, int root, int send_size, std::vector* sendcounts, int recv_size, std::vector* recvcounts, std::string send_type, std::string recv_type) : name_(name) , endpoint(root) , send_size(send_size) , sendcounts(sendcounts) , recv_size(recv_size) , recvcounts(recvcounts) , send_type(send_type) , recv_type(recv_type){}; virtual ~TIData() { delete sendcounts; delete recvcounts; } std::string getName() { return name_; } double getAmount() { return amount_; } virtual std::string print() = 0; virtual std::string display_size() = 0; }; class NoOpTIData : public TIData { public: explicit NoOpTIData(std::string name) : TIData(name){}; std::string print() override { return getName(); } std::string display_size() override { return ""; } }; class CpuTIData : public TIData { public: explicit CpuTIData(std::string name, double amount) : TIData(name, amount){}; std::string print() override { std::stringstream stream; stream << getName() << " " << getAmount(); return stream.str(); } std::string display_size() override { return std::to_string(getAmount()); } }; class Pt2PtTIData : public TIData { public: explicit Pt2PtTIData(std::string name, int endpoint, int size, std::string datatype) : TIData(name, endpoint, size, datatype){}; std::string print() override { std::stringstream stream; stream << getName() << " "; if (endpoint >= 0) stream << endpoint << " "; stream << send_size << " " << send_type; return stream.str(); } std::string display_size() override { return std::to_string(send_size); } }; class CollTIData : public TIData { public: explicit CollTIData(std::string name, int root, double amount, int send_size, int recv_size, std::string send_type, std::string recv_type) : TIData(name, root, amount, send_size, recv_size, send_type, recv_type){}; std::string print() override { std::stringstream stream; stream << getName() << " " << send_size << " "; if (recv_size >= 0) stream << recv_size << " "; if (getAmount() >= 0.0) stream << getAmount() << " "; if (endpoint > 0 || (endpoint == 0 && not send_type.empty())) stream << endpoint << " "; stream << send_type << " " << recv_type; return stream.str(); } std::string display_size() override { return std::to_string(send_size); } }; class VarCollTIData : public TIData { public: explicit VarCollTIData(std::string name, int root, int send_size, std::vector* sendcounts, int recv_size, std::vector* recvcounts, std::string send_type, std::string recv_type) : TIData(name, root, send_size, sendcounts, recv_size, recvcounts, send_type, recv_type){}; std::string print() override { std::stringstream stream; stream << getName() << " "; if (send_size >= 0) stream << send_size << " "; if (sendcounts != nullptr) for (auto count : *sendcounts) stream << count << " "; if (recv_size >= 0) stream << recv_size << " "; if (recvcounts != nullptr) for (auto count : *recvcounts) stream << count << " "; if (endpoint > 0 || (endpoint == 0 && not send_type.empty())) stream << endpoint << " "; stream << send_type << " " << recv_type; return stream.str(); } std::string display_size() override { return std::to_string(send_size > 0 ? send_size : recv_size); } }; } } extern "C" { extern XBT_PRIVATE std::set created_categories; extern XBT_PRIVATE std::set declared_marks; extern XBT_PRIVATE std::set user_host_variables; extern XBT_PRIVATE std::set user_vm_variables; extern XBT_PRIVATE std::set user_link_variables; extern XBT_PRIVATE double TRACE_last_timestamp_to_dump; /* instr_paje_header.c */ XBT_PRIVATE void TRACE_header(bool basic, int size); /* from paje.c */ XBT_PRIVATE void TRACE_paje_start(); XBT_PRIVATE void TRACE_paje_end(); /* from instr_config.c */ XBT_PRIVATE bool TRACE_needs_platform(); XBT_PRIVATE bool TRACE_is_enabled(); XBT_PRIVATE bool TRACE_platform(); XBT_PRIVATE bool TRACE_platform_topology(); XBT_PRIVATE bool TRACE_is_configured(); XBT_PRIVATE bool TRACE_categorized(); XBT_PRIVATE bool TRACE_uncategorized(); XBT_PRIVATE bool TRACE_msg_process_is_enabled(); XBT_PRIVATE bool TRACE_msg_vm_is_enabled(); XBT_PRIVATE bool TRACE_buffer(); XBT_PRIVATE bool TRACE_disable_link(); XBT_PRIVATE bool TRACE_disable_speed(); XBT_PRIVATE bool TRACE_onelink_only(); XBT_PRIVATE bool TRACE_disable_destroy(); XBT_PRIVATE bool TRACE_basic(); XBT_PRIVATE bool TRACE_display_sizes(); XBT_PRIVATE int TRACE_precision(); XBT_PRIVATE void instr_pause_tracing(); XBT_PRIVATE void instr_resume_tracing(); /* Public functions used in SMPI */ XBT_PUBLIC(bool) TRACE_smpi_is_enabled(); XBT_PUBLIC(bool) TRACE_smpi_is_grouped(); XBT_PUBLIC(bool) TRACE_smpi_is_computing(); XBT_PUBLIC(bool) TRACE_smpi_is_sleeping(); XBT_PUBLIC(bool) TRACE_smpi_view_internals(); /* from resource_utilization.c */ XBT_PRIVATE void TRACE_surf_host_set_utilization(const char* resource, const char* category, double value, double now, double delta); XBT_PRIVATE void TRACE_surf_link_set_utilization(const char* resource, const char* category, double value, double now, double delta); XBT_PUBLIC(void) TRACE_surf_resource_utilization_alloc(); /* instr_paje.c */ extern XBT_PRIVATE std::set trivaNodeTypes; extern XBT_PRIVATE std::set trivaEdgeTypes; XBT_PRIVATE long long int instr_new_paje_id(); void instr_new_variable_type(std::string new_typename, std::string color); void instr_new_user_variable_type(std::string father_type, std::string new_typename, std::string color); void instr_new_user_state_type(std::string father_type, std::string new_typename); void instr_new_value_for_user_state_type(std::string new_typename, const char* value, std::string color); /* instr_config.c */ XBT_PRIVATE void TRACE_TI_start(); XBT_PRIVATE void TRACE_TI_end(); XBT_PRIVATE void TRACE_paje_dump_buffer(bool force); XBT_PRIVATE void dump_comment_file(std::string filename); XBT_PRIVATE void dump_comment(std::string comment); /* Format of TRACING output. * - paje is the regular format, that we all know * - TI is a trick to reuse the tracing functions to generate a time independent trace during the execution. Such * trace can easily be replayed with smpi_replay afterward. This trick should be removed and replaced by some code * using the signal that we will create to cleanup the TRACING */ enum instr_fmt_type_t { instr_fmt_paje, instr_fmt_TI }; extern instr_fmt_type_t instr_fmt_type; } XBT_PRIVATE std::string TRACE_get_comment(); XBT_PRIVATE std::string TRACE_get_comment_file(); XBT_PRIVATE std::string TRACE_get_filename(); #endif SimGrid-3.18/src/instr/instr_resource_utilization.cpp0000644000175000017500000000623613217757321023461 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.hpp" #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_resource, instr, "tracing (un)-categorized resource utilization"); //to check if variables were previously set to 0, otherwise paje won't simulate them static std::set platform_variables; static void instr_event(double now, double delta, simgrid::instr::VariableType* variable, container_t resource, double value) { /* To trace resource utilization, we use AddEvent and SubEvent only. This implies to add a SetEvent first to set the * initial value of all variables for subsequent adds/subs. If we don't do so, the first AddEvent would be added to a * non-determined value, hence causing analysis problems. */ // create a key considering the resource and variable std::string key = resource->getName() + variable->getName(); // check if key exists: if it doesn't, set the variable to zero and mark this in the global map. if (platform_variables.find(key) == platform_variables.end()) { variable->setEvent(now, 0); platform_variables.insert(key); } variable->addEvent(now, value); variable->subEvent(now + delta, value); } static void TRACE_surf_resource_set_utilization(const char* type, const char* name, const char* resource, const char* category, double value, double now, double delta) { // only trace resource utilization if resource is known by tracing mechanism container_t container = simgrid::instr::Container::byNameOrNull(resource); if (not container || not value) return; // trace uncategorized resource utilization if (TRACE_uncategorized()){ XBT_DEBUG("UNCAT %s [%f - %f] %s %s %f", type, now, now + delta, resource, name, value); simgrid::instr::VariableType* variable = container->getVariable(name); instr_event(now, delta, variable, container, value); } // trace categorized resource utilization if (TRACE_categorized()){ if (not category) return; std::string category_type = name[0] + std::string(category); XBT_DEBUG("CAT %s [%f - %f] %s %s %f", type, now, now + delta, resource, category_type.c_str(), value); simgrid::instr::VariableType* variable = container->getVariable(category_type); instr_event(now, delta, variable, container, value); } } /* TRACE_surf_link_set_utilization: entry point from SimGrid */ void TRACE_surf_link_set_utilization(const char* resource, const char* category, double value, double now, double delta) { TRACE_surf_resource_set_utilization("LINK", "bandwidth_used", resource, category, value, now, delta); } /* TRACE_surf_host_set_utilization: entry point from SimGrid */ void TRACE_surf_host_set_utilization(const char* resource, const char* category, double value, double now, double delta) { TRACE_surf_resource_set_utilization("HOST", "power_used", resource, category, value, now, delta); } SimGrid-3.18/src/instr/instr_paje_values.hpp0000644000175000017500000000133013217757321021470 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PAJE_VALUES_HPP #define INSTR_PAJE_VALUES_HPP #include "src/instr/instr_private.hpp" #include namespace simgrid { namespace instr { class EntityValue { long long int id_; std::string name_; std::string color_; Type* father_; public: explicit EntityValue(std::string name, std::string color, Type* father); ~EntityValue() = default; const char* getCname() { return name_.c_str(); } long long int getId() { return id_; } void print(); }; } } #endif SimGrid-3.18/src/instr/instr_paje_values.cpp0000644000175000017500000000220013217757321021460 0ustar mquinsonmquinson/* Copyright (c) 2012-2017. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_values, instr, "Paje tracing event system (values)"); extern FILE* tracing_file; namespace simgrid { namespace instr { EntityValue::EntityValue(std::string name, std::string color, Type* father) : id_(instr_new_paje_id()), name_(name), color_(color), father_(father){}; void EntityValue::print() { if (instr_fmt_type != instr_fmt_paje) return; std::stringstream stream; XBT_DEBUG("%s: event_type=%u", __FUNCTION__, PAJE_DefineEntityValue); stream << std::fixed << std::setprecision(TRACE_precision()) << PAJE_DefineEntityValue; stream << " " << id_ << " " << father_->getId() << " " << name_; if (not color_.empty()) stream << " \"" << color_ << "\""; XBT_DEBUG("Dump %s", stream.str().c_str()); fprintf(tracing_file, "%s\n", stream.str().c_str()); } } } SimGrid-3.18/src/instr/jedule/0000755000175000017500000000000013217757322016516 5ustar mquinsonmquinsonSimGrid-3.18/src/instr/jedule/jedule.cpp0000644000175000017500000000256013217757322020475 0ustar mquinsonmquinson/* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/asserts.h" #include "simgrid/jedule/jedule.hpp" #if SIMGRID_HAVE_JEDULE namespace simgrid{ namespace jedule { Jedule::~Jedule() { delete this->root_container; for (auto const& evt : this->event_set) delete evt; this->event_set.clear(); } void Jedule::addMetaInfo(char *key, char *value) { xbt_assert(key != nullptr); xbt_assert(value != nullptr); this->meta_info.insert({key, value}); } void Jedule::writeOutput(FILE *file) { if (not this->event_set.empty()) { fprintf(file, "\n"); if (not this->meta_info.empty()) { fprintf(file, " \n"); for (auto const& elm : this->meta_info) fprintf(file, " \n",elm.first,elm.second); fprintf(file, " \n"); } fprintf(file, " \n"); this->root_container->print(file); fprintf(file, " \n"); fprintf(file, " \n"); for (auto const& event : this->event_set) event->print(file); fprintf(file, " \n"); fprintf(file, "\n"); } } } } #endif SimGrid-3.18/src/instr/jedule/jedule_sd_binding.cpp0000644000175000017500000000327113217757322022655 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/jedule/jedule.hpp" #include "src/simdag/simdag_private.hpp" #include "simgrid/s4u/Engine.hpp" #include "simgrid/s4u/NetZone.hpp" #if SIMGRID_HAVE_JEDULE XBT_LOG_NEW_CATEGORY(jedule, "Logging specific to Jedule"); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(jed_sd, jedule, "Logging specific to Jedule SD binding"); jedule_t my_jedule; void jedule_log_sd_event(SD_task_t task) { xbt_assert(task != nullptr); jed_event_t event = new simgrid::jedule::Event(std::string(SD_task_get_name(task)), SD_task_get_start_time(task), SD_task_get_finish_time(task), "SD"); event->addResources(task->allocation); my_jedule->event_set.push_back(event); } void jedule_sd_init() { sg_netzone_t root_comp = simgrid::s4u::Engine::getInstance()->getNetRoot(); XBT_DEBUG("root name %s\n", root_comp->getCname()); my_jedule = new simgrid::jedule::Jedule(); jed_container_t root_container = new simgrid::jedule::Container(std::string(root_comp->getCname())); root_container->createHierarchy(root_comp); my_jedule->root_container = root_container; } void jedule_sd_exit() { delete my_jedule; } void jedule_sd_dump(const char * filename) { if (my_jedule) { char *fname; if (not filename) { fname = bprintf("%s.jed", xbt_binary_name); } else { fname = xbt_strdup(filename); } FILE *fh = fopen(fname, "w"); my_jedule->writeOutput(fh); fclose(fh); xbt_free(fname); } } #endif SimGrid-3.18/src/instr/jedule/jedule_platform.cpp0000644000175000017500000001414513217757322022403 0ustar mquinsonmquinson/* Copyright (c) 2010-2017. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/jedule/jedule.hpp" #include "simgrid/host.h" #include "simgrid/s4u/NetZone.hpp" #include "xbt/asserts.h" #include #if SIMGRID_HAVE_JEDULE namespace { std::unordered_map host2_simgrid_parent_container; std::unordered_map container_name2container; } namespace simgrid { namespace jedule { Subset::Subset(int start_idx, int end_idx, Container* parent) : start_idx(start_idx), parent(parent) { nres=end_idx-start_idx+1; } Container::Container(std::string name): name(name) { container_name2container.insert({this->name, this}); } Container::~Container() { if (not this->children.empty()) for (auto const& child : this->children) delete child; } void Container::addChild(jed_container_t child) { xbt_assert(child != nullptr); this->children.push_back(child); child->parent = this; } void Container::addResources(std::vector hosts) { this->is_lowest = 1; this->children.clear(); this->last_id = 0; //FIXME do we need to sort?: xbt_dynar_sort_strings(host_names); for (auto const& host : hosts) { const char *host_name = sg_host_get_name(host); this->name2id.insert({host_name, this->last_id}); (this->last_id)++; host2_simgrid_parent_container.insert({host_name, this}); this->resource_list.push_back(host); } } void Container::createHierarchy(sg_netzone_t from_as) { if (from_as->getChildren()->empty()) { // I am no AS // add hosts to jedule platform std::vector table; from_as->getHosts(&table); this->addResources(table); } else { for (auto const& nz : *from_as->getChildren()) { jed_container_t child_container = new simgrid::jedule::Container(std::string(nz->getCname())); this->addChild(child_container); child_container->createHierarchy(nz); } } } std::vector Container::getHierarchy() { if(this->parent != nullptr ) { if (not this->parent->children.empty()) { // we are in the last level return this->parent->getHierarchy(); } else { unsigned int i =0; int child_nb = -1; for (auto const& child : this->parent->children) { if( child == this) { child_nb = i; break; } i++; } xbt_assert( child_nb > - 1); std::vector heir_list = this->parent->getHierarchy(); heir_list.insert(heir_list.begin(), child_nb); return heir_list; } } else { int top_level = 0; std::vector heir_list = {top_level}; return heir_list; } } std::string Container::getHierarchyAsString() { std::string output(""); std::vector heir_list = this->getHierarchy(); unsigned int length = heir_list.size(); unsigned int i = 0; for (auto const& id : heir_list) { output += std::to_string(id); if( i != length-1 ) { output += "."; } } return output; } void Container::printResources(FILE * jed_file) { unsigned int i=0; xbt_assert(not this->resource_list.empty()); unsigned int res_nb = this->resource_list.size(); std::string resid = this->getHierarchyAsString(); fprintf(jed_file, " resource_list) { const char * res_name = sg_host_get_name(res); fprintf(jed_file, "%s", res_name); if( i != res_nb-1 ) { fprintf(jed_file, "|"); } i++; } fprintf(jed_file, "\" />\n"); } void Container::print(FILE* jed_file) { fprintf(jed_file, " \n", this->name.c_str()); if (not this->children.empty()) { for (auto const& child : this->children) { child->print(jed_file); } } else { this->printResources(jed_file); } fprintf(jed_file, " \n"); } } } static void add_subsets_to(std::vector *subset_list, std::vector hostgroup, jed_container_t parent) { // get ids for each host // sort ids // compact ids // create subset for each id group xbt_assert( parent != nullptr ); std::vector id_list; for (auto const& host_name : hostgroup) { xbt_assert( host_name != nullptr ); jed_container_t parent = host2_simgrid_parent_container.at(host_name); unsigned int id = parent->name2id.at(host_name); id_list.push_back(id); } unsigned int nb_ids = id_list.size(); std::sort(id_list.begin(), id_list.end()); if( nb_ids > 0 ) { int start = 0; int pos = start; for(unsigned int i=0; i 1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[start], id_list[pos], parent)); start = i; if( i == nb_ids-1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[i], id_list[i], parent)); } } else { if( i == nb_ids-1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[start], id_list[i], parent)); } } pos = i; } } } void get_resource_selection_by_hosts(std::vector *subset_list, std::vector *host_list) { xbt_assert( host_list != nullptr ); // for each host name // find parent container // group by parent container std::unordered_map> parent2hostgroup; for (auto const& host : *host_list) { const char *host_name = sg_host_get_name(host); jed_container_t parent = host2_simgrid_parent_container.at(host_name); xbt_assert( parent != nullptr ); auto host_group = parent2hostgroup.find(parent->name.c_str()); if (host_group == parent2hostgroup.end()) parent2hostgroup.insert({parent->name.c_str(), std::vector(1,host_name)}); else host_group->second.push_back(host_name); } for (auto const& elm : parent2hostgroup) { jed_container_t parent = container_name2container.at(elm.first); add_subsets_to(subset_list, elm.second, parent); } } #endif SimGrid-3.18/src/instr/jedule/jedule_events.cpp0000644000175000017500000000530213217757322022056 0ustar mquinsonmquinson/* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/jedule/jedule.hpp" #include "simgrid/s4u/NetZone.hpp" #include "xbt/asserts.h" #if SIMGRID_HAVE_JEDULE namespace simgrid{ namespace jedule{ Event::Event(std::string name, double start_time, double end_time, std::string type) : name(name), start_time(start_time), end_time(end_time), type(type) { this->resource_subsets = new std::vector(); } Event::~Event() { if (not this->resource_subsets->empty()) { for (auto const& subset : *this->resource_subsets) delete subset; delete this->resource_subsets; } } void Event::addResources(std::vector *host_selection) { get_resource_selection_by_hosts(this->resource_subsets, host_selection); } void Event::addCharacteristic(char *characteristic) { xbt_assert( characteristic != nullptr ); this->characteristics_list.push_back(characteristic); } void Event::addInfo(char* key, char *value) { xbt_assert((key != nullptr) && value != nullptr); this->info_map.insert({key, value}); } void Event::print(FILE *jed_file) { fprintf(jed_file, " \n"); fprintf(jed_file, " \n", this->name.c_str()); fprintf(jed_file, " \n", this->start_time); fprintf(jed_file, " \n", this->end_time); fprintf(jed_file, " \n", this->type.c_str()); xbt_assert(not this->resource_subsets->empty()); fprintf(jed_file, " \n"); for (auto const& subset : *this->resource_subsets) { fprintf(jed_file, "