Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
                        University Research and Technology
                        Corporation.  All rights reserved.
Copyright (c) 2004-2006 The University of Tennessee and The University
                        of Tennessee Research Foundation.  All rights
                        reserved.
Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
                        University of Stuttgart.  All rights reserved.
Copyright (c) 2004-2006 The Regents of the University of California.
                        All rights reserved.
Copyright (c) 2006-2020 Cisco Systems, Inc.  All rights reserved.
Copyright (c) 2006      Voltaire, Inc. All rights reserved.
Copyright (c) 2006      Sun Microsystems, Inc.  All rights reserved.
                        Use is subject to license terms.
Copyright (c) 2006-2017 Los Alamos National Security, LLC.  All rights
                        reserved.
Copyright (c) 2010-2017 IBM Corporation.  All rights reserved.
Copyright (c) 2012      Oak Ridge National Labs.  All rights reserved.
Copyright (c) 2012      Sandia National Laboratories.  All rights reserved.
Copyright (c) 2012      University of Houston. All rights reserved.
Copyright (c) 2013      NVIDIA Corporation.  All rights reserved.
Copyright (c) 2013-2019 Intel, Inc.  All rights reserved.
Copyright (c) 2018-2001 Amazon.com, Inc. or its affiliates.  All Rights
                        reserved.
Copyright (c) 2021-2023 Nanook Consulting.  All rights reserved.
$COPYRIGHT$

Additional copyrights may follow

$HEADER$

===========================================================================

This file contains the main features as well as overviews of specific
bug fixes (and other actions) for each version of PRRTE since
version 1.0.


3.0.1 -- TBD
------------------------------------
PR #1696: Always set waitpid-fired flag when proc exits
PR #1694: Fix passing of envars for spawn
PR #1692: Multiple commits
  - Plug small memory leaks
  - Adhere to project's variable declaration convention
  - Cleanup initial implementation of allocate and session control
  - Protect one more place for session_control
  - Fix the routing for non-tree-spawn launch
  - Cleanup prte_info output
  - Ensure we exit cleanly when a daemon fails to start
  - Store prted URI under correct name
  - Remove duplicate computation of routing tree
  - Minor cleanup of verbose output
  - Send direct to HNP if we don't know a route
  - Cleanup some mangled formatting
  - Cleanup some debug output
  - Update the pmix server integration
PR #1686: Update config/oac submodule
PR #1682: Multiple commits
  - Remove wrapper definitions
  - Extend example to check for sessionID and jobID
PR #1678: Multiple commits
  - Add --debug-daemon to prted options
  - Add the rml and routed types to prte_info array
PR #1674: Ensure cmd line errors provide error messages
PR #1672: Multiple commits
  - Correct default binding for map-by node/slot
  - Provide better error message for cross-package binding
  - Add missing command line directives
  - Fix minor issues found in tests
  - Silence compiler warning in test/connect
  - Silence compiler warnings in examples
  - Use exit macros in example dmodex
  - Remove antiquated code that can cause problems
  - Update sphinx requirements
  - Remove the PMIX_SIZE_ESTIMATE support
PR #1660: Correct function name when using older hwloc versions
PR #1658: Multiple commits
  - Properly handle overload-allowed
  - Define MIN for environments that lack it
  - Fix a typo in the README
  - Fix Sphinx parsing error
PR #1644: Multiple commits
  - Add a spawn_multiple test
  - Provide an estimate of the size of registration data
  - Provide size estimates for collective operations that return data
  - Add an option to display the available CPUs
  - Fix generate_dist type
  - Protect against missing attribute definition
  - Set distances array type
  - Add "parseable" display qualifier
  - Generate distances by default
  - Show help message when mapping too low
  - Fix round-robin by obj with multiple cpus/rank
  - Fix map-by slot and node for multiple cpus/rank
PR #1633: First cut of Sphinx/ReadTheDocs docs
PR #1626: Remove unused variable
PR #1622: Multiple commits
  - Don't error out on binding if not required
  - Check for and track scheduler attachment
  - Accept local group op upcalls
  - Add an IOF stress test
  - Protect against earlier PMIx versions
  - Pass our event base for use by the PMIx library as an aux base
  - Update to compatibility with PMIx master branch
  - Silence warning - add return values
  - Begin adding support for new server module fn pointer
  - Designate the framework project as PRTE
  - Minor touchups of the "fault" example
  - Include topology in allocated node info
  - Remove debug
  - Extend query support to include allocation info
  - Update version to 3.0.1
PR #1600: Multiple commits
  - Minor change to check_os_flavors
  - Don't emit an error output if not needed
PR #1597: Fix oac_check_package.m4
PR #1595: build: fix bashisms in configure


3.0.0 -- 13 Nov 2022
------------------------------------
**** NOTE: This release represents a major update to the PRRTE
code base from prior releases. Many of the changes will be
transparent to the user. However, there are a few key areas that
may (likely) impact users. These include:
  * changes to the cmd line interpreter. PRRTE has provided an
    abstraction layer that allows programming library developers
    and environment vendors to customize the cmd line of PRRTE
    binaries (e.g., prterun and prun). This is governed by the
    detection of a "personality" that is to be used for defining
    the cmd line options. The personality can be set either directly
    on the cmd line itself (via the "--personality foo" option) or
    is detected by PRRTE thru the environment.
  * a more extensive "--help" support system.
  * reduction in the "--rank-by" options.
  * addition of new "--output", "--display", and "--runtime-options"
    command line directives
  * removal of the "mindist" mapper due to lack of a maintainer
  * remove of man pages for PRRTE tools - this has been replaced
    with a hierarchical help system. Thus, the command "prun --help"
    will output the former man page for the prun command. Users
    can then access more detailed help on any command line option
    by simply adding it to the help command - i.e., "prun --help <option>"
There are quite a few more changes than listed here, but they mostly
pertain to code internals. However, these dictate that PRRTE v3.0.0
require use of PMIx v4.2.2 or above to compile. Note that this doesn't
impact the PMIx level associated with any applications executed by
PRRTE - the only requirement is that applications link against a
PMIx version that meets the PMIx cross-version requirements (i.e., a
minimum of PMIx v2.5).

Detailed changes:

PR #1591: Multiple commits
  - fix conflict
  - remove unused vars in tools area
  - adjust prterun to prefix ns: for dvm option
  - Cleanup the --dvm option a bit
  - Some more minor cleanups of --dvm
PR #1586: Multiple commits
  - Extend PBS allocation detection to cover Cobalt
  - Broaden ability to require PRRTE to discover slots
  - Fix default slots count when using --use-hwthread-cpus
  - Cleanup schizo/prte help files
  - schizo-ompi: add examples for verbose [dvm] block
  - schizo:ompi: require arg when specifying dvm
  - rank_file/LSF: Fix LSB_AFFINITY_HOSTFILE ranking policy
  - Fix the load-balancing and span round-robin mappers
  - doc: Fix --rank-by help output
PR #1575: Ensure we init the available node bit tracker
PR #1572: Fix some default map/rank policy settings
PR #1570: Multiple commits
  - rmaps/base: Fix segv on failed-map
  - ompi/schizo: Fix deprecated message for rankfile
  - Rankfile slots fix and LSF support fix for LSB_AFFINITY_HOSTFILE
PR #1564: Add an example that retrieves node IDs
PR #1562: Remove check for resilience that caused a segfault
PR #1557: Update NEWS
PR #1556: Multiple commits
  - Support query of pset membership
  - Cleanup grpcomm cruft
  - remove some unused vars
  - squash warnings
  - unused var
  - more unused vars
  - remove unused vars and fix rc/ret typo
  - misc unused var cleanups
  - Reduce debugger confusion
  - Pickup missing changes
PR #1548: Multiple commits
  - alps fixes for mca move
  - schizo/ompi: Fix --report-pid/sid.
  - Change --stop-in-* to take optional arguments.
  - Update Open MPI mpirun help text
  - Fix memory leaks in RML and at job termination.
  - plm/tm: Fix build breakage
PR #1539: Multiple commits
  - Fix a typo and expand debugger example range to cover MPI
  - ras/lsf: Fix build breakage
  - Catch some more component updates
  - Allow prterun to act as prun
PR #1534: Multiple commits
  - Fix print statement
  - Push launch-agent CLI into the env
  - Fix mapping by pe-list when oversubscribed
PR #1530: Actually support the output-proctable option
PR #1528: Add missing commits and new additions
  - Complete help text on notifications
  - BuildRequires: gcc
  - Plug a memory leak
  - schizo/ompi: Fix --use-hwthead-cpus option
  - Fix --preload-binary
PR #1522: Roll version to rc2
PR #1518: schizo_ompi: use the OMPI_MCA_PREFIXES env var
PR #1517: Multiple commits
  - Remove the --with-ft configure option
  - Replace prte_output with pmix_output
  - Cleanup code clause identification
  - Fix a few minor things
  - Fix setting of runtime options
  - Update the schizo/ompi component
  - ompi/schizo: Fix --do-not-launch and --show-progress
  - schizo/ompi: Fix --display topo
  - schizo/ompi: Remove --display-devel-allocation
  - Added specfile BuildRequires
  - Add action to check commits on pull requests
  - Restore the PRRTE application-level fault tolerance modes
  - Update minimum PMIx version to v4.2.2
PR #1501: Multiple commits
  - Delete PRRTE MCA infrastructure
  - Remove stale m4
  - Handle default MCA params
  - Rename component libraries and component struct names
PR #1498: Remove leftover BINDTONONE SET debug code
PR #1495: Multiple commits
  - Revise passing of runtime options
  - Add/modify some spawn-related tests
  - Update to use PMIx CLI option macro and "check true"
PR #1491: Multiple commits
  - stream-buffering is an OMPI option, so make it an MCA
  - Deprecated the --stream-buffering CLI option
  - Fix hetero topology operations
PR #1487: Use correct target path for Flex check
PR #1485: Fix bug in not updating cached daemon topologies
PR #1483: Fix oversubscription to not require binding
PR #1481: Multiple commits
  - Add support for querying process group attributes
  - Initial work on --enable-devel-check
  - Fix free in hostfile
  - Continue progress on enable-devel-check
PR #1474: Multiple commits
  - Upgrade the cmd line option checker
  - Enable the "show-progress" runtime option
  - Fix the stack trace handshake
  - Add more debug to the node list construction procedure
PR #1469: Multiple commits
  - Raise the minimum required PMIx level to v4.2.1
  - Separate out the runtime controls options
PR #1466: Multiple commits
  - Ensure tools get string explanation for non-norm term
  - stop-in-app does not require an argument
  - Fix compile error with FT enabled
  - schizo/ompi: Fix --do-not-launch
  - Fix typo in help message
  - schizo/ompi: Add translation for --merge-stderr-to-stdout
  - java: add ease-of-use feature for ompi java apps
  - Fix second pass on map-by obj
  - Remove stale references to alloc-dev as display options
  - Get the "--display topo" option to work
PR #1456: Multiple commits
  - Fix synthetic topo support and too early sanity check
  - Correct the support check logic
  - Remove the "report" qualifier from "--bind-to"
  - Fix bind-upward check and cleanup bind output
PR #1450: Update current state of PRRTE help docs
PR #1448: Provide upgrade warning for ancient Slurm versions
PR #1447: Fix app/local rank computation
PR #1444: Fix keepalive CLI option
PR #1441: Include the PMIx libdir in the ssh prefix operation
PR #1440: Do not error out if lib is symlinked to lib64
PR #1437: Avoid loopback interfaces if not needed
PR #1433: Update the job object's binding policy
PR #1431: Fix the tm configure logic
PR #1429: Update help docs
PR #1427: Cleanup the PRRTE rpm spec file
PR #1423: Fix typo in ppr mapper
PR #1417: Remove stale file reference
PR #1415: Update release files for rc1
PR #1414: Remove stale files
PR #1412: Restrict PRRTE to PMIx v4.2+
PR #1410: Remove schizo/hydra component
PR #1409: Add new output options to help string
PR #1407: Support a broader range of IO output formats
PR #1404: Fix a hang in daemon callback
PR #1402: Cleanup race condition in daemon reports
PR #1399: Multiple commits
  - Allow mapping in overload scenario if bind not specified
  - Ensure rankfile and seq mappers computer local and app ranks
PR #1394: Add forwarding of stdin to indirect example
PR #1391: Change the default mapping for --bind-to none option to BYSLOT.
PR #1388: Handle clean shutdown of stdin
PR #1398: Fix stdin forwarding across nodes
PR #1386: Update the dmodex example
PR #1385: Return the PMIx version of "not supported"
PR #1383: Fix resource usage tracking and remove stale mapper
PR #1380: Return zero status when outputting help/version info
PR #1378: Fix a few bugs/leaks in the OOB subsystem
PR #1376: Remove stale Java support
PR #1374: Avoid PMIx server release of HWLOC topology
PR #1373: Use fileno_unlocked if available
PR #1372: Incremental valgrind improvements
PR #1368: prrte: fix core dump while printing stack-trace
PR #1365: odls: fix alps compilation problem
PR #1363: Support colocation of spawned procs
PR #1359: configure.ac: update directory space check
PR #1357: configury: do look for sed
PR #1355: Remove setting of PRTE_MCA_prte_base_help_aggregate.
PR #1353: Restore noloop for logging
PR #1350: Use PMIx_Log() for show_help() messages
PR #1348: Update to account for PMIX_MYSERVER_URI
PR #1347: Streamline operations a bit
PR #1340: Error out if no atomic support is available
PR #1337: Add atomistic error outputs
PR #1336: Restore detailed error msg for some codes
PR #1334: Correctly add qualifiers to options
PR #1333: dist: minor fixes to enable all_in_one rpm
PR #1331: Remove non-existent function
PR #1328: schitzo/ompi: trivial helpfile update
PR #1327: rmaps/ppr: Fix case where oversubscribe is ignored
PR #1325: Cleanup report of bad executable name
PR #1323: Revise show_help to use PMIx IOF
PR #1319: prtereachable: missed something in pr 1315
PR #1318: prte: check if dvm actually got set up
PR #1315: prtereachable: fix problem with nl-route
PR #1314: Correct --do-not-launch option
PR #1312: Restore use of "--cpu-bind=none"
PR #1311: ompi/schizo: Expose "--mca" when parsing command line
PR #1310: Pass the allow-run-as-root option to the backend daemons
PR #1307: Fix indirect slurm launch
PR #1306: Protect against proxy confusion
PR #1303: Add some missing help verbiage
PR #1302: Bugfix: ompi_schizo would modify a const string in base_expose()
PR #1301: slurm: fix breakage owing to rlm refactor
PR #1299: Correctly determine when to daemonize backend prted
PR #1298: Some really minor cleanups
PR #1297: Add missing CLI option and parsing
PR #1293: build: check_package static improvements
PR #1292: build: Fix output stream bug in summary
PR #1291: Fix a problem with the "canonicalize_path" function
PR #1290: build: Remove unnecessary configure tests
PR #1289: build: Remove .m4 files from dist list
PR #1288: build: Do not use pkg-config with libevent
PR #1287: schizos: add --version output for ompi and prte
PR #1286: Build system improvements around check_package
PR #1285: Fix compile warnings and valgrind errors
PR #1275: ras/pbs: fix arguments invokation in configur
PR #1269: schizo/ompi: Convert all single dashes to double dashes
PR #1268: Use a pointer array to track our local requests
PR #1267: Add debugger.h to tarball
PR #1265: Use pmix_hotel_checkin() in eviction callback.
PR #1261: Ensure that stdin goes to all specified targets
PR #1260: Ensure the proc PIDs are updated upon launch
PR #1258: Carry across the PRRTE toolname to PMIx
PR #1257: Fix some issues with tool help
PR #1256: Ensure the proctable includes absolute paths to executable
PR #1254: Remove stale files
PR #1247: Fix includes for data-structures in prtereachable
PR #1246: Fix one more place in dmodex example
PR #1244: Fix ipv6 build
PR #1242: Correct some pmix class status code checks
PR #1241: Fix some PMIx vs PRRTE confusion
PR #1240: Fix PRTE_RML_SEND() call to squash bug/warning.
PR #1239: More FT-enabled compile fixes for RML
PR #1238: Fix rml send use when FT enabled
PR #1236: Few small changes to make_dist_tarball
PR #1235: Add a GitHub Action for the PRRTE Special Builds
PR #1234: Eliminate PRTE_FLAGS_UNIQ
PR #1232: RML Rework: Stage 3 - remove routed framework
PR #1231: Remove confusing garbage file
PR #1229: RML rework: Stage 2 - convert RML framework
PR #1228: Rewrite CHECK_PACKAGE to support pkg-config
PR #1227: Fix some ordering issues on cleanup, and a couple memory leaks
PR #1226: Fix find and replace errors for pmix_fd
PR #1225: RML rework: Stage 1 - remove unused components
PR #1224: Convert keyval_parse to pmix_keyval_parse
PR #1223: Replace context_fns with their PMIx equivalents
PR #1220: Convert prte_cmd_line to pmix_cmd_line
PR #1218: Update PMIx configury
PR #1217: Convert multiple PRTE utilities to their PMIx equivalents
PR #1216: Convert printf to pmix_printf
PR #1215: Use pmix_argv functions and remove the prte versions
PR #1214: Convert basename.h to pmix_basename.h
PR #1213: Fix --mca processing.
PR #1210: Update TM configure logic to include tm library location
PR #1207: Ensure tool procs are not counted against resource limits
PR #1202: Fix typo
PR #1201: Fix the command line parser
PR #1200: Fix the param and show-version options of prte_info
PR #1198: Cleanup a bit in prep for PMIx sync to v5 std
PR #1195: Remove HWLOC version block
PR #1194: Enable per-job specification of the exec agent
PR #1190: Add missing examples to tarball
PR #1189: Ensure an error gets reported when rankfile fails


2.0.2: 11 Feb 2022
-------------------
**** NOTE: This release contains a workaround that resolves the prior
**** conflict with HWLOC versions 2.5.0 through 2.7.0 (inclusive).
**** Those versions of HWLOC are now supported.

PR #1197: Cleanup a bit in prep for PMIx v5 release
PR #1195: Remove HWLOC version block
PR #1190: Add missing examples to tarball
PR #1189: Ensure an error gets reported when rankfile fails


2.0.1: 1 Feb 2022
------------------------------------
**** NOTE: As of v2.0.1, PRRTE no longer has a dependency on "pandoc"
**** for building and installing man pages.

**** NOTE: PRRTE has identified a bug in HWLOC versions 2.5.0 thru
**** 2.7.0 (inclusive) that causes PMIx to segfault during certain
**** operations. We have worked with the HWLOC developers to fix
**** that problem, and the fix will be included beginning in HWLOC
**** version 2.7.1. Accordingly, PRRTE now checks for the problem
**** HWLOC versions and will error out of configure if they are
**** encountered.

PR #1185 and 1186: Update HWLOC version detection
    - Reject versions 2.5.0-2.7.0, inclusive
PR #1183: Always build the PBS scheduler support
PR #1182: Cleanup handling of allocated node names
PR #1169: Updates for rc5
   - Enable support for PMIX_IOF_OUTPUT_RAW attribute
   - Update NEWS
PR #1167: Updates
   - Be more flexible in library handling
   - Update libevent/hwloc handling to match PMIx
   - Remove event header defines
   - Minor cleanups and ensure no local IOF copy
     when persistent
   - change the pcc wrapper compiler to a symlink
     to pmixcc
   - Cleanup code a bit
   - Select all adapters with IPv4 addresses within
     specified subnet ranges
   - validate input for mpiexec --with-ft options
PR #1158: Updates
   - Fix uninitialized variable
   - Update NEWS
PR #1157: Fix brackets around code clause that caused
     erroneous "unable to find tune file" error
PR #1156: Handle default param case for HWT CPUs
PR #1155: Cleanup qualifier detection for mapping policy
PR #1154: Increase speed of rank computation
   - Remove spurious break in loop
PR #1153: Ensure the direct-multi test ranks-by slot
PR #1151: Revamp map-by NUMA support
PR #1150: Tranlate legacy OMPI params
PR #1147: Updates
 - Minor infrastructure and example cleanup
 - Allow external caller to set prte_tool_basename
 - Restore support for map/rank/bind by NUMA domains
 - Standardize on use of HWLOC_OBJ_NUMANODE
 - Activate the colocation code and debug it
 - Update man pages
PR #1145: delete use of PMIX_CHECK_BROKEN_QSORT refs
PR #1137: Protect against PMIx v4.1.1-only definitions
PR #1135: Final update v2.0.1 for rc3
 - Show MCA base params in prte_info
 - Silence variety of warnings
 - Start enabling picky compilers when requested
 - Remove unused variables in BMG component
 - Increase verbosity of detected error
 - Remove broken qsort support
 - Add a legacy example
 - Don't specify parent ID from tool
PR #1104: Update v2.0.1 for rc3
 - Silence Coverity warnings
 - Update portable platform file
 - Only order DVM termination once
 - Abort configure if gcc is <= 4.8.1
 - Fix segmentation fault caused by wrong size for vpids buffer
PR #1097: Update v2.0.1 for rc2
 - Update NEWS and VERSION
 - Circulate and report local completion status for collectives
 - Don't require pandoc to make a tarball
 - Do not register xml topos
 - Remove the Markdown source files for PRRTE man pages
PR #1089:
 - Cleanup timeout support to differentiate spawn vs execution time limits
PR #1086:
 - Use the info list array feature to construct registration data
 - Fix path check for pandoc
PR #1083:
 - Add MCA param to control device distance computation
 - Correctly report --output-directory and --output-filename as deprecated
 - Check for libevent minimum 2.0.21
 - Properly handle generic "orte" MCA params
 - Cleanups associated with IOF operations
 - Avoid use of MCA params for singleton and report-uri
 - autogen.pl: ignore all excluded components
 - Don't allow PRRTE IOF to block during delivery
 - Update ignores to include Coverity working directory
 - Ensure we hold the IOF data until the PMIx library is done with it
 - Add an "unlock" call to balance the code
 - Use correct ninfo when passing into APIs
 - Don't locally output stdout/err if prterun has a parent


2.0.0: 29 July 2021
------------------------------------
This is the initial production release of the PRRTE software system. It
contains full support of the PMIx v4.1 Standard plus extensions as provided
by OpenPMIx v4.1. It therefore requires support from that release or above.

Delineating all the changes from the original v1.0.0 tarball would be
overwhelming. The original tarball was provided solely as a means for
packagers to prepare for the eventual production version and was not
intended for general use. As expected, the code base has been extensively
revised/rewritten since that time.

Further changes shall be tracked here in follow-on releases.


1.0.0: 17 July, 2019
--------------------

Initial public release.
