Notified rma sm - #28
Open
joe-explr wants to merge 19 commits into
Open
Conversation
Signed-off-by: Joseph Schuchart <joseph.schuchart@stonybrook.edu>
This commit adds notification support to the OSC SM component by implementing the put_with_notify, get_with_notify, rput_with_notify, and rget_with_notify functions. These functions perform the same operations as their non-notify counterparts but also increment notification counters after the data transfer completes. The changes include: - Added function pointer types for notify variants in osc.h - Added function prototypes in osc_sm.h - Implemented the notify functions in osc_sm_comm.c - Updated the module template to register the new functions - Removed TODO comments that have been addressed Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
put_with_notify get_with_notify Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
put_with_notify
get_with_notify
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
…for a single and multi rank window. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
…eph.antony18@gmail.com>
… <jajoseph.antony18@gmail.com>
MPI-5.1 names the error class for an invalid notification index MPI_ERR_RMA_NOTIFICATION. Rename the placeholder used by the notified RMA work to match the standard. The class was never registered with the error code subsystem, so MPI_Error_string() and MPI_Error_class() did not know about it; add the missing CONSTRUCT_ERRCODE()/OBJ_DESTRUCT() pair in errcode.c. Also fix the binding generator's ERROR_CLASSES list, where the entry was inserted without a trailing comma and so was silently concatenated with the following 'MPI_ERR_TYPE' element rather than added as a class of its own. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Add the two remaining notification-management procedures from MPI-5.1 section 12.6.1: MPI_WIN_SET_NUM_NOTIFY is a blocking, synchronizing collective that sets the number of notification counters attached at the calling MPI process to exactly num_notifications and resets all of them to zero. MPI_WIN_GET_NUM_NOTIFY is local and returns the number of counters attached at target_rank. Both are wired through the osc framework as new module entry points, so components that do not implement them return MPI_ERR_UNSUPPORTED_OPERATION rather than crashing. The osc/sm implementation carves a fixed per-rank counter region out of the shared segment at window creation, which is therefore the effective MPI_WIN_NOTIFICATION_NUM_UB; a request beyond that capacity is rejected with MPI_ERR_ARG. Each rank publishes its own attached count into the shared segment, so the collective needs only a barrier -- no counts have to be exchanged -- and MPI_WIN_GET_NUM_NOTIFY is a plain shared-memory read. Also add the missing put_notify/get_notify entries to interface_profile_sources, which were omitted when those two procedures were introduced. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Three correctness problems in the osc/sm notified communication path: 1. The notification counters were plain uint64_t but are incremented concurrently by remote origins with opal_atomic_add() and polled by the local rank. Type them opal_atomic_int64_t so that the reads in MPI_WIN_GET_NOTIFY_VALUE are atomic and cannot be hoisted out of a caller's polling loop. 2. The notification index was validated *after* the data movement, so an erroneous call had already overwritten the target window (or, for get, the origin buffer) by the time the error was returned. MPI-5.1 section 12.6.1 makes referencing an out-of-range counter erroneous at initiation, so hoist the check above ompi_datatype_sndrcv() in all four notified operations. The check is factored into a helper, which also fixes MPI_GET_NOTIFY returning OMPI_ERR_BAD_PARAM instead of MPI_ERR_RMA_NOTIFICATION. 3. The get paths used opal_atomic_rmb() before incrementing the target's counter. The notification tells the target that the get has read the window, so the constraint is load-before-store, which a load-load fence does not express; opal_atomic_add() is relaxed and adds no ordering of its own. Use a full opal_atomic_mb(). In MPI_WIN_GET_NOTIFY_VALUE the barrier was likewise placed before the counter load, where it ordered nothing; move it after so that it gives the acquire semantics the caller needs. MPI_WIN_RESET_NOTIFY_VALUE also gains the trailing barrier for the same reason. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.