An example is given below to how to create a modal dialog window in Oracle Forms for asking user input and return the value to any text item.
The following is the screen shot of this demo and this form could be downloaded from the following link: Custom_diag.FMB

This form is containing the following objects:

You can see in above screen shot of object navigator that there are two data blocks, one is the Main and another is DIAG_BLOCK for dialog window purpose and also one more canvas FOR_MODAL_WIN created for modal window MODAL_DIALOG.
There are some special property settings for FOR_MODAL_WIN canvas and MODAL_DIALOG window object. Below are the screen shots of both object properties:
For Canvas:

For Window:

On clicking of the buttons just setting focus to the blocks by Go_Block statement, which you can see in the downloaded form Custom_Diag.fmb