In order to trigger parent page, in Apex 5.0, when user close the modal without to click any button. First, you will need to define the following Javascript code in your parent page.
function customEvent(event, data){
apex.event.trigger(document, event, data);
}
Then, in the modal page, you should set the following code to the attributes section
close: function() { customEvent('customDialogClose', {modalPageId: 'MODAL_CLOSE_FIXED'});}
Finally, in parent page, you need to create a custom dynamic action and set the attributes as below.