First step is to create library that will contain JavaScript that will be executed when button is clicked:

OnRibbonButtonClicked = function(grid){
    //put your logic here

    //code that refreshes grid
    grid.refresh();
}

Second step is to customize ribbon (I use for it Ribbon Workbench). As a sample I took contact entity. Select Subgrid Option in Ribbon Workbench:

Create new button:

Create new command and configure action of created command. Key point here is to pass grid as a parameter of handler:

Click OK and define command for button:

Publish solution:

Recheck that it works: