Rows, Columns and Fields can be deleted from the layout by clicking the trash/rubbish can/bin for the relevant field, column or row.
You can't delete a row or column unless it is empty.
In addition to the delete icon for each individual field, column or row there is a 'Clear Grid' icon at the top of the layout editing page. This will remove all rows, columns and fields in one go (with a warning first!).