That should be the case since 2 days or so. Olas relay actually wasn't deleting before. It could be that you need to clear cache on olas so that it's gone from your app.