Google supports different ways that allows you to decide outside of crawling and indexing of the URLs. If you might want to block some files, directories, and even your total web page from Google Research, consider our guideline about strategies to avoid written content from showing up in search engine https://zakabet.com/