A common procedure for search engines is to try load the robots.txt file before they start to browse the server, that is to index the site into the search engines database. This file's content can for example prevent the search engine from seeing the site.
A similar file is called sitemap.xml, where search engines can get a list of all the pages of the site that want to be taken into account.
Don't Create the robots.txt or sitemap.xml File Yourself – Let WordPress Create Them!
Often people go the the server with an SSH/SFTP-connection and create these files from an old habit. In WordPress's case this is wrong. You should let WordPress create the files on the fly. When search engines asks for the sites robots.txt-file, WordPress gives an answer to the search engine even though there is no robots.txt-file.
If you want to tailor the robots.txt-file you can do this by adding a function to the theme's functions.php file that is registered to the do_robotstxt event.
Example:
function example_disallow_directory() {
echo "User-agent: *" . PHP_EOL;
echo "Disallow: /forbidden/directory/" . PHP_EOL;
}
add_action( 'do_robotstxt', 'example_disallow_directory' );
WordPress's core does generate sitemap.xml-file on it's own since version 5.5 but you can replace default file with SEO plugins such as SEO Framework and Yoast SEO.
How to Disallow user-agents With robots.txt
By adding following example to robots.txt-file you could disallow googlebot user-agent from crawling example directory and it's contents:
User-agent: googlebot
Disallow: /example