Instructions for this level:
In this level the password again is in a txt file. This time it's not as straight forward as just viewing the source. You wouldn't even find the page by using a search engine.
The reason for that is because search bots have been excluded but where and heo? That's what you need to work out.
Here is some information that might help you in this level:
Web site owners use the /robots.txt file to give instructions about their website to web robots; this is called The Robots Exclusion Protocol. If a robot wants to vists a Web site URL, say https://www.hackthis.co.uk/levels/ Before it does so, it firsts checks for https://www.hackthis.co.uk/levels/robots.txt, and finds:
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any these pages on the site. In this case: https://www.hackthis.co.uk/levels/
Robots can ignore the /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention to the robots.txt file. The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use. So think about this when you try to complete Hackthis main level 7.
This site will help you construct a robot.txt file if you are not sure what you would need to include in it: