en-GB language should be included by default, and that error message should never appear on your English website. About this topic you are right. However, I can't further investigate about your problem without knowing the URL of your website.
learn more link is a complete waste of time
For unexperienced users it is useful, because they follow the document step by step.
Experienced users (as probably you are), tend to skip important steps, so it results in a waste of time, like you pointed out.
Please be sure to set COM_FOXCONTACT_PARTIAL to 0
twice.
Admin side in administrator/language/en-GB/en-GB.com_foxcontact.ini and
Frontend side in language/en-GB/en-GB.com_foxcontact.ini
However, the URL of your website would be really useful in order to understand what's happening.