Agency Social Worker and County Social Worker are required to visit your home to verify the health and well being of the child, and in the case of the agency worker, also ensure that foster parents have the support needed to be successful.
Foster parents cannot deny or refuse entry to the home. That is not only unacceptable but potentially a violation of regulation because social workers must be able to confirm that children are fine and in a nurturing environment.
Foster parents may coordinate the "announced" visits so that these occur at a time that is convenient for all parties. However, agency and/or county worker are also required to conduct "unannounced" visits, meaning that they should show up without prior warning to the foster parent's home.
Do not take this personal. Develop a cordial and mutually respectful partnership. Ultimately, everyone is interested in the well being of the child. The fact that social workers visit your home is not a sign of distrust or that they don't value the foster parents. Instead, they are merely complying with the legal requirements of ensuring that children are safe.