While I will answer your questions to the best of my abilities, please realize that any of the tools listed here require moderate to heavy configuration and a good understanding of systems security. There is no tool you can just turn on and forget about, and shoving a system full of tools which you don't fully understand will do nothing but decrease performance and increase attack surface area. Each of the questions you asked could have its own answer with many pages of resources, so I am trying to keep it simple. Please get yourself familiar with these tools before using them!
File integrity scanners that use a database of known good files, or create a list of every file ever seen and alert the user of a never before seen executable. The anti-virus companies probably have something like this but I doubt it's public.
There are many file integrity checkers available, such as AIDE. These integrity checkers create a protected list of hashes of all files on a system, so changes to any files will result in an alert. Updating the database is supposed to be done immediately after any software upgrade. If you want an integrity checker that only operates on executables and actively prevents modified executables from running, you should consider IMA, the Integrity Measurement Architecture, for Linux. This is a security feature built into the kernel which will verify any file before execution to ensure that the hash of the file (stored in an extended attribute) is valid. The root of trust goes down using a hash tree to a single trusted hash, usually stored in secure hardware such as a TPM. Note however that IMA necessarily breaks demand paging, which may impact performance for large executables.
Memory scanners that look for unusual memory contents based on a database of what a particular operating system and motherboard BIOS areas should have. It could also check if the chipset and IOMMU is configured in an unusual way.
Although I am not aware of anything like this which operates at runtime, you can use the CHIPSEC framework. CHIPSEC is a collection of utilities which verify various firmware attributes to ensure the firmware has configured the system correctly for maximum security. Useful modules and common vulnerabilities are listed on their wiki. The framework is primarily designed to scan for firmware misconfigurations which make it possible to overwrite the firmware at runtime.
I do not know of any utility which checks to make sure the IOMMU is configured correctly. Generally, you can ensure it is working by seeing if your kernel log is providing information about DMAR, an ACPI table provided by the BIOS that gives the IOMMU information about the memory ranges to protect. I believe tboot does do some basic sanity checks in the process of implementing DRTM. Note that there may be ways to bypass the IOMMU by exploiting kernel vulnerabilities, and an IOMMU will not properly isolate the system if x2APIC or Interrupt Remapping are not supported, or if ATS (Address Translation Services) is supported and enabled on any PCIe device.
Hardware security is complicated and frankly a little scary. One paper explains how to switch the IOMMU to a mode which is typically considered more secure, yet which allows a GPU to perform malicious attacks on system memory. Intel's datasheets for each individual processor model can stretch to many thousands of pages of dense technical information (MSRs, MMIO, etc). Of that, a large percentage of information is kept intentionally hidden, available only in NDA-ridden confidential documents. Understanding an IOMMU's limitations is not a simple task.
Rootkit busters that scan for things such as abnormal hooks in the kernel and abnormal threads running.
This is difficult, because generally security software will be running under the kernel. A rootkit that has compromised the kernel will be able to modify the rootkit scanner at will. There are a number of kernel modules which attempt to scan for abnormal hooks (such as a hijacked syscall table), but they are often trivial to defeat and tend to rely on security through obscurity. There are a few hypervisor-based solutions, but they are often rather experimental or platform-specific.
Resident data scanners that scan for unusual things in memory such as encrypted data, and network traffic that contains executable binary code or unusual encrypted traffic. It could also communicate with a separate network monitoring computer to check for any hidden by rootkit network traffic that is showing up on the monitoring computer and causing a discrepancy.
This seems to be the same as the previous question about "rootkit busters". Scanning the memory would not be possible to do reliably if there is a rootkit on the system, and even if there are not, this tends to rely on heuristic analysis. Unlike a static file on the computer, memory can be constantly changing, self-editing, and highly obfuscated. For example, functions can be kept encrypted and decrypted right before execution, before being overwritten. Software can execute using x86's turing-complete page fault handler. Threaded code (unrelated to multithreading!) can be used to hide the intention of code, as it is essentially intentional ROP, executing gadgets out-of-order.
Intrusion detection systems, especially anomaly-based, that monitor network traffic going in and out of a network or host (running on a dedicated computer).
This is called an NIDS, or Network Intrusion Detection System. They often use a highly configurable ruleset which may or may not be free of cost. One of the most popular NIDS tools is Snort, which has both free, community rulesets as well as commercial, professionally-developed rulesets. If you get familiar with the syntax, you can also write your own anomaly-detection rules. Snort can be a bit resource heavy and has a large attack surface area, which makes it ideal to run on physical firewalls, as you mentioned. It can send its logs to other servers, remotely.
Any primarily heuristic threat detection programs and anything that you have in mind that is along these lines!
If the detection system runs on the host (as opposed to over the network as above), it is an HIDS, or Host Intrusion Detection System. HIDS software often runs alongside integrity checkers, or may have their own built in. There are many viable options for an HIDS, such as OSSEC. Note that these are often heavy on resources and are not particularly effective unless you are alert and maintain it correctly. There is no "set and forget" HIDS software that just works out of the box. It will need to be custom-tailored to your system if you want it to provide any improvement in security.
