First, some comments
I'll be frank here. A lot of your setup is poorly thought out or outright dangerous. For example, using a proxy after Tor (it's not TOR) can greatly decrease anonymity as you now have a centralized exit point (even with the use of rotating proxies). This is exacerbated by using a VPN, since now rather than having a protective chain of relays, you have two points, both controlled by individual entities, which can collaborate to fully deanonymize you. But let's look past that for now. Some comments about the assumptions...
- One home internet connection from an ISP that keeps a close eye on potential TOR usage. The connection is being shared by a couple other devices and people.
Being shared by multiple people is irrelevant, as TCP/IP fingerprinting is able to distinguish multiple devices coming from a single IP. If you need to reduce the chance of an ISP knowing you are using Tor, you may have to use pluggable transports with a bridge. These are designed to obfuscate the Tor protocol in various ways, though the use of a bridge is only intended if you are being blocked. It is not always sufficient for simply avoiding membership attribution.
- One solid VPN provider that resides in a non 14 eyes jurisdiction, doesn't have a record of collaborating with intelligence agencies, offers stable servers in a multitude of locations, employs a zero-logs policy running everything from RAM memory and doesn't even ask for account names or emails.
A VPN being in a non-SSEUR (14-eyes) jurisdiction can actually make anonymity worse, as now rather than going through small ISPs in a "bad place", you are going through the border of a "bad place", complete with all their IXP taps. This can be bad enough that some circuits that go through many topographically distant countries can even lead to complete deanonymization. Furthermore, even if the VPN claims to provide zero logs, their ISP and DC certainly does log, it would be unheard of not to. See this answer which provides an example. I strongly recommend you read it.
Payment was done through bitcoin.
I'm sure you're already aware of this, but bitcoins aren't anonymous. Ensure payment was done with initially anonymously obtained bitcoins.
- A standard whonix setup(gateway, workstation) running as virtual machines on the host using VirtualBox
VirtualBox is rather insecure, as are most hypervisors. It can be especially vulnerable to leaking graphics information when hardware acceleration for graphics is used. Whonix is best used with physical hardware isolation, so a simple 0day in the hypervisor will not bring down your whole anonymity system. Using hardware isolation will reduce the attack surface to that of the networking protocols, Tor control port, and SOCKS protocol.
- A list of 1000 socks5 proxies which may or may not have their activity monitored and logged by a potentially hostile 3rd party which will be used at browser-level to access a website
This is dangerous, far more dangerous than using plain Tor exit nodes. Chances are a large number, if not all, of these proxies are managed by a single entity. This makes it far more dangerous than Tor, where each exit tends to be operated by a different entity.
Furthermore, there is the risk of the proxies being too close to your guard or VPN topographically (or even on the same subnet!). Tor ensures that none of the three relays in any given circuit are too close, but this guarantee breaks down as soon as you use "extra" proxies. If your VPN or guard and one of the proxies is part of the same DC, it can completely deanonymize you. This goes against "anonymity 101 common sense", but is quite true. There's zero chance that there will be no extensive monitoring between the border of your country and your "safe, non-SSEUR country", but there is a non-zero chance that there will be no monitoring at the junction between two small ISPs in the same country.
In the case of you -> vpn -> node1 -> node2 -> node3 -> proxy -> website, The three nodes are guaranteed not to be too close, topographically. There is no way for Tor to guarantee that vpn and proxy are not right next to each other, blowing your anonymity. This is the same reason using Tor over Tor is a horrible idea.
You may want to read Measuring and mitigating AS-level adversaries against Tor, which explains this phenomenon.
Machine -> Home connection/Personal IP -> VPN IP -> TOR -> Firefox ESR with a Socks5 Proxy-> Website
You should absolutely not be using plain Firefox ESR, even with "privacy" extensions. You should only ever use Tor Browser, or you will be subject to completely accurate fingerprinting (for example, through audiocontext fingerprinting). There are more than a dozen extremely important features which Tor Browser provides which are lacking on Firefox ESR and cannot be implemented in addons, requiring patches to the source code as well (like Tor Browser's reduced timing granularity of javascript.now()). Tor Browser is far more than just Firefox and a few addons.
To answer your questions, though...
- Is there any way in which the ISP can know that we are on TOR and/or using Whoenix.
Yes, it can know you are using Tor. A VPN typically uses UDP or TCP for communication, and there is no padding. Tor on the other hand uses its own protocol, where data is sent in groups of cells of 514 bytes each (previously 512). This means that your ISP will see VPN traffic going through in bursts of 514 bytes, indicating use of Tor. There is likely also a way to determine that you are using Whonix with a high likelihood of Whonix makes any unique connection attempts at startup (such as update checks) that could be detected through a type of passive fingerprinting called website fingerprinting.
- Could 'website' ever find out what the originating IP address really was?
Potentially, if it was coordinating with an entity that could see both your proxies and your VPN. Other than that, no, not with your first assumption in place, assuming it also extends to 0days and not just backdoors.
- Is there any way to correlate what happens at the 'website' level with the originating IP address/connection
See previous answer. This question is basically the same as your second one.
- Would using a public internet connection instead make a big difference in the anonymity of the setup?
That depends on many factors. A public internet connection may have more fine-grained logging. It may be subject to attacks on the WiFi protocol. Unlike a hardwired home connection, a public wireless connection allows countless others in close range to determine your unique hardware (MAC address randomization is not enough), and potentially mount website fingerprinting attacks.
Recommendations
Rather than just criticizing this setup, I think some recommendations are in order.
- Use Whonix with physical isolation, or Tails.
- Use stock Tor Browser with the security slider set to high, without any other changes.
- Install open source firmware on your router such as OpenWrt.
- Do not use anything extra like a VPN or proxy.
- Ensure all your software is up to date, and keep track of disclosed vulnerabilities.
- Avoid WiFi because of fingerprinting attacks from a local adversary and insecure WiFi chips.
- And as always, be mindful of your OPSEC.
