Without a proper cryptoanalysis there's not much to say definitely, but this sounds a lot of alarm bells.
First, the key exchange algorithm. The secret is never sent in the clear, which is good, but it is send via the network, which is bad. There are ways for both parties to share a secret that is never sent via the network, in any way. See e.g. Diffie-Hellman. Or it's version using Elliptic curves. Good thing they use an ephemeral key, but do they have any protection against MITM attacks?
Second, the communications key. It appears it's generated by client. And is dependent on the client's pseudo-random number generator. Computers are bad at generating random numbers, and javascript (having noharder access to the OS which can gather some entropy) is even worse. So, the key to communications depends only on one, possibly poor pseudo-random number generator. Bad, it could depend on both client and server.
Third, the xAES. While it's possible that they've made some improvements to it, it's not probable. Much more probable is that they've slightly disturbed the needed interactions between the phases, making the outcome much less secure. There are many such attempts in the wild, and most of them are considerably worse than original (a.k.a. crackable immediately).
Fourth, the xAES key length. This one merits it's own paragraph. While it is generally true that more is better for the key size, the 8192 key size for this sounds unreasonable. Such key sizes are common for the asymmetrical algorithms, which need much longer key sizes (see e.g. https://crypto.stackexchange.com/questions/6236/why-does-the-recommended-key-size-between-symmetric-and-asymmetric-encryption-di) (due to the fact that the public and private keys are interconnected), not for symmetric ones. Having a key that is more than 10x longer than commonly used keys? Sounds fishy. Or like Let's make the folks believe we use correct key length values, because the number 8192 shows up often as keysize. Only it's not that keysize.
Overall: No concrete proof, but I'd consider it snake-oil.
After question update:
I have no knowledge of US Patent System and I have no desire to learn anything (I'm not from US and have yet to see when any patent did any good). I'm also not a cryptography expert, although I do know more than the average guy, having even some academic exposure and sort of a hobby.
Every serious cryptography book/course/whatever warns you from proprietary, unreviewed crypto algorithms, and for a good reason: It's very easy to make small mistakes that make the system entirely insecure. Only way to ensure it's more secure then the next one is to have a lot of independent review, and a lot of cryptographers trying to break it. If there's a lot of publications that tried to think of something for that algorithm, but with no real success, than we can be pretty sure this algorithm is secure. the xAES algorithm as far as you say has no such review. This does not say it's not secure. It does say that no one bothered to check it's secure. It might be as secure as AES, it may be more secure, it may be less secure than a CAESAR cipher, it may have a backdoor that allows to read everything if you know it. While there are unreviewed algorithms that are secure (AES was an unreviewed algorithm at some point in time), the odds are pretty much against it.
The fact that we know nothing of the authors of it doesn't help. Of course there's no guarantee that a known name will come up with a good algorithms, the chances are bigger, because those guys know about why those thousands of algorithms were not secure. From someone with no publication record? From someone anonymous? It could be good, but it's just a wild guess.
TL;DR: Without having the algorithm and some proficient guys spend some time cracking it, it's a wild guess. This is security, we don't believe in wild guesses.
And we get to the key length again. I can't add anything new to it. Except that it gets better - previously they claimed a key of 8kbit, now 16kbit. Funny, a few days and the maximum used key is doubled? Is this a fixed algorithm, or are they just updating it as we speak? Again, the key of that size (whether 8- or 16- kbit) is just unreasonable for symmetric algorithm. To compare: My browser has a lot of security certificates installed. These are from various CAs. Most of those have a 2048 key. 4x smaller than the first figure for xAES key size. And we are talking here about asymmetric keys, which tend to be at least 10 times larger then the corresponding symmetric keys (xAES) is a symmetric key. The tell you: the community believes reasonable size is x. We tell you we have 32x and show no reason why we believe x is not enough. And now we have 64x with even less reason.
Either 256 bit keys are secure, and having 32 times larger keys is just a waste of resources, or they have a reason to believe it's not, in which case I'd like to see that reason. Larger keysize won't get us far with quantum cryptography, it's an entirely different beast (imagine a machine that tries all keys at once, sort of a multitude of parallel universes with the correct one becoming reality).
Finally, let's return to their quotes and analyse it a bit:
xAES is a expenontial difficult level to break from base of AES.
If you take 2 times larger key, and assume the algorithm is secure, you have 2 times harder algorithm. So yes, if you have exponentially larger key, you have exponentially harder cipher. Sort of makes sense. However I'd like to see the proof of the assume the algorithm is secure
part.
AES has limit > of 265bits keysize, where xAES has been developed to overcome all AES limitation.
I assume the 265 bits is a typo, AES uses 128, 192 or 256 bit keys.
For xAES we can have a size of 16kbit (~264 times AES),
Explain to me how 16384 is 264 TIMES more than 256? 16384/256=64. The key is 64 times larger. The keyspace (number of possible keys) is 264 times larger. This is confusing terms.
which will be extremely difficult even if one uses quantum computer to break it.
See above.
xAES (extended AES) is our patented encryption algorithm, based on AES, but has bigger key-size (maximum 8192 bits), much complexer & longer polynomial (in step MixColumn), using Gray-Sbox or Sbox flexibly (in step SubBytes),
Not much to add to Thomas Pornin's answer. But: What is a much more complex polynomial? Polynomials are simple. Longer polynomials are just longer, not more complex. You could say that 100000 is more complex than 100.
and some more security modifications (salt, key-expansion, etc...).
Pretty easy to make a security modification that reduces security and very hard to have a security modification that increases security. There are a lot of very subtle interconnections. You can make a gear in a machine run twice as fast, but it will then wreck havoc in a machine where all other gears run at previous speed.
Salt: Random, usually not secret, data used for one-way function ("hashes"). See https://en.wikipedia.org/wiki/Salt_(cryptography). Could be understood as padding, which is adding (random or deterministic) data to either fill blocks or increase 'randomness' of message. Not part of a symmetric cipher, but of it's way of use.
key-expansion: a process of having a smaller key and producing a larger key or key set from it, with various properties as needed. The process of generating keys for subsequent rounds of symmetric cipher is sometimes called that (as we are expanding the key into several subkeys used in rounds), so AES already has this.
Oh look, we stuff in some fancy crypto-jargon. Too bad from a different context.
It looks that their CEO has no idea what he's talking about. And he sounds that more and more with each answer. No gaining points here.
Let me quote Phill Zimmermann, the guy that wrote PGP:
Anyone who thinks they have devised an unbreakable encryption scheme either is an incredibly rare genius or is naive and inexperienced. Unfortunately, I sometimes have to deal with would-be cryptographers who want to make "improvements" to PGP by adding encryption algorithms of their own design.
I remember a conversation in 1991 with Brian Snow, a highly placed senior cryptographer with the NSA. He said he would never trust an encryption algorithm designed by someone who had not "earned their bones" by first spending a lot of time cracking codes. That made a lot of sense. I observed that practically no one in the commercial world of cryptography qualifies under this criterion. "Yes," he said with a self-assured smile, "And that makes our job at NSA so much easier." A chilling thought. I didn't qualify either.
https://www.philzimmermann.com/EN/essays/SnakeOil.html
Lets's see Bruce Schneier's checklist for snake oil: https://www.schneier.com/crypto-gram/archives/1999/0215.html#snakeoil. In my opinion points 1, 3, 4, 5 and 7 fit perfectly. That's 5 out of 9.