Double encryption with home brew algorithm

Question

Everyone says that home brew encryption is not a good idea because one day the algorithm will be known to attacker which sounds reasonable. So that seems it's main problem, on the other hand known algorithms are known to everyone and are most-likely safe to use. Would it make sense to some paranoid developer to create a scheme like that AES(HomeBrew(openText)) where HomeBrew will either provide an encryption or some other kind of transformation to make it harder for the attacker as they will have to guess those things even if they manage to break AES somehow (provided they do not have the algorithm). So won't we get best of both approaches in this case?

Standard encryption that is proven to be good
Unknown algorithm "just in case" that even in case of leaking will not provide much to attacker unless they know how to break AES

Seems like this is actually a very good point: if AES is broken then you should assume that your whole encryption scheme is broken instead of relying on unverified scheme which is probably a question of "when", not "if". So according to this logic there is just no point in adding that layer because 1. If AES is broken your are screwed anyway 2. If AES is not broken: no need in another part. Though perhaps depending on the case it might help to postpone the attacker e.g. by a week which can be a good thing if your transmitted data is valid for less than a week. — Ilya Chernomordik, Jul 18 '15 at 22:12
Moreover, aside from security considerations, it's worth noting that AES is also verified for *correctness*, which guarantees you that you can decrypt everything that you have encrypted. In your HomeBrew solution you might find yourself unable to decrypt your own data because of bug in your coder/decoder. — el.pescado - нет войне, Jul 19 '15 at 08:03
This related discussion may contain relevant information: [can multiple encryption of data with multiple keys increase the security?](https://security.stackexchange.com/questions/87053/can-multiple-encryption-of-data-with-multiple-keys-increase-the-security). It does not deal with the home brew algorithm specificities, but covers security implication of cascading several crypto algorithms with different keys. — WhiteWinterWolf, Jul 20 '15 at 13:21

score 12 · Accepted Answer · edited Mar 17 '17 at 10:46

12

Kerckhoffs's principle states:

A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

Therefore if AES is broken, your homebrew algorithm is likely to be much easier and would have been broken already.

I realise part of your original question stated:

provided they do not have the algorithm

However, as this violates Kerckhoffs's principle your home brew approach is in itself flawed and insecure.

The main reason this is a problem is that it is unquantifiable how much more security it adds to your system. I refer you to Thomas Pornin's answer to a similar question here:

An additional twist is that algorithm obscurity can harm security. What I explain above is that obscurity cannot be trusted for security: it might increase security, but not by much (and you cannot really know "how much"). It turns out that it can also decrease security. The problem is the following: it is very hard to make a secure cryptographic algorithm. The only known method is to publish the algorithm and wait for the collective wisdom of cryptographers around the world to gnaw at it and reach a conclusion which can be expressed as either "can be broken that way" or "apparently robust". An algorithm is declared "good" only if it resisted the onslaught of dozens or hundreds of competent cryptographers for at least three or four years.

So therefore you should use another algorithm if you are worried that AES may soon be broken (that or don't use AES at all). I'm assuming the rest of your system isn't 100% secure - so instead of wasting time, effort, energy and resource on creating a home brew algorithm this effort should be spent elsewhere, where it can be quantifiable that it is actually increasing security. All code has a cost, and it is not just the salary of the junior developer that is creating it. It needs to be maintained and understood by everybody - and each extra person that learns how it works is an extra avenue for the algorithm to be leaked. Don't waste your time.

edited Mar 17 '17 at 10:46

Community

1

answered Jul 18 '15 at 18:13

SilverlightFox

33,698
6
69
185

You are saying the approach is insecure, but the whole point is that it is covered by a secure approach, so it does not make it worse, but can possible make it better if you are lucky to keep the algorithm in secret, so it makes life of an attacker harder even a "little bit" compared to AES and can give you some time, isn't it? – Ilya Chernomordik Jul 18 '15 at 18:59
1

You're adding extra complexity, and complexity doesn't sit well with security. Also if AES is broken you can never be sure that your algorithm is secret. If that was ever transmitted by AES then it could be known. – SilverlightFox Jul 18 '15 at 19:04
But then no one got a chance to crack it until AES is broken in the first place, and it can still buy you some time? And more importantly - can it make it worse somehow, so if we apply AES on top, does it mean it is at least as good as AES? – Ilya Chernomordik Jul 18 '15 at 19:09
3

Why not use another established algorithm rather than home brew? – SilverlightFox Jul 18 '15 at 19:26
7

`AES(k_1, HB(k_2, pt))` can *easily* be completely broken if `HB` is designed by an amateur. For example, if `HB` uses loop bounds or lookup tables based on the plaintext, it can trivially leak enough information for an attacker to recover it *without* having to break AES. Leave cryptography to professional cryptographers. – Stephen Touset Jul 18 '15 at 20:49
Thanks for extended version, I actually asked this more as a theoretical question with AES as an example, so I am not too worried about it being broken. By home brew I did not only mean only full blown algorithm, but some simple transformation that is easy to implement, but provided it's not known to attacker yet, it will increase his cost (quite unquantifiably though as you say). – Ilya Chernomordik Jul 18 '15 at 23:03
But it seems that being "unquantifiable" is the best possible answer for the question :) – Ilya Chernomordik Jul 18 '15 at 23:06
@StephenTouset I don't really understand that. Let's suppose that HB totally sucks, and is the equivalent of ROT-13 in strength. Let's also assume k_2 is completely independent of k_1. Don't you still get the full protection of AES since AES is wrapping everything from the home-brewed thing? – Patrick M Jul 19 '15 at 03:15
1

@PatrickM In addition to probably not providing much extra security, you open yourself up to a whole variety of side channel attacks that were (hopefully) already considered and mitigated in the standardized encryption you're using. – Chris Hayes Jul 19 '15 at 03:50
If you really want/need to mitigate the risk that $goodcryptoalgorithm will be broken, chain $goodcryptoalgorithm2 with its own key. – Deduplicator Jul 19 '15 at 12:49
@PatrickM Read up about side channel attacks. A poorly-implemented transformation can leak information about the plaintext, where the attacker doesn't even need to look at the ciphertext. For instance, if you use the plaintext as a loop bound, the difference in time it takes to perform the transformation on different plaintexts can be used by an attacker to learn the plaintext. – Stephen Touset Jul 20 '15 at 19:38

BoppreH · Answer 2 · 2018-09-02T16:22:21.083

6

It could work, and would be security-in-depth instead of security-by-obscurity, but there are a few ways to mess this up catastrophically:

Using a HomeBrew algorithm that is vulnerable to side-channel attacks. Now the attacker can do a simple timing or cache analysis, for example, and bypass the AES part entirely. Are you that secure in your implementation?
Reusing material between the algorithms. If you reuse keys and nonces, for example AES(HomeBrew(openText, "secret"), "secret"), it might be vulnerable to cryptanalysis. This is a distant threat, but necessary if you want to keep the same security standard of just AES(openText).
Using a brittle HomeBrew implementation. If it might break when the text contains nulls, or are above a certain size, or are below a certain size, or look like this, or contains valid unicode, or contains invalid encodings, etc. This is a programming concern, but opens your software to anything from denials of service to improper authentication or worse.

All in all, I find this a fun idea, but not a practical one. Maybe using something that is not homebrew would work better; there are safer ways to add depth to your security.

edited Sep 02 '18 at 16:22

answered Jul 18 '15 at 17:39

BoppreH

336
1
10

How can an attacker bypass the AES part completely? Provided I got your answer correctly, they can get the result of homebrew encryption they can as well get the open text itself I guess so it's too late to save anything. – Ilya Chernomordik Jul 18 '15 at 19:02
1

If `HomeBrew` leaks the plaintext through side channels (timing, cache, external requests, etc), there's nothing that `AES` or further ciphers can do to re-secure it. You don't need to see the ciphertext of the homebrew encryption, just its side-effects. Maybe doing `HomeBrew(AES(text))` fixes this, because it would only leak the AES ciphertext, but it's still risky. – BoppreH Jul 18 '15 at 19:42
1

Actually, if we do it the way you say HomeBrew(AES(text)) and suddenly this exposes AES in some way, we will probably be famous by breaking AES as we have essentially found a transformation making AES vulnerable, meaning HomeBrew(AES(text)) cannot be worse than AES(text), isn't it? :) (Provided we don't mess with key used for AES or with the open text after AES) – Ilya Chernomordik Jul 18 '15 at 22:20
1

@IlyaChernomordik: suppose that "HomeBrew" is actually AES but used in the decryption direction; this is a perfectly fine block cipher, and is as secure as normal AES (since any weakness in the decryption direction would still count as a weakness). However, in that case, AES(HomeBrew(x, k), k) = x for all plaintexts x and all keys k; i.e. no encryption at all. You cannot get weaker than that. Therefore, the combination of AES and HomeBrew _with the same key_ (or related keys) can be utterly weak even if AES is perfectly secure. – Thomas Pornin Jul 18 '15 at 22:45
That's why I wrote: provided we don't mess with key used for AES or the open text. Guess you have to be really "smart" to apply the reversed one :) Is it still possible then? I have asked a broader question for it here: http://security.stackexchange.com/questions/94338/can-applying-second-cipher-including-home-brew-reduce-security – Ilya Chernomordik Jul 18 '15 at 22:47
@IlyaChernomordik: as for side-channel attacks, they may expose the plaintext. In that sense, HomeBrew(AES(x)) is better than AES(HomeBrew(x)): if HomeBrew works on already encrypted data, then it cannot reveal the plaintext, however leaky it may be (as long as you take care to have unrelated keys, see previous comment). – Thomas Pornin Jul 18 '15 at 22:47

score 4 · Answer 3 · answered Jul 18 '15 at 11:21

By itself, your home-brew algorithm is a form of security through obscurity. However, when applied in combination with a known good encryption, your home-brew algorithm may be considered a defense in depth strategy. Quoting directly from Wikipedia,

A system may use security through obscurity as a defense in depth measure; while all known security vulnerabilities would be mitigated through other measures, public disclosure of products and versions in use makes them early targets for newly discovered vulnerabilities in those products and versions. An attacker's first step is usually information gathering; this step may be delayed by security through obscurity.

If a vulnerability in AES is publicly disclosed today, it would still take some time for an attacker to figure out how to crack your home-brew algorithm, giving you time to switch to another algorithm. As the saying goes, "don't put all your eggs in one basket".

For the really paranoid, you can adopt two layers of different standard encryption such as AES+DES. Having multiple layers of different encryption is not cost-free though. The trade-off is that your ciphertext becomes chunkier.

*"it would still take some time for an attacker to figure out how to crack your home-brew algorithm"* - There's no guarantee of that. You should see some of the homebrew algorithms that people have come up with.... — D.W., Jul 18 '15 at 23:20
Homebrew algorithms may mitigate AES's security. What if they accidentally reinvent the AES decryption algorithm? — PyRulez, Jul 19 '15 at 19:40

score 2 · Answer 4 · answered Jul 18 '15 at 10:31

Dieter Vandenbroeck wrote an article on when security through obscurity makes sense:

Usually, the principle of security through obscurity is just utterly wrong. The idea behind this principle is to have obscurity in your solution as the principal means of security. Of course, this is not a valid substitute for real security: once your obscurity schema is broken, your security would be broken. It might form a defense against automated scripts, but an attacker with sufficient determination will always be able to fight through this defense.

In your case you won't have an automated script and by the time AES can be cracked in an automated way, homebrew will probably be even more flawed and as easy to crack. So no I do not think homebrew would give you an additional security benefit compared to plain AES.

Yes, but is a home-brew algorithm classical security through obscurity? I mean, if you family name is Shneier, El Gamal, or KoreK you might in fact be on the safe side with such a scheme. — Konrad Gajewski, Jul 18 '15 at 11:08

Konrad Gajewski · Answer 5 · 2018-09-01T18:40:21.537

It does make sense, even if you use ROT13, since you will add a layer of security. AES(HomeBrew(openText)) will always be more secure than AES(openText)...
...unless this algorithm of yours adds some weakness to AES that will make cryptanalysis easier. Which I doubt.
But I'd personally stick with known algorithms and use something like AES(Threefish(Plaintext)).
However, where is the fun in that?
And how do you intend to keep your algorithm secure, if you intend to use it?

I generally wouldn't bother.

score 1 · Answer 6 · answered Jul 06 '22 at 08:19

I know this is an old question, but I don't agree much with the other answers. My opinion is that when you add a layer of obscurity on top of something that is proven to be secure, then the overall security is not lost, and some more security might be gained.

In particular, I would suggest that the custom algorithm should be applied last, after AES, that is: customEncryption(AES(plaintext)). The reason is that your custom encryption (which might turn out to be a simple obfuscation algorithm) is very likely to be a very poor algorithm which leaks information and is easy to analyze when applied to the plaintext. But if you apply a poor algorithm on random data, then it is going to be harder to analyze because (pseudo-)random data is supposed to have (almost) no statistical data to analyze. Also, if you apply your crappy algorithm last, then you can avoid the risk of leaking data via any side channels, because even if anything went wrong, you would leak data that is already encrypted with AES.

Whether all this is worth it or not is a different question though, and depends on your threat model. Remember that on one hand secret software is hard to distribute securely, and on the other hand "unbreakable" crypto could always be defeated by other indirect means (infecting the machine, rubberhose approaches, etc.). So don't think that your idea is the solution to all problems in cryptography.

score 1 · Answer 7 · answered Jul 20 '15 at 15:33

I agree with abligh.

Undertaking "Homebrew_Crypto(AES(plaintext))" will likely not add a great deal to the security of the system -- but what it WILL do -- even against a state-level attacker -- is simply slow them down. (It will slow down unsophisticated attackers a lot; for No Such Agencies it will force them to use a few more cycles of supercomputer time, which may get you brownie points in their "Enemies Of Our Beloved National Security State" list.)

One really funny variation of this would be :

"Homebrew_Crypto(Twofish_Or_Some_Other_Good_Crypto_Algorithm(AES(plaintext)))" -- at the risk of significantly slowing I/O, of course. Kind of like those Russian dolls that nest one inside the other.

Note that the latter option was available in Truecrypt and I believe in its successors like Veracrypt.

score 0 · Answer 8 · answered Jul 19 '15 at 13:02

There is another reason why this might help. Rather than doing AES(Homebrew(Plaintext)), let's assume you do Homebrew(AES(Plaintext)).

Let's also assume the worst case, and though you think you are crypto genius, your Homebrew algorithm is no more useful than ROT-13.

In this case, today you have a combined scheme which is no worse than AES alone (save that it might be a bit slower), but no better. If your Homebrew is better than ROT-13, who knows the combined scheme might be slightly better than AES alone.

However, let's now assume that some time in the future, AES gets broken. A zero day exploit becomes available, and an exploit is widely publicised. Script kiddies galore attempt to find AES encrypted material and decrypt it. Assuming they aren't targeting you specifically, they are unlikely to be able to decrypt your data using the hack, as they won't think to apply some other decryption first, and there are low-hanging fruit elsewhere. This at the very least gives you a little more time to change your underlying encryption to something that isn't broken.

Whether that's worth the speed and maintenance penalty is another question entirely.

score -1 · Answer 9 · answered Jul 18 '15 at 20:50

Standard encryption that is proven to be good
Unknown algorithm "just in case" that even in case of leaking will not provide much to attacker unless they know how to break AES

Except you now have TWO standards to maintain, and a weakness in any ONE of them will weaken the second, because two ciphers in tandem effectively produce a third cipher which is subject to each one's weaknesses. The homebrew is probably going to spoil it for everyone, at the end of the day. :)

If you have one cipher which is KNOWN to be stronger than the other, then effort would be better spent in ensuring that it is properly deployed.

Even established protocols with provable security should only be combined in certain cases; cascading unsuitable algorithms can result in underlying patterns emerging when the individual ciphers effectively "undo" each other's work, simply because their methodologies are counter productive.

It is far more likely that a homebrew cipher is going to weaken the result rather than strengthen it. In my opinion, it's best not to knit you own crypto.

This is one of the rare cases where a chain may be as strong as the strongest link, instead of the weakest one. You may be interested in [this question](http://security.stackexchange.com/q/87053/5405). — S.L. Barth, Jul 19 '15 at 09:23

Double encryption with home brew algorithm

9 Answers9

Linked