Warning: some parts of this answer might be unpleasant.
To answer the specific question of how your changes alter the characteristics of MD5, we must first restate what the MD5 security characteristics are. MD5 is a cryptographic hash function, so it is supposed to be resistant to collisions, preimages and second preimages. MD5 is not good at resisting collisions, since efficient methods for building collisions have been discovered since 2004. My own code (an implementation of Klíma's attack) produces, on average, one collision every 14 seconds, when it runs on a 2.4 GHz Intel Core2 CPU. As far as we know, MD5 seems to be strong against preimages and second preimages (a theoretical attack with cost 2123.4 has been described, and that's better than the generic attack of cost 2128, but not much better).
How do your variants fare ? Although this depends a lot on the precise modifications you intend, we can say the following:
Changing the round constants, replacing the sine with a cosine, should have no influence whatsoever on the security. None of the published attacks exploits any special structure of the constants; and, indeed, the known collision attacks are differential cryptanalysis which is oblivious to actual constant values (at least for the attack's efficiency).
Changing the fixed IV should have no bearing either. When the first MD5 collision was publicized, the result was much decried because the researchers got the endianness wrong and thus computed the collision not on MD5, but on some other function which differs from MD5 precisely on the IV. Their method did not depend on specific IV characteristics, so a few days later they ran the code again, this time with the right IV, and produced the real first collision on MD5.
Altering the bitwise functions in the rounds will have an impact, since it will change the differential paths which collision attacks use. However, the functions that MD5 uses are not especially weak; there is no indication that any other choice would make the function stronger. There is, however, a good probability that other functions might make any MD5 variant substantially weaker.
To sum up, among the changes that you suggest, there are some changes which should not change security, and some others which have a rather high probability of decreasing security quite a bit -- and beginning with a function which is already broken. That's an achievement, but not a very positive one.
It seems that you want to use your MD5 variant to generate randomness. It must be said that:
Using a hash function to build a PRNG yields poor performance. Hash functions like MD5 are very good at processing a lot of input data -- but for a PRNG, you want to spew a lot of output data, and hash functions usually suck at it. A properly optimized MD5 function, used to hash, e.g., successive values of a counter, might get you 100 MB/s worth of pseudo-random data, which is not bad, but any decent AES implementation on the same CPU will be almost twice faster.
Using a hash function to build a PRNG yields poor security. To get proper randomness out of a hash function, you need to believe that the function is a random oracle. "Believe" is the right word: we already know that MD5 (or SHA-1 or SHA-256) is not a random oracle (the length extension attack is enough to show it, and it can even be proven that it is not ultimately possible, in a very mathematical way, for a concrete function to be a random oracle). To make a sturdy PRNG from a hash function, it is best to use a more elaborate (but more expensive) construction, namely HMAC_DRBG.
Regardless of how you extend an initial random seed into a long sequence of pseudo-random bits, obtaining that initial seed is a harder problem. Many have failed.
Considering the flimsy and tenuous nature of the security properties of a hash function on which a hash-based PRNG relies, altering that hash function without any rational reason does not look like the best way to achieve security. It shall be noted that being a "perfect" hash function, with rock solid resistance against collisions and all kinds of preimages, does not make it good for a PRNG job. Being appropriate at being integrated in a PRNG is another security characteristic, that given hash functions may or may not exhibit; and it has been much less studied than resistance to collisions.
Your methodology sounds, at best, dubious. Things look as if you were throwing arbitrary tweaks at MD5, without any justification or rationality. This is more ritualistic than scientific. I have nothing against religions; however, while theosophical reasoning might give you good insights into Good and Evil, it is known to be inefficient at thwarting evildoers on an immediate basis. Jesus ensured redemption for us all (at least so goes the theory, according to the pope), but he still got nailed to the cross and died.
To sum up, your variants are unlikely to increase the security of MD5, while they are likely to decrease it; and the baseline is that MD5 is not strong to begin with; and even if it was strong as a hash function, it would not necessarily be strong as a PRNG.
Your are basically taking the problem in the totally wrong direction; in fact, in several totally wrong directions simultaneously. Before even thinking about assessing the appropriateness of any given algorithm for a job, you should first define that job with precision. If the job is generating randomness (to be precise, generating bytes which appear random, aka "unpredictable" for outsiders -- a computer being the ultimate deterministic machine, it cannot be really random), then using a hash function is not the smartest idea ever. Even more using a hash function of questionable repute like MD5. And twisting the function internals in the hope of making the function "better" is akin to performing brain surgery with a soldering iron: at least, it will make the failure indisputably spectacular.
If you need randomness, use what your operating system provides (/dev/urandom
, CryptGenRandom()
, os.urandom
, java.security.SecureRandom
... depending on your OS and programming environment). The OS is better at it than you. Just let it do its job.