One fundamental division in shared-key (secret key) cryptography is between stream ciphers and block ciphers. Stream ciphers operate on individual bytes, processing input data (plaintext) as it is available, encrypting them and outputting encrypted bytes (ciphertext). On the other hand, block ciphers always process blocks of data in some predefined size (typically 16 or 8 bytes) at once, and output encrypted blocks. Because there is more data (a block) available to these algorithms, there is more maneuvering space for bit shuffling and mixing, making them somewhat easier to prove safe. This property also allows more flexibility to create different variations of block cipher algorithms. All this has resulted in block ciphers being more studied in recent years and definitely more numerous. In order to apply block ciphers on regular data (which is usually larger then a single block and even so often cannot be divided into regular sized blocks), additional chaining and padding algorithms must be used.

Almost all stream ciphers widely used today are "combiner type" ciphers, which operate by generating a stream of pseudo-random bytes (keystream) which depends exclusively on the given key, and which is combined with input data by using the XOR operation. This method is trivial to decode but introduces a significant weakness: if the same keystream is ever generated more than once (which usually boils down to: if any possible key is used more than once), the cipher is trivially broken at the XOR step. This key reuse is on of the major reasons why WiFi "WEP" encryption is useless.

The complete separation of keystream generation and data encryption (the XOR step) seems a bit inelegant so I have experimented a bit with designing stream ciphers which have these steps integrated. In particular, ciphers which maintain some internal state which is modified by the data being encrypted. Thus, generated keystreams are also plaintext-dependant and a few bytes random bytes (nonce) encrypted (and transmitted) at the beginning of the stream completely destroy any correlation between keystreams generated from the same key. The major practical advantage of this approach over simply using some well-known block cipher like AES is that it eliminates the need for block chaining and padding protocols.

I am not a cryptography expert, just an enthusiast, any none of the described ciphers have been reviewed by experts in the field. I have stated my assesments of some of the algorithms but until somone actually proves they are good (or bad), they should not be considered generally safe.

All algorithms currently experimented on can be downloaded here. The source compiles out of the box on FreeBSD, will probably require trivial or no changes to the Makefile to compile on Linux. Uses OpenSSL.

adlerscOne of the fastest ciphers here (~~35 MB/s) but probably completely unsafe. Uses adler32() function for mixing data, which has too little internal state (32 bits) to be of any use. Also, the function's algebraic properties probably make it unsuitable for this task.
md4sc, md5sc, aesscThese slow stream ciphers use the MD4, MD5 or AES algorithms as hash functions around which data mixing and keystream generation are constructed. The MD4 and MD5 variants are actually slower than the AES variant (2-2.5 MB/s vs 8 MB/s). I believe all these ciphers are Safe (with a capital S), even the MD4/MD5 ones because they are used in a way that doesn't give the potential attacker even one whole hashed block to use in analysis.
rc4vscThis algorithm uses RC4 as the central algorithm (note: RC4 itself isn't broken in WEP) but modifies it with a key and input data dependant "stream advance" (dry run) steps. This makes it at least as safe as a plain RC4 keystream cipher but for the added security depends very much on the initial nonce (I estimate that using it with decent security requires at least 16 bytes with good entropy to be encrypted and transmitted at the stream start). Performance is merely decent at ~~ 30 MB/s. The algorithm could be made to require a shorter nonce but at the expense of speed (which in the safest case comes down to ~~ 1 MB/s).
rc4v2scThis algorithm uses RC4 for its basis, modified by permutating the permutation array itself depending on the input plaintext. It is probably safe when used with a good (and not necessarily long - 8 bytes should be enough) nonce, but with mediocre performance of ~~ 11 MB/s.
rijscThis algorithm resembles the AES variant but with a reduced number of Rijndael rounds (2 rounds). Unfortunately the performance is again mediocre (~~ 13 MB/s) at a probably significant risk caused by severely reducing the base cipher rounds. This variant is probably a dead end.

Performance in the above table is given as an estimate on a 2.4 GHz Core 2 CPU.

It has been fun experimenting with these algorithms and while doing so I got some more ideas to try in the future. I am mostly disappointed by the performance I got from the algorithms - I would expect a good stream cipher to clock at least 50 MB/s on the given CPU, but none of the variants came close to this performance.

Again, these ciphers have not been reviewed! Use at your own risk!