As I've said in a few talks, the best way to improve your systems is by first notdoing "dumb things". I don't mean you or your development staff is "dumb", it'seasy to overlook the implications of these types of decisions and not realize howbad they are for maintainability let alone scaling. As a consultant I see this stuffall of the time and I have yet to ever see it work out well for anyone.Images, files, and binary dataYour database supports BLOBs so it must be a good idea to shove your files in thereright? No it isn't! Hell it isn't even very convenient to use with many DB languagebindings...

I have been playing around with Hybrid Word Aligned Bitmaps for a few weeks now, and they turn out to be a rather remarkable data structure.  I believe that they are utilized extensively in modern column oriented databases such as Vertica and MonetDB.Essentially HWABs are a data structure that allows you to represent a sparse bitmap (series of 0's and 1's) really efficiently in memory.  The key trick here is the use of run length encoding to compress the bitmap into fewer bits while still allowing for lightening fast operations.  They key operation from my perspective ...

After having started to build my own internet search engine as described in a previous blog post, I now have read some papers and books about web search engine architecture and information retrieval to complete my hobby project. Here is a list of papers and books that I highly recommend to anybody who is interested in this topic:1. Google: data structures and algorithms by Petteri Huuhka2. The Anatomy of a Large-Scale Hypertextual Web Search Engine by the Google founders Sergey Brin and Lawrence Page3. Introduction to Information Retrieval by Christopher D. M...

Mysql is a database, and to gain access to the data within C++ you will need to be able to “talk” to the database via queries (just like on the mysql command line interface e.g. select * from tablename), the connection process is very similar to the command line interface you will need to supply connection details as in hostname (localhost normally), username, password, database to use and also there are other details that you can pass e.g port number more information can be gained from the MYSQL API pagesTo start with I created a struct that will hold the username, host etc detail...

An important aspect to keep in mind with databases is the cost of cache reheating after a server restart. Consider the following diagram which shows several cache servers (e.g., memcached) in front of a database server.This sort of setup is common and can work quite well when appropriate; it removes read load from the database and allows more RAM to be utilized for scaling (when the database doesn’t scale horizontally). But what happens if all the cache servers restart at the same time, say, on a power glitch in a data center?We then have a cache reheating scenario. After the bounce the...

Whamcloud, the startup created in July 2010 to continue development of the open source Lustre supercomputer file system, has secured a $2.1m contract from OpenSFS to spruce it up with new features and functions.Lustre – used on about 60 per cent of the largest supercomputers in the world – is a parallel clustered file system designed for both supporting petabytes of files and giving high-speed access to the data stored on the file system. Lustre was created by Peter Braam when he was a researcher at Carnegie Mellon University, and was commercialized when he created Clus...

Database is one of the inevitable parts of a software application these days. It does not matter at all whether it is web or desktop, client server or peer to peer, enterprise or individual business, database is working at backend. Similarly, whether it is healthcare of finance, leasing or retail, mailing application or controlling spaceship, behind the scene a database is always in action.Moreover, as the complexity of application increases the need of stronger and secure database emerges. In the same way, for the applications with high frequency of transactions (e.g. banking or finance appli...