We’re at 5 weeks since we let people onto ChemSpider to crawl our web. The service is now enabling over 650 visitors per day on average in the past week to search the database and utilize the services. At present the database has grown to 10.6 million structures. However, as stated at the time of release we would limit the database to around 10 million structures solely for the purpose of testing.At present we have 800,000 molecules from 4 new contributors waiting to be added to the database after prediction of all associated properties and then de-duplicated with the existing content. We are in the process of converting over 10 million NEW structures from SMILES format. These will also be passed through all prediction algorithms, added to the database and then de-duplicated. There are a number of other databases to be delivered to ChemSpider for preparation in the next few days. With full disclosure you should be aware that ChemSpider is served up from two Dell servers. They host the transactions, the database, the web server, the webpage, our email system. They are also, in parallel, converting SMILES to connection tables (millions of them) and are predicting a series of properties on every structure (check ChemSpider..youâ€™ll see them all). Some of these properties take many seconds since they are complex calculations.Bottom lineâ€¦these servers are in dire need of air conditioning systems. They are running flat out. Our ability to provide fast searches, especially structure and substructure searches, while also being able to perform transaction-based predictions is already starting to fail. People are reporting performance issues so we have moved our predictions to the evening. It is clear we need a new server already (likely two) and much earlier than expected. I guess that’s what you call one of the struggles of success. I spent today reading about how one of the founders of YouTube kept extending his credit card bill to cover their technical costs (Time, January 2007). Well, we are not YouTube, this is not Silicon Valley and we’re not putting our families at risk. As already discussed on this blog we may need to seek sponsorship. As it is we made the painful decision today that we have to start some form of advertising. We will be judged on this. And we acknowledge it. But our intention is to stay faithful to the community to have the service remain free but offer the best services and throughput that we can. Only more computing power will allow this at this stage. We will stumble along for the next month with what we have but if the dataset grows to the expected 15-20 million structures we will need to expand our plastic boxes. Hopefully our users will understand.
The Increasing Demands on ChemSpider and Servers are not cheap