I was listening to Tim Bray of Sun giving his keynote at QCon this week in San Francisco. Tim is the author of XML and a real technology visionary, specially in the area of storage and data management. With the surge in web applications with very high scalability needs (millions of users), a fundamental rethinking of data management is happening. Here are the main points:

• between the application code and the disk/cache, these are the traditional layers - Object/Relational mapping, SQL engine, and OS & File System. Between layers what gets transported are these - objects from Application code to O/R mapping, then SQL to the SQL engine, then normalized tuples to OS & File systems, and finally bytes to the cache/disk.

• These layers add latency and overhead and become huge bottlenecks for web applications catering to millions of users. Hence, some application code is directly manipulating large hash tables for key/value pairs. They claim this approach makes it 100 times faster than RDBMS.

• Application code can directly map to RDBMS, bypassing the O/R layer (which is hard). PHP does that, but its hideous and ugly.

• How about application code going directly to the OS and file system, bypassing the entire database layer? They use XML, JSON, plain text and media files.

• There is a project at Sun called Drizzle that is doing radical minimization of MySQL, calling it a lightweight SQL DB for the cloud and web. It aims to throw away large chunks of code from MySQL.

• Experiements are going on with "column-oriented" database. They have huge number of columns and SQL-like queries for speed.

• Google Application Engine goes directly to storage via Google Big Table (persistent layer, but not SQL). This is like column-oriented table.

• Document-oriented databases are appearing. CouchDB and Amazon's SimpleDB are examples.

• Other interesting approaches include use of AtomPub, where application code directly interacts with the web space via HTTP. It's nimble and can get faster performance. Google uses that in its online stuff. Rumor has it that Microsoft's recently announced cloud OS Azure uses that also.

• REST is a good architecture style as evidenced by Amazon's successful deployment.

Bottom line - Web application designs are not CPU-limited. They are persistence-limited and I/O-limited. Therefore, new innovations are
coming up to enable such applications handle data efficiently.

I had a chance to give a talk about "Enterprise RIA Development Using Curl" at the Software Development Best Practices conference in Boston last month. Presentations at this conference are organized into 90-minute "classes." The length of these classes is a challenge but also an opportunity to tell the Curl story more completely than is possible in your typical conference presentation. In case you're curious about what gets covered in a 90-minute talk about Curl for enterprise applications, I've attached a PDF file of the presentation to this blog post.

The talk itself went well and the audience asked good questions. Afterward, I had a chance to attend a few other presentations. I concluded that the SD Best Practices team really vet their speakers much more carefully than average: the quality of the presentations that I attended was uniformly excellent, and far above what I have come to expect at a typical conference.

Curl was also invited to enter a team into the conference's Developer Bowl, a trivia contest about computer-related topics conducted in the style of the College Bowl. The other teams came from Cisco, IBM, and Nokia. In the first round, our Curl team, consisting of Christopher Barber, William Bardwell, Tom Lyons, and yours truly, triumphed over IBM. In the final round we were edged by Cisco on the final tie-breaking question.