Planet CouchDB

We just added Joyent Cloud to the list of Cloudant Data Layer locations.

Joyent Cloud is a high-performance environment for running apps and data layer services in the cloud. Backed by Joyent SmartOS - an operating system optimized for multi-tenant cloud environments - and the ZFS file system, Joyent Cloud enables Cloudant to take advantage of specific caching and data backup features in the ZFS file system and deliver outstanding data layer performance and reliability Aside from Cloudant, Joyent Cloud also powers a growing number of cloud platform and software services providers including Nodejitsu, StackMob, and GameSalad.

Learn more about Joyent Cloud

Create a Free Cloudant Data Layer running on Joyent Cloud

Getting started is easy. Just create a free Cloudant account, and select the Joyent Cloud option “Paloma” from the list of locations, as shown below.

Or Move Your Data Layer to Joyent Cloud

If you already have a Cloudant account and you want to move it to Joyent Cloud, follow these instructions:

1. Log into Cloudant
2. Choose the ‘Account’ tab
3. Change the placement of your cluster to ‘Paloma - Joyent, US Southwest’
4. Cloudant will notify you after your account has been relocated to Joyent Cloud.

April 12 is Iris Couch’s one year birthday. Today, we offer paid cloud CouchDB hosting plans.

How it Works

Our pricing plan has the details. It is relaxing and RESTful. Your log file is your receipt.

This month is completely free for everybody. May is the first month under the new policy.

• People who don’t accrue $5/month of usage don't owe anything for that month
• The vast majority of users don’t accrue $5/month in usage charges.
• Thus, by the transitive property of service pricing, most of you won’t pay anything
• Your account shows your historical usage since you signed up

Below, co-founders Jeff and Jason reflect on our milestone.

Jeff’s thoughts

I’m really proud of how dialed-in this service has become with Jason’s expertise. I’m excited about this launch and new features we have in development. Mostly, though, I’m really looking forward to not feeling so sad every month signing huge checks to pay our hosting bills.

We spent a lot of time mulling over how to price this so that beginners and tinkerers can still have a free sandbox, and I think what we’ve come up will protect that. Thus Iris Couch will remain both a platform and a tool to spread the word about how relaxing CouchDB is.

Jason’s thoughts

I am pleased with our first year. We have built technology I am proud of. Many people (Jeff in particular) have asked:

Why the delay for paid stuff? Aren't you hungry? Are you lazy?

We are both hungry and diligent. But despite being an Apache CouchDB developer, I am a sysadmin first. Before I take your money to steward your data, I have an ethical duty to produce a proven, reliable service.

• We proved our mettle, in production, for two years (CouchOne plus Iris Couch)
• We have had zero data loss incidents.
• We have zero single points of failure.
• We have a confirmed, tested backup process
• We have a confirmed, tested off-site disaster recovery process

Is it perfect? No, it is better than “perfect.” It is very good. We built something befitting a proper database company.

Cloudant is hosting the CouchDB Summit this week in Boston and we will be capping off the two-day affair with a public drinkup on Tuesday night. We invite everyone to come visit with the comitters and join us in wrapping up the summit. Just look for the folks in good lookin’ Cloudant swag. Details:

• Tuesday April 10th
• 6 - 9pm
• Lucky’s Lounge
• 355 Congress Street, Boston, MA.
• All Beers on Cloudant’s Tab

Just now, the Apache CouchDB project released version 1.2.0.

I am happy to announce that all Iris Couch accounts are now running this release. You needn’t do anything, the upgrade is automatic. Just relax on the Couch!

Notable improvements:

• Persistent cookies—users remain logged in through a browser restart
• More secure _users database
• Better, faster replication
• Reduced (and faster) disk usage

We also upgraded to the new Ocasta Labs Facebook Authentication plugin. It is now easy and relaxing to build a Couch app letting users log in via Facebook.

Please email us if you have questions or issues, or visit our support forum. Thanks!

I seem to have not blogged about this, but Open Advice, our book on Free and Open Source Software: what we wish we had known when we started, was published last month.

The book was edited by Lydia Pintscher and includes essays from 42 authors, many of whom you'll recognize if you tend to go to FOSS conferences. The LWN book review concludes:

Open Advice is a book that will be helpful to those who are new to FOSS, but, because of the individual voices, styles, and tones, it doesn't read like a "how to". It could even be recommended to those who aren't necessarily interested in contributing, but are curious about what this "free software thing" is all about. It is, in short, a great book for a variety of audiences and the (mostly) two or three page essays make it easy to read, while the anecdotes and recollections personalize it. The authors, editor, and everyone else who helped should be very pleased with the result. Readers will be too.

I probably shouldn't give the ending away, but my essay on cross-project collaboration, a subject I've also blogged about, ends with:

Good luck with breaking down the project boundaries! In most cases it works if your ideas are good and presented with an open mind. But even if you do not find a common ground, as long as your implementation solves the use case for you it has not been in vain. After all, delivering software, and delivering great user experience is what counts.

The book is licensed under CC-BY-SA, and is available as free download in ePub, mobi and PDF formats, and as paperback from Lulu. The book sources are available on GitHub, patches welcome!

On March 8th we hosted a webinar featuring presentations from Meteor Solutions and SoftLayer. Ben Straley, CEO of Meteor Solutions, gave a presentation on how his company developed their social audience marketing platform on the Cloudant data layer (powered by Softlayer). Afterward, Nathan Day, Chief Scientist at SoftLayer, elaborated on SoftLayer’s inner workings and high-performance network, which allows it to provide a global platform for web applications. Kicking things off I described some of the pertinent technology behind and features of the Cloudant Data Layer, and how, together with SoftLayer, we helped Meteor Solutions keep up with their ever growing data volumes and customer demands.

For those that missed it (or just want to view it again), we recorded the event, which you can view below. If you are interested in a copy of the slides, you will find those at the link below the video.

Download the slides here

We are happy to announce that Andy Palmer, serial entrepreneur and Boston startup luminary, is joining the Cloudant Board of Directors. Andy has a remarkable track record of starting and building amazing companies in Boston. His experience with and deep knowledge of Big Data systems (which he recently wrote about for GigaOm) will be invaluable as we build out the Cloudant Data Layer. Having previous worked with Andy, I know first hand how committed he is to building great companies in New England. His energy and passion are second to none and we at Cloudant are privileged to have him join our board.

For more information about Andy and Cloudant see our press page here

We are happy to announce today that Cloudant is hosting CouchDB Summit: Boston on April 9th and 10th. It promises to be a great, productive time and we are excited to introduce everyone to The @Cloudanator in person!

Here is the announcement e-mail that was sent to the Apache CouchDB user and developer mailing lists today.

Hello community,

For a few months now we at Cloudant have been talking about hosting a CouchDB Summit in Boston for the Apache CouchDB committers and some community members. This got pushed to the front burner due to recent events in the community, chief among them the purchase of a kegerator.

The goal is to give the CouchDB community the chance to be in the same room to discuss the direction of CouchDB over a two day period. Some admirable goals could include what CouchDB 2.0 should look like, how to get there, talking about auth, etc. Plus plenty of hacking time.

The results of the discussions will be posted to the community for decision making. Think RFC.

This is an invite only event with a max capacity of 20 to 30 people. We want to make sure that there is progress and flooding a room would inhibit that. The Apache CouchDB Committers are of course invited, plus some community members.

Apache CouchDB Commiters, please let me know if you would need financial assistance to attend. Cloudant is open to providing financial assistance for flights.

The particulars…

When: April 9 & 10

Where: Boston, MA, USA (venue to be announced)

$$$?: Cloudant is NOT charging for the event

Party?: Yes, there will be a PUBLIC drinkup for the community. Details to be posted separately.

RSVP: Invites are going out this and next week.

Let me know if you have any questions or concerns. We hope to make this a fun, positive experience for the community.

Cheers,

---

Sam Bisbee

Congratulations and a hearty round of applause are in order. Noah Slater, speaking on behalf of the Apache CouchDB Project Management Committee, recently announced that our very own Paul Davis has been appointed to that very PMC. According to Noah’s email to the CouchDB user and dev lists:

Paul Joseph Davis, a long time contributor to the CouchDB project, has been appointed to the PMC. He’s an asset to this community, and we’re delighted to have him on board. Please take a moment to congratulate him and show your appreciation for all his hard work so far.

Paul is an outstanding engineer and a pillar of the CouchDB community. We consider ourselves extremely lucky to have lured him to Cloudant last fall. He continues to be a staggering force within Cloudant, working on all aspects of our product, and still has the energy to contribute heavily to CouchDB. It’s a pleasure to see him rewarded for his efforts.

I have no doubt that he will make a fantastic member of the CouchDB PMC. With Paul helping to steer the project, and with version 2.0 coming up this year, I believe CouchDB has a very bright 2012 ahead.

I have a confession to make. We have been neglecting our friends in Europe. More than a year ago we launched our multi-tenant data layer with clusters on the east and west coasts of the US. Since then we have added shared clusters in Singapore and Japan and private clusters in numerous sites around the world. But, despite the requests from developers in Berlin, Amsterdam, Dublin, and more, we resisted opening up a shared cluster in Europe.

This was folly and it ends today. We are happy to announce the immediate availability of our brand-spanking-new EU-based cluster, “Jenever.” Jenever lives in SoftLayer’s new data center in Amsterdam just a few short milliseconds round trip from almost anywhere in the EU and UK. No longer will our European developer friends be forced to send requests or users across the Atlantic to hit the DB.

BUT WAIT THERE’S MORE! You can now choose the location of your data when you sign up or at any time. We maintain four shared clusters around the world, “Julep” in Northern Virginia, “Meritage” in San Jose, “Sling” in Singapore, and “Jenever” in in Amsterdam. When you sign up for a new account you will be asked where you want your data to live. If you ever want to change clusters, you can switch via your ‘accounts’ page. Note that switching clusters happens seamlessly but asynchronously. It takes some time to move an account so don’t expect this to happen instantly.

We at Cloudant believe that your data should live where your application lives, and your application should live where your users live. Our goal is for the Cloudant data layer to cover the globe, providing developers all around the world low-latency access to their data. With Jenever in Amsterdam, we take one step closer to this goal.

It is a big day here at Cloudant HQ; we are announcing the release of BigCouch 0.4! This release, which brings BigCouch into API equivalence with Apache CouchDB 1.1.1, has been baking for a while, and we are excited that it is now ready for public consumption. Instructions for installing and using BigCouch 0.4 can be found on the BigCouch page. Users running Debian Squeeze, Ubuntu (LTS or newer) or RedHat / CentOS / Amazon Linux are welcome and encouraged to use our prebuilt distributions based on Erlang/OTP R14B01 and SpiderMonkey 1.8.5.

There are a whole slew of new features and performance improvements in this release. You can read the entire list here, but I wanted to highlight a few of our favorite features. These should convince you to upgrade if you haven’t already.

For my money, the killer addition to BigCouch 0.4 is support for ‘zones,’ which allows you control over where you place the individual copies of your data. Cloudant’s hosted service makes extensive use of this feature to protect against isolated failures in individual data centers. As an example, our west coast cluster “Meritage” is actually spread over two different data centers in two distinct geographic regions. We use zone-support to ensure that at least one copy of all of our customers’ data exists in both locations. That way, if one location falls into the ocean or is subject to nuclear attack, our customers’ data is still safe and available. With this feature now in BigCouch, you can extend clusters across multiple locations.

Another important improvement we added, taken from Apache CouchDB 1.1, is support for the replicator DB to help manage multiple replications. The BigCouch replicator DB works the same as the Apache CouchDB version. A nice writeup of usage can be found here.

We’d like to thank everyone in the CouchDB community that has contributed bug reports, patches, feature requests, etc. for BigCouch and to all the brave souls willing to try out the bleeding-edge releases. This release would not be nearly the upgrade it is without all of you. Special shout out goes to Benoit for his help with FreeBSD and native SSL support.

Finally, for those following along with recent events in the land of CouchDB, we’re dedicated to donating and integrating BigCouch to Apache CouchDB. This sort of integration won’t happen overnight, so we are still planning on making releases to BigCouch while we iron out the divergences in the two projects. We very much appreciate the support and enthusiasm we have recieved from the CouchDB community to date, and we look forward to working with that community to strengthen Apache CouchDB.

Cloudant is happy to be sponsoring Node Summit next week in San Francisco. To keep the party going an extra day we will be hosting a CouchDB drinkup Thursday night to touch base with our west coast crew. This is instead of Benoit and Randall's meetup which they kindly merged with us - community powers unite!

Just look for the folks in good lookin' Cloudant swag.

• Thursday Jan 26th
• 5:30 - 8:30
• 21st Amendment Brewery
• 563 2nd St, San Francisco
• All Beers on Cloudant's Tab

Damien Katz’s blog post, The Future of CouchDB deserves a response.

Damien tackles difficult problems, thinking clearly and executing well. If he is focusing on Couchbase, then Couchbase will be outstanding.

Both CouchDB and Couchbase are domain-specific databases. Moreover, these domains intersect minimally. The projects represent non-overlapping magisteria; and one might say instead that the future of awesome datatabases is Couchbase.

We are interested in CouchDB’s domain, web applications modeled after the web itself: standard, decentralized, and skewed toward empowering regular people.

… is not Damien Katz.

TL;DR

The blog post Damien Katz wrote earlier today, doesn't mean much or anything for the Apache CouchDB project (or memcache project for that matter). If anything it's a public note that Damien Katz acknowledged that he moved (on) from CouchDB to Couchbase.

Short story, long

I'm not a contributor to CouchDB by means of code, (but) I blog a lot, I maintain the FreeBSD port, wrote a book and have an opinion on many things CouchDB. I've been a CouchDB-user for something like four years (since pre-0.8 times) and a BigCouch-user (and Cloudant customer) of about 1.5-two years.

I am not sure what Damien Katz tried to achieve when he posted his message to the community and while I personally find it ignorant (to say the least), it worries me how it is perceived by the general public.

Talk is cheap

Of course it may sound like the end of the world when the creater of CouchDB quits his own project, but truth to be told, Damien Katz left CouchDB a long time ago. Couchbase moved past CouchDB long before they announced it. Basically when they started integrating Membase, though there are all kinds of notable contributions from Couchbase employees (e.g. Filipe and Jan).

Damien himself hasn't (actually) commited in over a year to CouchDB. Which makes his move no real surprise, just the way he decided to communicate surprised me. Especially since he said to have no regrets, I find the tone and statements in his blog post rather questionable. That is both from a personal and professional perspective.

I just attended a Couchbase event in Berlin last year where talks about CouchDB were given along with newer Couchbase developments. So personally, while I welcome clarity, all too sudden changes in strategy don't make me happy. If I was new to CouchDB and/or Couchbase, this would look like a headless chicken (excuse the image) and way too much drama to get into.

And on a professional level, Damien's posts invalidates the efforts of many people who both contribute and work with Apache CouchDB on a daily basis.

Then later today Damien shared this:

TIL, if you create an open source project, you should stick with it forever and ever. Family can live off unicorns and stardust. — Damien Katz

Real talk

First off, the most people in this discussion (excluding HN, of course ;-)) are actually active Open Source contributors one way or the other. Many of us have other projects (plural) besides CouchDB. It's not that we troll about something we have no idea about.

Secondly, it's not the fact that Damien left, it's how he left.

No one blames people for moving on: it happens all the time. I do it all the time — write code, push it out, move on.

If code is good enough it'll be picked up, if not, it'll rott on Github for forever. It happened to other projects and it happened to CouchDB. But why would anyone pronounce a project dead where he is not anymore invested in?

Anyway: I wish Damien good luck in the future.

So where is CouchDB at?

I wrote a blog post about the current state of CouchDB last year (2011) in early December.

A few things have changed since then:

Overall, I still see contributions from notable community members all around. Not sure if it's my own perception, but there has been more activity as of late. A new release of Apache CouchDB is just around the corner. Overall, good times for Apache CouchDB indeed.

The other thing that happened after my blog post is that Couchbase said in their 2011 review (which was published in late December) that it would officially step off Apache CouchDB and contribute documentation and OSX builds (aka CouchDBX or Couchbase Single Server) to the Apache CouchDB project.

This is great news and announcing that they will step off the turf is fine too since it clears up a couple misconceptions people may have about Apache CouchDB and the former Couchbase Single Server.

And all in, this makes Damien's blog post even more unnecessary and confusing to many users out there. Especially confusing for those who are not neck-deep in Apache CouchDB — and by that I mean: they are neither subscribed to a mailing list, take part on IRC, read the CouchDB planet or know any of the contributors directly.

In the end his blog post confuses people because it contains absolutely nothing but fear, uncertainty and doubt (FUD).

Outlook

I can't reveal too much because it's not my business to announce anything — let me just say that there are good times ahead for Apache CouchDB.

Fin

I'd like to get past all the drama and re-focus on what is important: CouchDB and our data. I don't care for the rest, I want to see exciting things from Apache CouchDB in 2012.

The CouchDB world is currently full of “The future of CouchDB” blog posts. It started with the blog post from Damien Katz the creator of CouchDB. Of course people were also concerned about the future of GeoCouch. No worries, it will be good.

The future of Apache CouchDB

The reactions were quite different. People who are not deeply involved with the CouchDB community think that this means the end of Apache CouchDB. My reaction was positive, I tweeted:

“It’s good to see the Damien is so open to [the] world”

The reason was, that for me it was pretty clear that it would happen, and I was just happy that Damien officially made the cut.

The reactions from CouchDB community members where pretty much what Till Klampäckel describes in his blog post. You could see it comming after Couchbase announced that they are not the CouchDB company and that their product won’t be Apache CouchDB compatible.

I agree with Till here, the way Damien wrote his blog post, isn’t the best imaginable. For outsiders, it really seems to be the end of Apache CouchDB, but it is not. For me it just shows, why foundations like the Apache Foundation are such a great idea. Even if the original creator leaves the project, it still lives on.

Apache CouchDB has a lot of contributers and the mailing lists and IRC channel is busy as always. That CouchDB has a future is also shown by the blog post from Cloudant. They will keep supporting Apache CouchDB.

The future of GeoCouch

After this quick recap what happened so far, it’s time to talk about the future of GeoCouch. As you may know, I work for Couchbase on the integration of spatial functionality into their product.

Currently the overlap between Apache CouchDB and the version Couchbase uses internally is still quite huge, but it will diverge more and more in the future. Thus it will get harder and harder to maintain a single version that supports Apache CouchDB and Couchbase.

The good news is, that GeoCouch is pretty much a data structure only. It's an R-tree that stores JSON documents. This can easily be used by CouchDB and Couchbase. Perhaps small wrappers will be needed, but those should be minimal.

The easiest way to understand how the future looks like is in a small illustration:

GeoCouch's core is the R-tree, it's the same code for CouchDB and Couchbase. On top of it there will be code that is specific to either CouchDB or Couchbase.

This means that the majority of the devlopment I do for Couchbase will also improve the GeoCouch you can use for CouchDB.

Conclusion

The future of all three, Apache CouchDB, Couchbase and GeoCouch looks bright.

CouchDB and it's users have been in a state of flux all year, with declarations of the project's death by outsiders and the rest of community scrambling to fall in-line behind their chosen implementation.

Let's relax and ignore the marching drums and nay-sayers for a moment. For me, CouchDB was never about a specific platform or Mongo vs. Couch, but about a more fundamental view of the world. CouchDB offered a refuge from the aggressive centralisation of the web. Its about decentralisation, smart clients and sharing data.

The cloud promised a new era of flexibility and reduced cost, but it's in opposition to one of the most important principles of the 'net. An open network without intermediaries, a network of decentralised services and truly public spaces. The cloud means centralisation. That may not chime with our delusion of the startup dream where anyone can change the world, but in reality Amazon and Facebook are not easily displaced. Centralisation risks control, these intermediaries are far easier to censor and lock-down than the edges of the network.

The most powerful feature CouchDB offers is a protocol for moving data around using HTTP. Master-master replication really shines in decentralised systems, and makes possible the idea of smart clients and services at the edge of the network, without completely throwing out all the benefits of the cloud. Or as Chris Anderson puts it 'Ground Computing'.

There are a number of implementations now working following this protocol, not just Apache CouchDB. There's PouchDB for the browser, the upcoming lightweight TouchDB from CouchBase and of course BigCouch from Cloudant. Rather than seeing this as fragmentation, I see it as a vibrant ecosystem of projects sharing the CouchDB protocol. That means more opportunities to share data between devices without locked down platforms sitting between them.

If replication addresses the difficulties of decentralised data, CouchApps address the difficulties of running web applications at the edges. CouchApps are a structured way of writing a data-powered application so it can be run in multiple places and on multiple platforms. Max Ogden and Mikeal are even exploring running them as Google Chrome extensions. I believe the combination of easy to install web apps and easy to sync data has the potential to unlock nascent demand for data ownership.

This is why I spend my time working on Kanso and CouchApps. The future of CouchDB is not Apache CouchDB, it's all of us working together. The fact that CouchBase Server does not follow the HTTP replication protocol and does not support CouchApps means it has stepped away from the rest of the community and failed to re-kindle the ideas that defined my earliest encounters of the internet. A massively decentralised network of plural participation.

Forget trying to be the next Mongo and get back to changing the world.

Ride with me

Background

• Damien Katz abandons Apache Couchdb
• Cloudant's response
• Till's response
• Volker Mische's response

The future of CouchDB is Apache CouchDB.

Cloudant over the past 3 years has built and refined BigCouch, a fault-tolerant, horizontally scalable clustering framework purpose-built for CouchDB. Today we announce our intent to contribute this work back to the community. Working with the ASF and the CouchDB community, we hope to integrate the core capabilities of BigCouch into Apache CouchDB. Hopefully this will put to rest the tired (and false) “CouchDB doesn’t scale” meme. BigCouch forms the bedrock of a globally distributed, sophisticated technology stack that we’ve had in production operation for over two years at scale.

Those of you building applications with CouchDB know that its performance has been improving by leaps and bounds. We’ve achieved this through low-level implementations of performance-critical routines and smart refactoring of the Erlang codebase, but without sacrificing the rock-solid durability and stable REST API people have come to expect from CouchDB. The growing community of active committers includes passionate, stellar developers. More and more people are using CouchDB for projects, big and small, every day.

We, along with a host of other companies, strongly support the open source community in building CouchDB and we do not plan on stopping. We have been fortunate in our ability to attract outstanding engineers, investors, and customers. We intend to continue devoting resources to Apache CouchDB and offer our help in any way the community desires. The future of CouchDB is CouchDB.

Damien Katz has been an excellent founder and steward of CouchDB. It’s been an honor to work with him, and we wish him well in his new location and mission.

Update, 2011-12-21: Couchbase posted their review of 2011 (the other day) — TL;DR: Couchbase Single Server (their Apache CouchDB distribution) is discontinued and its documentation (and its buildtools) will be contributed to Apache CouchDB.

When Ubuntu1 dropped CouchDB two weeks ago, there were a couple things which annoy (present tense) me a lot. Add to that the general echo from various media outlets blogs which pronounced CouchDB dead and a general misconception how this situation or CouchDB in general is dealt with.

Some people said I am caremad about CouchDB and that is probably true. Let me try to work through these things without offending more people.

Ubuntu1

What annoy[ed,s] me about this situation is that I wrote a chapter about Ubuntu1 in my CouchDB book. And while I realize that as soon as a book is published the information is outdated, I also want to say that I could have used the space for another project.

I talked to a couple of people about CouchDB at Ubuntu1 on IRC and no one made it sound like they are having huge or for that matter any issues.

Of course I neither work for Canonical or Couchbase. I haven't signed any NDAs etc. — but looking back a week or two my well-educated guess is that not even the people at Couchbase knew there were fundamental issues with CouchDB and Ubuntu1.

The NDA-part is of course an assumption: don't quote me on it.

Transparency

Scumbag Ubuntu1 drops CouchDB and doesn't say why. — myself on Twitter

First off: I'm not really sorry. I was abusing a meme and if you read my Twitter bio, you should not take things personal.

I also should have known better since it's not like I expect anything transparent from Canonical. (Just said it.)

When people are compelled to write a press release and put it out like that, they should expect a backlash. The reason why I reacted harsh is that Canonical didn't share any valuable information on why they discontinued using CouchDB except for: it doesn't scale.

And I'm not aware of anything concious to date.

Helpful criticism — how does it work?

Please take a look at the following email: https://lists.launchpad.net/u1db-discuss/msg00043.html

This email contains a lot of criticism. And it's all valid as well.

CouchDB feedback

Other examples:

• email on couchdb-user by James Marca
• second email on couchdb-user by James Marca

These are great emails because they contain extremely valuable feedback.

Deal with it!

In my (humble) opinion, these kind of emails are exactly what is necessary in CouchDB-land, and many other open source projects: criticism and a little time to reflect on not so awesome features. And then moving on to make it better. If the feedback cycle doesn't happen, there's no development or evolution — just stagnation.

And in retrospect I wish more people would share their opinion on CouchDB and this situation more often. Since I'm personally invested in CouchDB, it's hard to say certain things. Honesty is sometimes brutal, but it's necessary.

In summary, a CouchDB user like Ubuntu1 (or Canonical) doesn't have the civic duty to give feedback, but to desert a project while pretending to be an Open Source vendor, and not talking to the community of the project or sharing your issues in public, that is extremely unhelpful.

Overall it strikes me that the only thing to date known about Canonical's collaboration with CouchDB is the support for OAuth in CouchDB. And most people don't even know about that (or wouldn't know how to use it). It worries me personally to not know the kind of problems Canonical ran into because they seem so messed up that they couldn't be discussed in public.

CouchDB doesn't scale

One thing I was able to extract is: CouchDB doesn't scale.

Thanks! But no thanks.

I wrote a book on CouchDB and I pretty much used it all, or at least looked at it very, very closely. I also get plenty of experience with CouchDB due to my job. Indeed, there are many situations where CouchDB doesn't scale or where it becomes extremely hard to make it scale. Situations where the user is better of putting data somewhere else.

Myself (and I'm assuming others) enjoy to learn the reasons why things break, so we can take this experience and use it going forward. If this doesn't happen we might as well all subscribe to the koolaid of a closed source vendor and purchase update subscriptions, install security packs and happily live ever after.

A patch to make CouchDB scale?

Another piece of information I gathered from the various emails written is that Canonical maintained CouchDB-specific patches for Ubuntu1. However, it's unknown what the purpose of these patches were. For example, if these patches made CouchDB scale (magically) for Ubuntu1 or if the patchset added a new feature.

What I'd really like to know is why these patches were not discussed in the open and why no one worked with the project on incorporating them into upstream. The upstream is the Apache CouchDB project.

This is another example of where communication went horribly wrong or just didn't happen.

A CouchDB company

I'm a little torn here and I don't want to offend anyone (further) especially since I know a couple Couchbase'rs or original CouchOne'rs (Hello to Jan, JChris and Mikeal) in person, but seriously: a lot of people realized that CouchOne stopped being The CouchDB company a long time ago.

This is not to say that the CouchDB project members who are employed by CouchOne/Couchbase are not dedicated to CouchDB. But if I take a look at the mobile strategy and the more or less recent integration of CouchDB with Membase/Memcache, I must notice that these strategies are far away from Apache CouchDB. Big data (whatever that means), to mobile and back.

The conclusion is that the majority of work done will not be merged into Apache CouchDB and this is one of the reasons why the Apache CouchDB project hasn't evolved much in a long time.

Not all changes can go upstream

I realize that when a company has a different strategy, not everything they do can be send upstream. After all, most if not all companies operate in a world where money is to be made and goals are to be met. Nothing wrong there.

But let's take a look at the one project which could have been dedicated to Apache CouchDB: the documentation project.

CouchOne hired an ex-MySQL'er to write really great documentation for CouchDB. The documentation made sense, it was up to date with releases, contained lots examples and what not. But it was never contributed to the open source project. The documentation is still online today, though it's now the documentation of the Couchbase Server HTTP API.

Wakey, wakey!

So in my opinion the biggest news is not that Canonical stopped using CouchDB and it's also not outrageous to think that there can be one CouchDB company. The biggest news is that Couchbase officially said: "It's not us!".

Having said that and also not knowing much about Canonical's setup and scale, I still fail to even remotely understand why they didn't work with Cloudant who spezialize in making CouchDB scale all along.

CouchDB and Evolution

Of course it is unfair to single them (Couchbase employees) out like that. For the record, there are pretty vivid projects such as GeoCouch which are also funded by Couchbase and while being devoted to the project, these guys also have to meet goals for their company.

Add to that, that other CouchDB contributors involved have not driven sustantial user-facing changes in Apache CouchDB either. CouchDB is still a very technical project with a lot of technical issues to solve. The upside to this situation is that while other NoSQL vendors add new buzzwords to each and every CHANGELOG, CouchDB is very conservative and stability driven. I appreciate that a lot.

User-facing changes on the other side are just as important for the health of a project. Subtle changes aside, but today's talks on for example querying CouchDB are extremely similar to those talks given a year or two ago. Whatever happens in this regard is not visible to users at all.

Take URL rewriting, virtualhosts and range queries as examples for features. I question:

• the usefulness for 80% of the users
• the rather questionable quality
• the state of their documentation

Users need to have the ability to grasp what's on the roadmap for CouchDB. There needs to be a way for not so technical users to provide feedback which is actually incorporated into the project. All of these things aside from opening issues in a monster like Jira.

Since no one bothers currently, this is not going to happen soon.

Pretty candid stuff.

Marketing

In terms of marketing and with a lack of an official CouchDB company, the CouchDB project has taken a PostgreSQL-attitude in the last two years.

In a nutshell:

We don't give a damn if you don't realize that our database is better than this other database.

This is a little dangerous for the project itself because when I look at the cash other NoSQL vendors pour into marketing for their NoSQL database, I realized quickly that with the lack of support this project can go away pretty soon.

CouchDB being an Apache project doesn't save me or anyone either: clean intellectual property, deserted, for forever.

The various larger companies (let's say Cloudant and Meebo) are basically employed with their own forks with maybe too little reason to merge anything back to upstream yet. There are independent contributors Enki Multimedia who contribute to core but also sub projects like CouchApp.

And then, there's Couchbase which is trying to tie CouchDB behind Memcached. And from what I can tell pretty much abondens HTTP and other slower CouchDB principals in the process.

Is CouchDB alive and kicking?

You saw it coming: it depends!

Dear Jan, I'm still thinking about the email you wrote while I write my own blog entry. And honestly, that email and the general response raised more questions for myself and others than it answered.

I'd like to emphasize a difference I see (thanks, Lukas):

Core

Is the core of Apache CouchDB alive? — It's not dead.

• Yes, because some companies drive a lot of stability into CouchDB.
• No, because there's little or no innovation happening right now.

Ecosystem

There is a lot of innovation going on in CouchDB's ecosystem.

Most notable, the following projects come to mind:

• BigCouch
• Couchappspora
• CouchDB-lucene
• Doctrine ODM in PHP (and I'm sure there are similar projects in other languages)
• ElasticSearch's river
• erica
• GeoCouch
• Lounge (and lode)
• various JavaScript libraries to connect CouchDB with CouchApps or node.js
• various open data projects (like refuge.io)

Need more? Check out CouchDB in the wild which I think is more or less up to date.

Hate it or love it — there is plenty of innovating going on. And many (if not all) CouchDB committers are a part of it.

The innovation just doesn't happen in CouchDB's core.

Fin

My closing words are that I don't plan on migrating anywhere else. If anything, we have mostly migrated to BigCouch.

For Apache CouchDB, I think it's important that someone fills that void. That can be either a company, a BDFL or more engaging project leaders (plural). I think this is required so the project continues vividly.

Because I would really like to see the project survive.

2011 has been quite an eventful year for Cloudant. In the spring we outgrew our tiny basement offices in Davis Square and moved Cloudant HQ into Boston proper. We added a number of fantastic engineers to the team and we said goodbye to some of our earliest hires.

Going into the new year, we’ve expanded our platform to 5 geographic zones in the US and 3 more around the globe. Most importantly we more than doubled the number of developers, applications, small businesses, and enterprises using our hosted service – and found quite a few independent BigCouch users out there along the way. We continue to be surprised and delighted by the passion our users show for our products. It is that passion that drives us to continuously improve BigCouch and our hosted service.

With all that has been going on here, the founders decided it was time to expand the team. So today we are happy to announce that Derek Schoettle has joined Cloudant as our new CEO. Yes, Derek has an MBA – don’t hold that against him. His passion is working within early-stage companies bringing great technology to market, most recently at Vertica, and numerous startups before that. Derek has been an advisor to Cloudant for most of 2011 and was a natural choice when the time came. We are very excited to have him on board helping Cloudant grow.

With Derek on board, Adam, Mike and I will have more time to focus on the things we are passionate about . For me, that means working closely with our customers and users, focusing on building a great product that solves their data problems. Mike will continue to focus on customer success and outreach, and will hopefully have more time ‘on the road,’ evangelizing BigCouch to a wider audience. Adam will continue to work deep in the guts of Apache CouchDB, BigCouch, and the Cloudant platform, making all of our crazy thought experiments into real features.

2011 was a big year for Cloudant, but we have even bigger goals for 2012 (assuming the world doesn’t end.) Stay tuned to this channel as we announce some of the big features and plans we have.

Alan (for Adam and Mike)

Co-Founder, Cloudant

If you've ever wondered whether there's a tool that will automatically compress JavaScript files when you're pushing your CouchApp somewhere, the answer is that there exists at least one: couchapp-compress.

Basically, couchapp-compress is a small Ruby script that wraps the couchapp command line tool. It compresses a CouchApp's JavaScript files and puts them altogether into one file, and temporarily changes the CouchApp so that instead of all the single uncompressed JavaScript files the compressed one is used. It pushes the CouchApp and then restores the previous state, so that again everything looks like before couchapp-compress was executed.

Check out the README for more details if you're curious and want to give it a try.

On December 14th we hosted a webinar with our friends at 2600hz, the open-source VOIP platform. Darren Schreiber, CEO of 2600hz, gave a presentation on how his company uses the BigCouch data platform to better serve its customers. Darren gave some great examples of how moving to a scalable document store has helped his company move faster, save money, and avoid operational headaches.

In case you missed it, we recorded the webinar and you can view it below. If you just want the slides, follow the link below the video.

Download the slides here

    Invitees = Backbone.Collection.extend({
      model: Models.Invitee,

      url: function( models ) {
        return '/invitees?' + ( models ? 'ids=' + _.pluck( models, 'id' ).join(',') : '' );
      },
      parse: function(response) {
        debugger;
        return _(response.rows).map(function(row) { return row.doc; });
      }
    });

{
   "_id": "6bb06bde80925d1a058448ac4d006f6e",
   "_rev": "3-231654f6914afe2e20eb57a41ec8497a",
   "firstName": "Black",
   "lastName": "Francis",
   "type": "Person"
}

attributes: Object
  id: "6bb06bde80925d1a058448ac4d006f6e"

doc: Object
  _id: "6bb06bde80925d1a058448ac4d006f6e"
  _rev: "3-231654f6914afe2e20eb57a41ec8497a"
  firstName: "Black"
  lastName: "Francis"
  type: "Person"

    Invitees = Backbone.Collection.extend({
      // ...
      parse: function(response) {
        return _(response.rows).map(function(row) {
          var doc = row.doc;
          doc['id'] = doc['_id'];
          return doc;
        });
      }
    });

{
   "_id": "6bb06bde80925d1a058448ac4d004fb9",
   "_rev": "2-7fb2e6109fa93284c19696dc89753102",
   "title": "Test #7",
   "description": "asdf",
   "startDate": "2011-11-17",
   "invitees": [
       "6bb06bde80925d1a058448ac4d0062d6",
       "6bb06bde80925d1a058448ac4d006758",
       "6bb06bde80925d1a058448ac4d006f6e"
   ]
}

    Appointment = Backbone.RelationalModel.extend({
      // ...
      relations: [
        {
          type: Backbone.HasMany,
          key: 'invitees',
          relatedModel: 'Invitee',
          collectionType: 'Invitees'
        }
      ],
      // ...
    });

    Appointment = Backbone.RelationalModel.extend({
      // ...
      initialize: function(attributes) {
        if (!this.id)
          this.id = attributes['_id'];

        this.fetchRelated("invitees");
        // this.loadInvitees();
      },
      // ...
    });

    var AppointmentEdit = new (Backbone.View.extend({
      // ...
      showInvitees: function() {
        $('.invitees', this.el).remove();
        $('#edit-dialog').append('<div class="invitees"></div>');

        if (this.model.get("invitees").length == 0) return this;

        var view = new Invitees({collection: this.model.get("invitees")});

        $('.invitees').append(view.render().el);

        return this;
      }
    }));

app.get('/invitees', function(req, res){
  var options = {
    method: 'POST',
    host: 'localhost',
    port: 5984,
    path: '/calendar/_all_docs?include_docs=true'
  };

  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) { /* ... */ });

  var ids = req.param('ids').split(/,/);
  couch_req.write(JSON.stringify({"keys":ids}));
  couch_req.end();
});

    Invitees = Backbone.Collection.extend({
      model: Models.Invitee,

      url: function( models ) {
        return '/invitees' + ( models ? '?ids=' + _.pluck( models, 'id' ).join(',') : '' );
      },
      parse: function(response) {
        return _(response.rows).map(function(row) { return row.doc;});
      }
    });

Big day for Cloudant — and big news in the near future for all BigCouch users. We are happy to announce that biotech heavyweight and perennial Fortune 500 company Monsanto has chosen Cloudant to be the core of their new genome analysis platform.

We’ve been working with them for a few weeks now and we couldn’t be more thrilled with the partnership. We are no strangers to the analysis of the big bytes generated by the smallest of things. Our founders first devised our open-source platform BigCouch while at MIT, working on Large Hadron Collider experiments generating millions of data points per second from the collision of atomic particles. Turns out, Biotechnology & genome analysis is no different in this regard. As GigaOM summarized it: “innovation is rampant, but data growth is outpacing the ability to analyze it, making faster, cheaper and more scalable data systems integral to leveling the playing field”.

This is new ground for us — and for NOSQL as a whole. This Monsanto platform is not destined to toil away in one of their (many) R&D labs, only used by scientists with lab coats & beakers. We’re not powering the CMS of some administrative division or a minor part of their website. Cloudant’s BigCouch will be the core, for both storage and analysis of a new, company-wide platform powering a fundamental aspect of a Fortune 500 business: the analysis & identification of new traits & genomic combinations in agricultural crops. The data & reporting interfaces will be used across Monsanto and should be instrumental in the making of key business decisions.

Now, this also means big news for our users. Monsanto has been very eager to let us open-source many of the core enhancements we make to BigCouch for them. So while this won’t quite make it into the impending BigCouch 0.4 release (which is days away now), or on our cloudant.com clusters for another few months, you can definitely expect to see many Monsanto-driven improvements folded back into our offerings, such as support for more languages on view servers, new tools to model data workflows, as well as a flurry of improvements to our storage & retrieval algorithms.

At Cloudant, we’re all excited to see NOSQL platforms, including BigCouch, being used to power the next generation of business technology. So when announcements like this also translate to a more stable, performant & productive experience, for everybody else that uses a Cloudant product, we find it to be a big day indeed.

You can always contact us if you have any questions. There is also a press release available.

  function draw_calendar(year_and_month) {
    $('.year-and-month', 'h1').html(' (' + year_and_month + ') ');
    reset_calendar();
    add_dates_to_calendar(year_and_month);
  };

draw_calendar('2011-09');

➜  calendar git:(pagination) curl http://localhost:5984/calendar/_design/appointments/_view/by_month\?key\='"2011-10"'
{"total_rows":18,"offset":12,"rows":[
  {"id":"4a5600f5b2e36fc99d24fe9b8700037d",
   "key":"2011-10",
   "value":{
     "_id":"4a5600f5b2e36fc99d24fe9b8700037d",
     "_rev":"5-181418fffeacdc08b5fdda91525978b6",
     "title":"Update dialog errors",
     "description":"asdf1",
     "startDate":"2011-10-05"}},
  {"id":"8b5c80c0211068428272af4784000451",
   "key":"2011-10",
   "value":{
     "_id":"8b5c80c0211068428272af4784000451",
     "_rev":"1-87f5be84687eada2cd178c8dab7aa34c",
     "title":"Finish Beta",
     "description":"Book important",
     "startDate":"2011-10-31"}},
  {"id":"8b5c80c0211068428272af478400f7bb",
   "key":"2011-10",
   "value":{
     "_id":"8b5c80c0211068428272af478400f7bb",
     "_rev":"2-b1ce8c6c815a11f7b78b7db412988451",
     "title":"Go to bed early",
     "description":"asdf",
     "startDate":"2011-10-22"}},
  {"id":"8b5c80c0211068428272af478400fc93",
   "key":"2011-10",
   "value":{
     "_id":"8b5c80c0211068428272af478400fc93",
     "_rev":"1-b19359520433df557ad2fa0d56165f24",
     "title":"Validations",
     "description":"description should be required",
     "startDate":"2011-10-03"}},
   {"id":"956a5c19fd866a6a024bbb4c39002e3b",
    "key":"2011-10",
    "value":{
      "_id":"956a5c19fd866a6a024bbb4c39002e3b",
        "_rev":"2-0bbb8660f7f116cc813e0fd9093cec6a",
        "title":"Has Description",
        "description":"asdf",
        "startDate":"2011-10-13"}},
   {"id":"956a5c19fd866a6a024bbb4c390031a2",
    "key":"2011-10",
    "value":{
      "_id":"956a5c19fd866a6a024bbb4c390031a2",
      "_rev":"2-0568aded9df520a0c1c8bb2ee8961156",
      "title":"In-dialog errors","description":"asdf",
      "startDate":"2011-10-04"}}
]}

      var Collections = (function() {
        var Appointments = Backbone.Collection.extend({
          model: Models.Appointment,
          url: '/appointments',
          parse: function(response) {
            return _(response.rows).map(function(row) { return row.value ;});
          }
        });

        return {Appointments: Appointments};
      })();

// ...
      // Initialize the app
      var appointments = new Collections.Appointments();

      new Views.Application({collection: appointments});

      var today = new Date(),
        year = today.getFullYear(),
        month = today.getMonth() + 1,
        year_and_month = year + '-' + pad(month);

      appointments.fetch({data: {date: year_and_month}});

Got response: 400 localhost:5984/calendar/undefined
{ error: 'bad_request', reason: 'Invalid rev format' }

[info] 127.0.0.1 - - 'PUT' /calendar/undefined 400

    var faye = new Faye.Client('/faye');
    Backbone.sync = function(method, model, options) {
      faye.publish("/calendars/" + method, model);
    }

    var faye = new Faye.Client('/faye');
    Backbone.sync = function(method, model, options) {
      if (model.toJSON) model = model.toJSON();
      faye.publish("/calendars/" + method, model);
    }

Faye.Transport.WebSocket = Faye.extend(Faye.Class(Faye.Transport, {
  // ... 
  request: function(messages, timeout) {
    this._timeout = this._timeout || timeout;
    this._messages = this._messages || {};
    Faye.each(messages, function(message) {
      this._messages[message.id] = message;
    }, this);
    this.withSocket(function(socket) { socket.send(Faye.toJSON(messages)) });
  },
  // ...
});

client.subscribe('/calendars/update', function(message) {
  // HTTP request options
  var options = {
    method: 'PUT',
    host: 'localhost',
    port: 5984,
    path: '/calendar/' + message._id,
    headers: {
      'content-type': 'application/json',
      'if-match': message._rev
    }
  };

  // ...
});

    Backbone.sync = function(method, model, options) {
      var message = model.toJSON();
      if (!message._id && message.id) message._id = message.id
      if (!message._rev && message.rev) message._rev = message.rev

      faye.publish("/calendars/" + method, message);
    }

Got response: 201 localhost:5984/calendar/8b5c80c0211068428272af478400df1e
{ ok: true,
  id: '8b5c80c0211068428272af478400df1e',
  rev: '4-2929cf0e4933a4e32474145eb3e79f02' }

    var faye = new Faye.Client('/faye');
    Backbone.sync = function(method, model, options) {
      faye.publish("/calendars/" + method, model);
    }

    _(['create', 'update', 'delete', 'read', 'changes']).each(function(method) {
      faye.subscribe('/calendars/' + method, function(message) {
        console.log('[/calendars/' + method + ']');
        console.log(message);
      });
    });

client.subscribe('/calendars/delete', function(message) {
  // HTTP request options
  var options = {...};

  // The request object
  var req = http.request(options, function(response) {...});

  // Rudimentary connection error handling
  req.on('error', function(e) {...});

  // Send the request
  req.end();
});

  // HTTP request options
  var options = {
    method: 'DELETE',
    host: 'localhost',
    port: 5984,
    path: '/calendar/' + message._id,
    headers: {
      'content-type': 'application/json',
      'if-match': message._rev
    }
  };

  var req = http.request(options, function(response) {
    console.log("Got response: %s %s:%d%s", response.statusCode, options.host, options.port, options.path);

    // Accumulate the response and publish when done
    var data = '';
    response.on('data', function(chunk) { data += chunk; });
    response.on('end', function() {
      var couch_response = JSON.parse(data);

      console.log(couch_response)

      client.publish('/calendars/remove', couch_response);
    });
  });

    faye.subscribe('/calendars/remove', function(message) {
      console.log('[/calendars/remove]');
      console.log(message);
    });

    faye.subscribe('/calendars/remove', function(message) {
      console.log('[/calendars/remove]');
      console.log(message);

      calendar.appointments.remove(message);
    });

    _remove : function(model, options) {
      options || (options = {});
      model = this.getByCid(model) || this.get(model);
      // ...
    }

    // Get a model from the set by id.
    get : function(id) {
      if (id == null) return null;
      return this._byId[id.id != null ? id.id : id];
    },

        var Appointment = Backbone.View.extend({
          initialize: function(options) {
            // ....
            options.model.bind('destroy', this.remove, this);
            options.model.bind('error', this.deleteError, this);
            options.model.bind('change', this.render, this);
          },
          // ...
        });

        var Appointment = Backbone.View.extend({
          initialize: function(options) {
            // ....
            options.model.bind('remove', this.remove, this);
            options.model.bind('error', this.deleteError, this);
            options.model.bind('change', this.render, this);
          },
          // ...
        });

          var Appointment = Backbone.Model.extend({
          urlRoot : '/appointments',
          initialize: function(attributes) {
            // ...
            this.faye = new Faye.Client('/faye');
          },
          // ...
          sync: function(method, model, options) {
            if (method == "create") {
              this.faye.publish("/calendars/public", model);
            }
            Backbone.sync.call(this, method, this, options);
          }
        });

    var faye = new Faye.Client('/faye');
    Backbone.sync = function(method, model, options) {
      faye.publish("/calendars/" + method, model);
    }

    // Simple logging of Backbone sync messages
    _(['create', 'update', 'delete', 'read']).each(function(method) {
      faye.subscribe('/calendars/' + method, function(message) {
        console.log('[/calendars/' + method + ']');
        console.log(message);
      });
    });

      // Initialize the app
      var appointments = new Collections.Appointments;

      new Views.Application({collection: appointments});
      appointments.fetch();

// Faye server
var bayeux = new faye.NodeAdapter({mount: '/faye', timeout: 45});
bayeux.attach(app);

// Faye clients
var client = bayeux.getClient();

client.subscribe('/calendars/read', function() {
  // do awesome stuff here
});

client.subscribe('/calendars/read', function() {
  //  CouchDB connection options
  var options = {
    host: 'localhost',
    port: 5984,
    path: '/calendar/_all_docs?include_docs=true'
  };

  // Send a GET request to CouchDB
  var req = http.get(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    // Accumulate the response and publish when done
    var data = '';
    couch_response.on('data', function(chunk) { data += chunk; });
    couch_response.on('end', function() {
      var all_docs = JSON.parse(data);
      client.publish('/calendars/reset', all_docs);
    });
  });

  // If anything goes wrong, log it (TODO: publish to the /errors ?)
  req.on('error', function(e) {
    console.log("Got error: " + e.message);
  });
});

    window.calendar = new Cal();

    faye.subscribe('/calendars/reset', function(all_docs) {
      console.log('[/calendars/reset]');
      console.log(all_docs);

      calendar.appointments.reset(all_docs);
    });

        var Appointments = Backbone.Collection.extend({
          model: Models.Appointment,
          parse: function(response) {
            return _(response.rows).map(function(row) { return row.doc ;});
          }
        });

    faye.subscribe('/calendars/reset', function(message) {
      console.log('[/calendars/reset]');
      console.log(message);

      var all_docs = calendar.appointments.parse(message);
      calendar.appointments.reset(all_docs);
    });

client.subscribe('/calendars/update', function(message) {
  // HTTP request options
  var options = {...};

  // The request object
  var req = http.request(options, function(response) {...});

  // Rudimentary connection error handling
  req.on('error', function(e) {...});

  // Write the PUT body and send the request
  req.write(JSON.stringify(message));
  req.end();
});

  // HTTP request options
  var options = {
    method: 'PUT',
    host: 'localhost',
    port: 5984,
    path: '/calendar/' + message._id,
    headers: {
      'content-type': 'application/json',
      'if-match': message._rev
    }
  };

  // The request object
  var req = http.request(options, function(response) {
    console.log("Got response: %s %s:%d%s", response.statusCode, options.host, options.port, options.path);

    // Accumulate the response and publish when done
    var data = '';
    response.on('data', function(chunk) { data += chunk; });
    response.on('end', function() {
      var couch_response = JSON.parse(data);
      client.publish('/calendars/changes', couch_response);
    });
  });

  // Write the PUT body and send the request
  req.write(JSON.stringify(message));
  req.end();

Got response: 201 localhost:5984/calendar/66543e3457df7597f0e41764e500067c
{ ok: true,
  id: '66543e3457df7597f0e41764e500067c',
  rev: '3-243c4d7084fcdcb7c728917a73e94b97' }

Got response: 409 localhost:5984/calendar/66543e3457df7597f0e41764e500067c
{ error: 'conflict',
  reason: 'Document update conflict.' }

Got response: 409 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ error: 'conflict',
  reason: 'Document update conflict.' }

    faye.subscribe('/calendars/changes', function(message) {
      console.log('[/calendars/changes]');
      console.log(message);

      var model = calendar.appointments.get(message);
      model.set({rev: message.rev});
      model.set({_rev: message.rev});
    });

Got response: 201 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ ok: true,
  id: 'cd823d4aaaf358069f9a800410000b95',
  rev: '7-55b3b95b6996072aa1519b6070cf12b6' }

Got response: 201 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ ok: true,
  id: 'cd823d4aaaf358069f9a800410000b95',
  rev: '8-a06160ea8bef50bd71d92a73db61c14b' }

Got response: 201 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ ok: true,
  id: 'cd823d4aaaf358069f9a800410000b95',
  rev: '8-a06160ea8bef50bd71d92a73db61c14b' }
Got response: 409 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ error: 'conflict',
  reason: 'Document update conflict.' }

Got response: 201 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ ok: true,
  id: 'cd823d4aaaf358069f9a800410000b95',
  rev: '9-c09e4a0a8bbd5cc25a9e67b19b6a254b' }
Got response: 409 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ error: 'conflict',
  reason: 'Document update conflict.' }
Got response: 409 localhost:5984/calendar/cd823d4aaaf358069f9a800410000b95
{ error: 'conflict',
  reason: 'Document update conflict.' }

Earlier this month, I gave a presentation at GoGaRuCo about using CouchDB, and how many people do it in inappropriate or wasteful ways.

The centerpiece of my talk was a step-by-step example of best-practices for CouchDB interaction, leading to an improvement of 99% in the median time it takes to do a simple insert to CouchDB (or Cloudant). Admittedly, I start from a worst-case scenario, but the scary part is that it is not a totally unrealistic one; in fact, I’ve seen it replicated many times over.

After factoring my advice in, we have a performance improvement of 96% (if you stick to conversing with CouchDB through JSON, which is recommended for now), or 99% with my experimental patches to support MessagePack in CouchDB & CouchRest.

What’s more, none of the advice I give leads to a relaxation in consistency or durability. It’s plain optimization, no strings attached, just there for the taking. Many of the points I touch on are also valid no matter what language or library you are using (or are easily translatable to other ecosystems), so I would strongly encourage anybody using CouchDB and interested in performance optimization to look at the video.

Head on over over to Confreaks to see the video (the meat of the talk starts at the 5:10 mark).

The purpose of business analytics is to find data from the company's information systems that can be used to support decision making. What customers buy most? What do they do before a buying decision? What are the signs that a customer may be leaving?

For the last month we've been working in Salzburg to build such a system, the Intelligent Project Controlling Tool needed for running large collaborative research projects like IKS. Since the design we went with can be reused for other business analytics needs, I wanted to write a bit about it.

But first, here is how our system looks like:

Where does the data come from?

There are many ways to gather business data. Often the information systems already contain the data needed. But it may also be hidden in a jungle of spreadsheets. Or maybe some data is simply not available, and has to be filled in manually.

Handling all these cases in one system is a tricky question. To solve it, we went with a two-layered strategy:

• All data used for analytics is stored as Linked Data in a CouchDB system
• NoFlo workflows are used for gathering data from the diverse sources and convert it to the format needed

In IKS's case, much of the data was available in a series of spreadsheets. With these, we built the necessary workflows for first converting the spreadsheets into XML with Apache Tika, and then extracting the information from them in a sensible subset of JSON-LD.

Because IKS is a collaborative project, information needs to be gathered from a diverse group of partner organizations. Some of them have systems that provide the needed APIs (like Basecamp, which we use), and we can just periodically import the data. But with many we decided on a simple data interchange approach: spreadsheets handled over email.

In this approach, user files a data request into the system. This gets picked up by NoFlo, which sends an email with the appropriate spreadsheet template to the partner. Then it starts waiting for a reply. When a reply arrives, it extracts the data from the attached spreadsheet and imports it to the system.

Our NoFlo processes are mostly initiated by the CouchDB change notification API. We keep them running persistently using forever Node, so whenever some operation needs to be run it happens nearly immediately.

Ensuring data consistency

With any automation, and especially with the email-based data interchange, things can go wrong. Because of this we tag all data that we receive with its origin, whether it was some automated operation or an imported spreadsheet. These origins are called execution documents. Users can browse all completed workflow executions and see what data came in from them. These can then be either accepted or rejected.

This way if some partner accidentally sends faulty data, or something else breaks, the incorrect information received can be easily removed. CouchDB's versioning capabilities help here.

Analyzing the data

CouchDB is built on top of the concept of map/reduce. Here you can modify and combine the data in lots of different ways using simple JavaScript functions. In our case we elected to write all our CouchDB code in CoffeeScript for simplicity. For example, here is the reduce function in CoffeeScript that counts totals of time planned, time used, and time left per task or partner in a project:

(keys, values, rereduce) ->
    roundNumber = (rnum, rlength) ->
        Math.round(parseFloat(rnum) * Math.pow(10, rlength)) / Math.pow(10, rlength)
    data =
        planned: 0.0
        spent: 0.0
        left: 0.0

    if rereduce
        for reducedData in values
            data.planned += reducedData.planned
            data.spent += reducedData.spent
        data.left = data.planned - data.spent
        return data

    for doc in values
        if doc['@type'] is 'effortallocation'
            data.planned += roundNumber doc.value, 1
        if doc['@type'] is 'effort'
            data.spent += roundNumber doc.value, 1
    data.left = roundNumber data.planned - data.spent, 1
    return data

If you figure out a new way to look at the data you have, simply write the needed map and reduce functions and save them into the database. CouchDB will then run them against existing data and produce numbers.

Data visualizations

Numbers are good, but to really see the information buried in them you need some visualizations. For this we decided to follow the CouchApp idea where the user interface code is stored in the database together with the data itself. This way no application servers are needed, and you can take the whole system with you just by replicating the database. Think of the possibility of doing some analysis on your company while flying to a meeting!

The visuals are in our case provided by JavaScript InfoVis Toolkit, a nice, MIT-licensed interactive graph library.

CouchDB views handle the number crunching, then CouchDB list functions process the numbers into the format needed for visualization. This leaves only a minimal amount of work for the client side.

For consistency our application has been built with CoffeeApp, so all the database and user interface code is in CoffeeScript.

In a nutshell

Any business analytics system dealing with moderate amounts of data can be built following this approach.

• Apache CouchDB is the central data store
• All data is stored as JSON-LD entities
• NoFlo handles all data imports
• Analytics based on the data are done with CouchDB map/reduce
• Visualization happens with a CouchApp using JavaScript InfoVis Toolkit

This way you have a business analytics environment that is easy to extend with more data when it becomes available. New analysis can be done by writing reasonably simple map/reduce functions, and CouchDB's replication capabilities allow you to take the system and data with you.

Using JSON-LD for the data storage makes a lot of sense, as this way the relations between different pieces of information are easy to handle. And using URIs for data identifiers means you can easily mash up information coming from different sources together.

The two-layered approach of using NoFlo for data imports, and CouchDB for analysis also allows for clean separation of concerns. In our case, I did the workflow part of things, and Szaby built the visualizations.

The FOSS4G 2011 is over now. Time for a small report. The crowd was amazing and it was again the ultimate gathering of the Free and Open Source for Geospatial developer tribe. Solid presentations and great evenings.

My talk: The State of GeoCouch

I'm really happy how my talk went, I really enjoyed it. The were lots of people (although there was a talk from Frank Warmerdam at the same time) asking interesting questions at the end.

The talk is not only about GeoCouch but also gives you an overview of some of the features it leverages from Apache CouchDB. In the end you should have an overview why you might want to use GeoCouch for your next project.

You can get the slides right here.

• The slides (download optimized) as PDF (licensed under CC-BY-3.0-de).
• The slides with comments as HTML (licensed under CC-BY-3.0-de).
• The slides with audio. It’s the recording of the actual talk at the conference.

Other talks

I was happy to see that there was another talk about GeoCouch. Other talks I really enjoyed were:

• CartoSet, a new FOSS to create highly visual geo portals: So many true things were said, e.g. get the data from the customer first, before you start building anything.
• Introduction to OpenSource WebMapping: Beyond Google Maps: A very nice introduction off the full stack for web mapping. From the database, over the server to the client. It's for people that have no clue that they could have maps on their own infrastructure.
• Comparing GeoExt, MapQuery and Legato from a technical and collaborative point of view: A good comparison between GeoExt, MapQuery and Legato. I especially liked the “Bodo”-test.
• Beyond vectors: Adapting remote sensing research for environmental monitoring with open source hardware and software: Citizen mapping the BP oil spill with balloons and kites: Kite and balloon mapping is just awesome. It's kind of the OpenStreetMap for raster data.
• GeoGlobalDomination: The Musical: One of the highlights was the GeoGlobalDomination musical. Grab the video while it's hot.

And of course there were also great talks from in the plenary sessions from Paul Ramsey about Why do you do that? An exploration of open source business models and Schuyler Erle's so funny lightning talk about Pivoting to Monetize Mobile Hyperlocal Social Gamification by Going Viral

Code Sprint

At the code sprint I was working on MapQuery together with Steven Ottens and Justin Penka. Steven was working on TMS support, Justin on a 6 minutes tutorial and I on making manual adding of features possible.

The OpenLayers developers did the migration from Subversion to Git for their development. OpenLayers is now available on Github.

And luckily there was a fire alarm in between to take a group photograph.

Future of the FOSS4G

I really hope there won't be a yearly FOSS4G conference for the whole of the US. There should be regional events, as I think one big one would draw the attention away from the international conference. Why should you fly to Beijing for the FOSS4G 2012 if you can meet the majority of the developers in the US as well?

Final words

The FOSS4G was great. It was organized well and people were always out in the evenings. The only minor nitpick is that many people working remote had the city of their company in the name badge and not the one they live in. It seems that the original for you had to fill was confusing. So for next year it should perhaps say “Location where you live”. Hence I still don't believe that there were more Dutch than German people at the conference (Tik hem aan, ouwe! ;)

    window.Appointment = Backbone.Model.extend({
      // ...
      destroy: function() {
        Backbone.Model.prototype.destroy.call(this, {
          headers: {'If-Match': this.get("_rev")}
        });
      }
    });

{"ok":true,"id":"7acf98778a669f4d6fc33d6b340106de","rev":"1-21662a1368aa1592d1e5d1df710f6d8c"}

{
   "_id": "7acf98778a669f4d6fc33d6b340106de",
   "_rev": "1-21662a1368aa1592d1e5d1df710f6d8c",
   "title": "Delete me #2",
   "description": "asdf",
   "startDate": "2011-09-15"
}

      destroy: function() {
        Backbone.Model.prototype.destroy.call(this, {
          headers: {'If-Match': this.get("_rev") || this.get("rev")}
        });
      }

    window.Appointment = Backbone.Model.extend({
      get: function(attribute) {
        return Backbone.Model.prototype.get.call(this, attribute) ||
               Backbone.Model.prototype.get.call(this, "_" + attribute);
      },
      // ...
    });

    window.Appointment = Backbone.Model.extend({
      get: function(attribute) {
        return Backbone.Model.prototype.get.call(this, attribute) ||
               Backbone.Model.prototype.get.call(this, "_" + attribute);
      },
      destroy: function() {
        Backbone.Model.prototype.destroy.call(this, {
          headers: {'If-Match': this.get("rev")
        });
      },
      // ...
    });

<script type="text/template" id="calendar-event-template">
  <span class="event" title="<%= description %>">
      <%= title %>
      <span class="delete">X</span>
  </span>
</script>

window.EventView = Backbone.View.extend({
      // ...
      events: {
        'click .delete': 'test'
      },
      test: function() {console.log("delete")},
    });

window.EventView = Backbone.View.extend({
      // ...
      events: {
        'click .delete': 'deleteClick'
      },
      deleteClick: function() {
        this.model.destroy();
      },
      remove: function() {
        $(this.el).find('.event').remove();
      }
    });

e.destroy({headers: {'If-Match':e.model.get("_rev")} })

    window.Event = Backbone.Model.extend({
      // ...
      destroy: function() {
        Backbone.Model.prototype.destroy.call(this, {
          headers: {'If-Match': this.get("_rev")}
        });
      }
    });

!!!
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    link(rel='stylesheet', href='/stylesheets/blitzer/jquery-ui.css')
    script(src='/javascripts/jquery.min.js')
    script(src='/javascripts/jquery-ui.min.js')
    script(src='/javascripts/underscore.js')
    script(src='/javascripts/backbone.js')
  body!= body

#dialog(title="Add calendar event")
  #calendar-event-start-date
  p title
  p
    input#calendar-event-title(type="text", name="title")
  p description
  p
    input#calendar-event-description(type="text", name="description")

script
  $(function() {
    $('#dialog').dialog({
      autoOpen: false,
      modal: true,
      buttons: [
        { text: "OK",
          click: function() { $(this).dialog("close"); } },
        { text: "Cancel",
          click: function() { $(this).dialog("close"); } }
      ]
    });
  });

    window.AppView = Backbone.View.extend({
      el: $("#dialog"),
      events: {
        'click .ok':  'create'
      },
      create: function() {
        console.log("here");
        Events.create({
          title: "foo",
          description: "bar",
          startDate: "2011-09-01"});
      }
    });

    window.AppView = new AppView;

$('#dialog').dialog('open')

    window.AppView = Backbone.View.extend({
      el: $("#dialog").parent(),
      // ...
   });

script
  $(function() {
    $('#dialog').dialog({
      autoOpen: false,
      modal: true,
      buttons: [
        { text: "OK",
          class: "ok",
          click: function() { $(this).dialog("close"); } },
        { text: "Cancel",
          click: function() { $(this).dialog("close"); } }
      ]
    });
  });

app.post('/events', function(req, res){
  var options = {
    method: 'POST',
    host: 'localhost',
    port: 5984,
    path: '/calendar',
    headers: {'content-type': 'application/json'}
  };

  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });

  couch_req.write(JSON.stringify(req.body));
  couch_req.end();
});

Got response: 201 localhost:5984/calendar
Got response: 409 localhost:5984/calendar/7acf98778a669f4d6fc33d6b3400e480
Got response: 200 localhost:5984/calendar/_all_docs?include_docs=true

app.delete('/appointments/:id', function(req, res){
  var options = { /* Connection Options */  };

  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });

  couch_req.end();
});

Got response: 409 localhost:5984/calendar/7acf98778a669f4d6fc33d6b3400e480

If any error is encountered during the request (be that with DNS resolution, TCP level errors, or actual HTTP parse errors) an 'error' event is emitted on the returned request object.

HTTP/1.1 200 OK
X-Powered-By: Express
Connection: keep-alive
Transfer-Encoding: chunked

"error":"conflict","reason":"Document update conflict."}

  // ...
  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    res.statusCode = couch_response.statusCode;
    couch_response.pipe(res);
  }). // ...

    window.AppointmentView = Backbone.View.extend({
      initialize: function(options) {
        this.container = $('#' + this.model.get('startDate'));
        options.model.bind('destroy', this.remove, this);
        options.model.bind('error', this.deleteError, this);
      },
      deleteError: function(model, error) {
        // TODO: blame the user instead of the programmer...
        if (error.status == 409) {
          alert("This site does not understand CouchDB revisions.");
        }
        else {
          alert("This site was made by an idiot.");
        }
      },
      // ...
    });

...

  tr#week4

    td.sunday

    td.monday 22

    td.tuesday

    td.wednesday

    td#2011-08-25.thursday

    td#2011-08-26.friday

    td.saturday

...

{"total_rows":2,"offset":0,"rows":[

  {"id":"fdbed27594feb433c74e82eb910015e0",

   "key":"fdbed27594feb433c74e82eb910015e0",

   "value":{"rev":"2-b7c22d428e648a6cdd2978c213f79ec0"},

   "doc":{"_id":"fdbed27594feb433c74e82eb910015e0",

          "_rev":"2-b7c22d428e648a6cdd2978c213f79ec0",

          "startDate":"2011-08-25",

          "title":"create blog post",

          "description":"talk about node and CouchDB"}},

  {"id":"fdbed27594feb433c74e82eb91001f45",

   "key":"fdbed27594feb433c74e82eb91001f45",

   "value":{"rev":"1-2b18432cf6e63b82c6507ff28af9724c"},

   "doc":{"_id":"fdbed27594feb433c74e82eb91001f45",

          "_rev":"1-2b18432cf6e63b82c6507ff28af9724c",

          "startDate":"2011-08-26",

          "title":"blog again",

          "description":"add backbone into the node + couch mix"}}

]}

  $(function() {

    $.getJSON('/events', function(data) {

      $.each(data.rows, function(i, rec) { add_event(rec.doc) });

    });

  });

  function add_event(event) {

    var date = event.startDate,

        title = event.title,

        description = event.description;



    $('#' + date).html(

      '<span title="' + description + '">' +

        title +

      '</span>'

    );

  }

app.delete('/events/:id', function(req, res){

  var options = {

    method: 'DELETE',

    host: 'localhost',

    port: 5984,

    path: '/calendar/' + req.params.id

  };



  // Send the HTTP request with the DELETE options

  var couch_req = http.request(options, function(couch_response) {

    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);



    // Pipe the response from CouchDB to the browser

    couch_response.pipe(res);

  }).on('error', function(e) {

    console.log("Got error: " + e.message);

  });



  // Send the complete request.

  couch_req.end();

});

> e.destroy()

     => child

window.Event = Backbone.Model.extend({});

    window.Event = Backbone.Model.extend({

      urlRoot : '/events',

      initialize: function(attributes) { this.id = attributes['_id']; }

    });

{

   "_id": "a38f51509190f265959bbb2b5d001128",

   "_rev": "1-174f31204df52e79a92c1ac875ac09a2",

   "startDate": "2011-09-01",

   "title": "foo",

   "description": "bar"

}

Got response: 409 localhost:5984/calendar/a38f51509190f265959bbb2b5d001128

DELETE /calendar/a38f51509190f265959bbb2b5d001128?rev=1-174f31204df52e79a92c1ac875ac09a2 HTTP/1.0

DELETE /calendar/a38f51509190f265959bbb2b5d001128 HTTP/1.0

If-Match: "1-174f31204df52e79a92c1ac875ac09a2"

app.delete('/events/:id', function(req, res){

  var options = {

    method: 'DELETE',

    host: 'localhost',

    port: 5984,

    path: '/calendar/' + req.params.id,

    headers: req.headers

  };



  var couch_req = http.request(options, function(couch_response) {

    // ...

  couch_req.end();

});

> e.destroy({headers: {'If-Match':'1-174f31204df52e79a92c1ac875ac09a2'} })

Got response: 200 localhost:5984/calendar/a38f51509190f265959bbb2b5d001128?rev=1-174f31204df52e79a92c1ac875ac09a2

➜  repos  express calendar   

   create : calendar

   create : calendar/package.json

   create : calendar/app.js

   create : calendar/public/stylesheets

   create : calendar/public/stylesheets/style.css

   create : calendar/public/javascripts

   create : calendar/public/images

   create : calendar/views

   create : calendar/views/layout.jade

   create : calendar/views/index.jade

➜  calendar git:(master) ✗ npm install express jade

jade@0.14.2 ./node_modules/jade 

express-unstable@2.4.3 ./node_modules/express

├── mime@1.2.2

├── connect@1.6.0

└── qs@0.3.1

app.get('/events', function(req, res){

  var options = {

    host: 'localhost',

    port: 5984,

    path: '/calendar/_all_docs'

  };



  http.get(options, function(couch_response) {

    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);



    res.contentType('json');



    // Send all couch data to the client

    couch_response.on('data', function (chunk) {

      res.write(chunk);

    });



    // When couch is done, so is this request

    couch_response.on('end', function (chunk) {

      res.end();

    });

  }).on('error', function(e) {

    console.log("Got error: " + e.message);

  });

});

  http.get(options, function(couch_response) {

    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);



    couch_response.pipe(res)

  })

app.get('/events/:id', function(req, res){

  var options = {

    host: 'localhost',

    port: 5984,

    path: '/calendar/' + req.params.id

  };



  http.get(options, function(couch_response) {

    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);



    couch_response.pipe(res);

  }).on('error', function(e) {

    console.log("Got error: " + e.message);

  });

});

Really excited about a new project I started recently to enable phone-based speech recognition for 311 service requests.

Here is a screen cast demonstrating the solution.

a

This blogpost originally appeared as “NoSQL: CouchDB with CloudBees and Cloudant” on the CloudBees Blog.

In this tutorial I will build a CouchDB based url shortener service (http://cloudbe.es) using CloudBees and the services/add-on platform with Cloudant. CouchDB is a popular NoSQL database - specifically you could called it a “document database” - i.e. it is all about storing documents which consist of sets of fields in a JSON document against a key. There is no schema - a document can consist of any JSON structure - there is no requirement that the same fields be present in each document unlike rows in a RDBMS. Of course CouchDB has a lot of the other benefits of NoSQL databases - high scalability and ease of use. Cloudant provides an excellent hosted CouchDB service. This takes care of all the management of CouchDB you would normally need to do - compaction, backups, and more.

Building a URL shortener

A url shortener is just that - takes a long messy URL and generates a token/shorter URL you can pass around.

Step 1: Get a cloudbees.com account, signup to a free RUN@cloud service

Step 2: Go to the subscription page — and choose Cloudant

Step 3: Create a new web app skeleton

For this I will use the Play! Framework - a simple framework for building RESTful web apps. I can use the command

play new urlshort --with scala

This will create an empty project using scala. To deploy to CloudBees- a handy way is to add cloudbees 0.2.1 to the dependencies file (don’t worry - source repo will be available to look at).

Step 4: Create an empty database

From grandcentral.cloudbees.com/services you can navigate to your Cloudant/CouchDB console. Any time you need to get back to the CouchDB console you can navigate from grandcentral.

This shows the “Futon” web interface to browse your data, quite a handy tool. Here you can create a new database - I called mine “urly”. Once you have a database, at the top left corner you can see an icon which will take you to the Cloudant management screen - click through to the newly created database, and you will be able to set permissions, and generate an api key:

The important bit is to note the api key details (a user name and password that you will use in your app). The user just generated should also get read/write access to your database (set it with the checkboxes).

Step 5: Set up credentials in your application config

In this case, as I am using the Play! framework - the configuration is in application.conf. The keys I set up were:

    #CloudBees details
    bees.api.key= key for deploying
    bees.api.secret= secret for deploying
    bees.api.domain=michaelvideo
    bees.api.name=urly

    #Cloudant details
    couch.user=api key user
    couch.password=api key password

I put the CloudBees details in there for easy deployment but technically you don’t have to (you can pass them to command line if you like).

Step 6: Talking to CouchDB

One of the really nice things about CouchDB is that you don’t need any drivers or dependencies to talk to it — https is a “native” api for it. So all that is needed is a simple http/REST/WS client.

Urls are how you address content - you can store a document in a nominated key for example by PUT’ing a JSON body to https://michaelvideo.cloudbees.cloudant.com/urly/ (urly is the name of the database). Updates to data require you to specify the current version tag of the data (field in the JSON body) as a form of optimistic locking.

Getting the data back is a simple GET ! (obviously there is a lot more you can read about if you need). In this case the keys are the randomly generated “slugs” that form the short URL.

Step 7: Wire up the application

Descending from the mountain top, a simple codebase which generates random url “slugs” and then does appropriate http redirects was carved in stone (also available on github):

Authentication with CouchDB is using http basic over SSL (using the api keys mentioned earlier).

Step 8: Deploy

(This will use the Play! CloudBees plugin - thanks Ivan !)

I then set up a domain and a record pointing to the host for the app, running at http://cloudbe.es/

Note that there was no need to migrate any data, or setup a schema (just create the database by name - which I could have also done programmatically).

The data is safely stored in CouchDB (below is browsing it with Futon):

MapReduce and Views

A further enhancement is to create views. CouchDB uses a Map Reduce model to provide views and aggregate calculations.

A view is similar to what relational databases offer and only requires a map function to be provided. Reducing is a more complicated topic and I won’t show it, but it could be used, for example, to calculate aggregates (sum or average over some range of documents). The advantage of a Map Reduce model is that the calculation can be distributed and run in parallel with the data - good for very large sets of data.

The Map function above simply takes a document and emits another document which is made up of the slug and the agent string (each browser provides an agent string). The key in this case is the url we stored - this means we can (if we want) lookup slugs based on URL (the reverse of what we were doing before). Note that as the fields in each document are optional - even if there is no agent field the above view will still work (just will have no agent field in the view output).

The functions are written in javascript - and are themselves stored as documents (with a special naming convention). The results of these views are accessed the same as the normal documents you write data to.

Hopefully from this you get a taste of the convenience and power of CouchDB and Cloudant.

You can sign up for both CloudBees (if you don’t have an account) and the Cloudant service on the CloudBees website.

As we just announced, Cloudant has added fault-tolerant, federated full-text search to our document database offering.

Cloudant Search builds upon our existing database to offer full-text indexing and search. It makes use of the Apache Lucene query syntax and large portions of the Lucene framework, but uses a new data storage model entirely within the CouchDB database framework. The inverted index (the map of terms to documents in which they appear) is calculated using our incremental map-reduce framework. This means the database is always available for document insertion and reading even if the indexing task backs up - other search solutions only allow documents to be added as quickly as they can be indexed. Newly indexed documents are immediately searchable – there is no latency while the inverted indices are rebuilt.

Documents within Cloudant are distributed to multiple nodes using consistent hashing in a ring like structure. The inverted index data associated with each document is saved on the same node as the document itself. Searches are done at the shard level, and results are combined in a federation layer. If a particular node in the cluster is offline, other copies of the shard are used. Cloudant utilizes the same underlying distribution and federation technology from the open-source Bigcouch project.

Important features of Cloudant Search include:

• Text analysis can utilize existing Java based Lucene analyzer. Java code can be loaded into the database and stored along with the search indices.
• Support for indexing in many languages (Java, Javascript, python, erlang, ruby, etc.) User indexing code is also stored in database.
• Ability to run ad-hoc queries against multiple independent map-reduce views of data.
• Full support for all JSON types during indexing (Strings, Numbers, JSONObjects). Pure Lucene search solutions only support String tokens.
• Support for following query types: Boolean, Range (text, numerical, date, and compound key ranges), Phrase, Sorted, Prefix, Dates
• Ability to query a “stale” snapshot of the database in case where indexing has not caught up to document insertion
• Horizontal scalability through addition of database nodes
• HTTP based search API consistent with CouchDB view API.

We see several distinct use cases where Cloudant Search will be superior to existing products. Since Cloudant runs as both a low cost hosted service and as a dedicated deployment for the largest customers, developers can build products around Cloudant Search and know that they will scale to terabytes of indexed data and high IOPS rates. An ideal use case would be mobile applications that require real-time geographical bounding box search coupled with text queries. Cloudant Search’s native support for numerical index types allows this. Cloudant Search’s reliance on CouchDB views also opens up unique possibilities. Multiple algorithms can be used simultaneously to search, score and sort documents.

To get started with Cloudant search, you first need to specify how your database is indexed by uploading a new CouchDB design document (indexing instructions). Then you can use the search API. There is also a full description of the query syntax.

I’ve always strongly felt that using NOSQL wasn’t so much a choice as a necessity. That most successful NOSQL deployments start with the intimate knowledge that your set of requirements — from speed & availability to operational considerations and budget — cannot be met with a relational database, coupled with a deep understanding of the tradeoffs you are making. Among those, perhaps no tradeoff has been felt more deeply by NOSQL users worldwide, than the eponymous loss of a natural, instantaneous way of accessing your data through a structured query language. We all came up with our own remedies; more often than not, that substitute was based on MapReduce: Google’s novel, elegant way of explicitly parallelizing computation over distributed, unstructured data. But as the joke goes, it’s always been a non-starter for the more novice users out there, and where suits & ties are involved.

CouchDB Views (as our brand of MapReduce is called) come with additional concerns, as they are pre-computed and written to disk. While this is fine — and actually, extremely useful — for the use-cases and small scales a lot of Apache CouchDB deployments reside at (single instances working off a limited dataset), this behavior is somewhere North of nagging and South of suicidal for the data sizes & use-cases most Cloudant customers have to deal with. Part of the promise of our industry is — or should be, anyway — to make your life & business easier, no matter how much data you have. And so, while CouchDB Views have been, and will undoubtedly remain, an essential tool to index, filter & transform your data, once you know what to do with it; and while its various weaknesses (explicitly parallelized syntax, lengthy computation, heavy disk usage) are also the source of its most meaningful strengths (distributed processing, high performance on repeated queries, persistent transformations), we at Cloudant saw a clear opportunity to offer a novel, complementary way to interact with your data.

A way that would allow you to interact with your data instantaneously; wouldn’t force you to mess around with MapReduce jobs or complex languages; a way that would not require you to set up a third-party, financially or operationally expensive solution.

We call this way Cloudant Search. And today, we’re proud to announce its immediate availability, as a public beta.

Want to easily find all the documents that contain the word “bieber”? This is the Cloudant Search query you have to write:

bieber

Want to find all the records that have “my world” in their title? Just write:

title:"my world"

How about only finding all the artists who were born in Canada in February & March 1994?

type:artist country:canada dob<date>:[1994-02-01 TO 1994-03-31]

How about people who are fan of either Justin Bieber or Justin Timberlake, that live in JBiebz’ hometown in Ontario — wait, what’s it called, Stratwood? Stratburg? Strat- something. Oh, I’ll just search it:

type:person fan-of:(justin (bieber OR timberlake)) city:strat* state:ontario

And that’s just scratching the surface and you can see more query examples in our knowledge base. Of course, all those queries are also served instantly, and new records you put in your database will be indexed in real time.

But Cloudant Search goes beyond ease of access for developers, analysts & end-users.

For the IT & DevOps teams out there, and for our own sake (since, as maintainers of hosting service cloudant.com, we have the distinction of being the only NOSQL company tasked with operating its own product around the clock), we worked tirelessly to make it capable of tackling terabytes and easy to maintain under load. It is baked in directly into the Cloudant codebase and leverages existing, proven technology, such as the Lucene indexers, CouchDB’s view indexing logic and of course, Cloudant’s own Dynamo-inspired distribution algorithms. Compared to external search & analytics solutions such as Hadoop, SAS, Solr, Sphinx or ElasticSearch, this integration & simplicity effectively translates to more flexibility to address ever-changing use-cases, a drastically reduced operational complexity and, at the end of the day, a much lower total cost of ownership for start-ups and established businesses alike.

Finally, for the power-users out there, we’ve made Cloudant Search extremely extensible. As our documentation details, you can write your own indexing algorithms, leverage Lucene tokenizers or even, customize the Cloudant Search indexer itself, with a bit of Java code.

Now, I’d be engaging in doublespeak if I didn’t make it clear that Search is still bound by some of the tradeoffs that epitomize our industry. Like some of our esteemed colleagues, we strive to make our users & customers aware of the adjustments that, yes, define our weaknesses but more importantly, are also the source of our strengths. Among those, I should mention that Cloudant Search will require an initial, one-time indexing and this single index itself will take up disk space. To limit resource consumption and self-induced podiatric harm, wildcard (“som*thing”) or suffix (“*thing”) searches are not allowed (only prefix searches like “some*”). And yes, some of its more advanced features like the dynamic sorting of results, are still memory-bound and as such, should be used carefully.

There are no silver bullets in this industry. That is something that the Cloudant team has kept in mind through the months we spent developing Cloudant Search. But more than an admission of eventual defeat, we’ve always taken it as a challenge to innovate at the pain-points. To explore the areas beyond the boundaries of our existing trade-offs. To find tasteful solutions to gritty problems. Today, we’re happy to bring you the fruits of the labor, share our response to this problem, and in a way, pass the challenge on to you: we can’t wait to see what you will do with it. We’re confident that Cloudant Search will open new doors, from user-facing analytics dashboards, to efficient Object-Document Mappers, to killer apps we haven’t even dared to think of.

So go ahead and read more about the technology, peruse the docs, activate Search on a database through your cloudant.com dashboard or contact us to see how this technology can be deployed in your datacenter.

Starting today, all Iris Couch accounts have several new features.

CoffeeScript

We support CoffeeScript.

Our founding mantra is “orthodox CouchDB”; however, I’ve bent the rules, backporting from Apache CouchDB trunk, keeping parity with Couchbase Single Server.

Here is a simple CoffeeScript design document, formatted for clarity.

{ "_id"     : "_design/coffee_app"
, "language": "coffeescript"
, "shows"   : { "hello_world": "(doc) -> 'Hello, #{doc._id}!'" }
, "filters" : { "has_foo"    : "(doc) -> doc.foo"              }
, "views":
  { "foo": { "map"   : "(doc) ->
                          if doc.foo
                            emit(doc.foo, 1)"

           , "reduce": "(keys, values, rereduce) ->
                          sum = 0
                          for x in values
                            sum = sum + x
                          sum"
           }
  }
}

Mobile Futon

1. Get out your iPhone or Android.
2. Go to Futon: https://example.iriscouch.com/_utils/
3. Nice, isn’t it? That is Dale Harvey’s Mobile Futon.

You can disable Mobile Futon by clicking the Configuration link in Futon. Under httpd, set mobile_futon to false.

Sammy Futon

We heard the rumors too: an elusive “alternative Futon.” It has neither name nor master. Some say it evolved from proper Futon. Others say “devolved” would be more accurate.

In any case, we’ve captured a live specimen, installed it alongside its cousin, and named it Sammy Futon to sound adorable.

You can enable Sammy Futon by clicking the Configuration link in Futon. Under httpd, set sammy_futon to true.

Facebook Authentication

Iris Couch now supports Ocasta Labs’s excellent Facebook Authentication module. Your users can log in to your application using their Facebook account.

This is not quite turn-key. (What can I say? It’s Facebook.) To use CouchDB Facebook authentication, visit the Configuration link in Futon, and input your information in the [fb] section. See Martin’s documention, under Configuration.

Feedback

If you have feedback or issues, please let us know at the Iris Couch Get Satisfaction forum, or email us@iriscouch.com.

Thanks

Other people wrote this code; we simply take the credit. We are grateful to the following people for their inspiring CouchDB projects.

• Jan Lehnardt at Couchbase, for implementing CoffeeScript support
• Martin Higham at Ocasta Labs, and Sander Dijkhuis for Facebook authentication; and Ed Moore at Ocasta Labs for helping me understand it all
• Dale Harvey at Couchbase, for Mobile Futon
• Benjamin Young, Dale Harvey (both at Couchbase), Sam Bisbee, and Mikeal Rogers (now at Yammer) for Sammy Futon

Derrick Harris and Michael Stonebreaker say Facebook’s MySQL is a fate worse than death.

Indeed, it seems Facebook's Burden is worthy of our pity.

Not that it’s necessarily Facebook’s fault, though. Stonebraker says the social network’s predicament is all too common among web startups that start small and grow to epic proportions.

Oh, cry me a bleeding river, Mr. Stonebreaker.

What’s too common is brilliant web startups that start small and die smaller.

Time-to-market. Easy. These are what businesses need. That is why SQL remains competitive and compelling. Scalability is worthless until you’ve got a superior getting-started pitch.

We are glad to hear that Paul Davis, long-time Apache CouchDB committer and community leader, is joining Cloudant.

Cloudant is just brilliant. They are building awesome software and services around Apache CouchDB. May Paul find his new role rewarding, and may Cloudant continue its brilliance.

The NOSQL movement originally found its community roots in some the “edgier” platform & languages of this world; CouchDB in particular, has always felt strongly backed by a certain kinship with the Ruby and Rails communities, amongst others. But as one of the most mature NOSQL projects out there, CouchDB now also enjoys energetic support in the more established communities such as PHP, Java or even .NET.

As it turns out NOSQL isn’t the only movement kindling a relationship with the .NET framework. The “Cloud” has also taken root in that community, and a service called AppHarbor has arisen to meet the demand for reliable, elastic & pay-as-you-grow .NET hosting.

I’d be beating around the bush if I introduced AppHarbor as anything else but “Heroku for .NET”. That description may sound catchy but in and of itself, does not a great service make. No, what really caught our eye about our fellow ex-YCombinator brethren Michael, Rune & Troels, is their flawless execution of the concept and their tactful tweaks to the Platform-as-a-Service model. By giving you access to the Common Language Runtime and the .NET framework, AppHarbor opens more doors for developers than just single-language execution. You instantly gain access to the years of loving care, performance tuning and functional extensions that Microsoft has poured into its platform, as well as support for a flurry of languages.

So today, we’re happy to announce the Cloudant add-on for AppHarbor. It will give you instant provisioning of our hosted CouchDB solutions, for all your hosted .NET apps. Your Cloudant databases will be co-located in the same datacenter as AppHarbor, so you can enjoy the same reliability & performance you’ve come to expect from Cloudant, with the absolute minimal latency possible. And on the business side, you’ll also get the accounting convenience of a single monthly bill for both your app & data hosting expenses. (Don’t forget to put a coversheet on that TPS report before running it by the boss, though!)

So go on ahead and get started with one of the many CouchDB libraries available for the .NET framework, such as Divan, Relax, Hammock or EasyCouchDB. Or peruse this recent introduction to CouchDB for .NET developers. (And please send us an email to show us what you’ve built with AppHarbor + Cloudant!)

What are you still doing here? Go sign up for AppHarbor… You’re in good hands with Michael, Rune & Troels.

We are thrilled to announce that noted Apache CouchDB community member & committer Paul Davis will be joining Cloudant in July.

Paul had been on our “must-hire” list for a while now, and he will be joining a technical team that now includes a third of the currently active committers to Apache CouchDB! We’ve always tried to be faithful stewards and dedicated supporters of the Open Source community behind CouchDB, and this commitment will only grow stronger with Paul’s inclusion in Team Cloudant.

Paul’s background in large distributed systems & data analytics, combined with his expertise in Erlang and hardware optimization will undoubtedly come in handy as he helps us solidify our quality of service, add features to our platform and get BigCouch ever closer to 1.0.

You can learn more about Paul through his github, twitter or website.

Bottoms up, Paul. We couldn’t be happier to have you working with us.

Yesterday, Cloudant Chief Scientist (and VP of powder-blue shirts) sat on a panel at GigaOm’s Structure Conference. Along with representatives from Xeround, ParAccel, and NimbusDB, Mike discussed the ins and outs of running databases in a Cloud environment. As the other three panelists represented more traditional relational and data-warehousing tchnologies, they turned to Mike to represent NOSQL databases as uniquely suited for high performance, elasticity, and operational stability.

We at Cloudant are delighted to have been chosen for this panel. This is the third year in a row we have sent delegates to Structure and we have always found the speakers, panels, and hallway conversations to be very enlightening.

You can watch a video of the panel, or read PCWorld’s writeup.

Today, Cloudant is proud to be a launch partner of the CloudBees Ecosystem. Effective immediately, our line of hosted CouchDB cloud solutions is available directly through CloudBees’ Java platform, with a simple API, unified billing and integrated account management.

CloudBees started out in 2010, with the goal of giving Java developers & organizations a platform, that not only lets them run their applications online easily, but also gives them specific tools to develop, test & integrate them. This twist in the usual PaaS model means they’re the only platform out there that aims to cover the entire lifecycle of traditional Java application development. So today, we’re thrilled to be part of this vision, by helping them launch an add-on program that also features New Relic, SauceLabs, Sonar & JFrog.

Considering the recent increase in CouchDB adoption in the Java community (bolstered by the existence of several drivers & persistence layers, such as our own fork of couchdb4j, ektorp, jcouchdb and jrelax), we’re more than happy we answered CloudBees call, integrated deeply into their platform, and we can’t wait to see what you will build with us!

So go on ahead and check out the details of their announcement or sign up for the CloudBees platform right away!

Update, 2011-06-16, 12:15 AM Thanks for the comments.

(I swear I had something like that before and it didn't work!) Here's the solution:

$userId = 61382;
$docId  = 'CLD2_62e029fc-1dae-4f20-873e-69facb64a21a';
$body   = '{"error":"missing","reason":"problem?"}';

$client = new Zend_Http_Client;
$client->setAdapter(new Zend_Http_Client_Adapter_Test);

$couchdb = $this->getMock(
    'CouchDB',
    array('makeRequest',),
    array($userId, $this->config,)
);
$couchdb->expects($this->once())
    ->method('makeRequest')
    ->will($this->returnValue(new Zend_Http_Response(200, array(), $body)));
$couchdb->setHttpClient($client);
$couchdb->getDocument($docId);

--- Original blog entry ---

I wrote a couple tests for a small CouchDB access wrapper today. But when I wrote the implementation itself, I realized that my class setup depends on an actual CouchDB server being available and here my journey began.

Example code

Consider the following example:

My objective is not to be able to test any of the protected methods directly, but to be able to supply a fixture so we don't have to setup CouchDB to run our testsuite. My fixture would replace makeRequest() and return a JSON string instead.

Suppose you begin keeping a personal library. You’ve rented a floor from the building in the Matrix. It’s just room after room, all identical; except the big main room at the end with the source. Helmut Bakaitis is your librarian, and he can perform a couple of minor miracles.

Your library facility

The big main room holds all your stuff. There is no organization at all. It’s just piles and piles of things. You can keep scientific journals, your childhood pop-up books, your private literature, an old globe that still says U.S.S.R.—pretty much anything. Just throw it in the pile. On your way out the door, the librarian asks, “what do you call that thing?” and you’re like, “whatever.” As far as adding to your library, nothing could be simpler.

To access the stuff in your library, you’ll exploit the librarian’s miracles. The librarian can make unlimited copies of your stuff, instantly! He’s got a full arts-and-crafts kit: colored paper, popsicle sticks, scissors, and sticky tack. He can complete any project you can conceive, instantly!

The librarian keeps similar copies of your library in every normal room on the floor. Normal rooms are impeccably organized: long shelves, sorted, everything in its place. Each room’s shelves are sorted differently, depending on what you want. Pick out a room and tell the librarian how to sort the shelves. You’ll say, "This is the titles room. Everything in here is always shelved by title." Poof! It’s just like your main library, but everything is shelved by title.

Go to the next room. You say, "Everything here is shelved by author." Poof! Now you’ve got two rooms, totally organized. When you need Asimov, you know just where to find him. On and on, you can use every room on the floor, and you can make specialized rooms. You can say, "This is the personal room. It’s got my private literature and nothing else." We’ve all got a personal room; but different people’s rooms will be differently-populated. Of course, the librarian can expand or pare-down your normal-room copies: whatever you need to look things up easily.

If you like, you can alter or even destroy your stuff in the big main room. The librarian ensures that normal rooms reflect changes from the source material. For example, if you scribble corrections on your globe, you won’t find the USSR in your normal rooms anymore.

But the big main room is disorganized. How will you find anything? Fortunately, the librarian remembers it all. Just ask him, “Hey, where’s my globe?” And there’s your globe! Start scribbling.

The End.

Key (Spoilers)

Really excited to launch a new OpenGov project in Philadelphia - Phind It For Me.

The service is built on PHLAPI and the point data sets it houses. As such, one could understand why I’d be interested in enhancing the data sets currently in PHLAPI.

I’m really excited about this project - source code available on GitHub - and would love to see if there is an interest in launching in other cities with CouchDB-based geospatial data repositories, like Baltimore.

It’s built on the awesome new SMSified platform from Voxeo (disclaimer, I work there) and uses a Node.js module I built for working with the SMSified API.

As always, dear readers, any comments or feedback is welcomed.

Do head on over to the project website and check it out!

a

One of the hardest parts of running a global business like Cloudant isn’t necessarily on the technical side of things. While we certainly got better at developing & operating a fleet of clusters that now spans both US coasts & Asia, we’ll admit to still being somewhat under-skilled at tackling some of our marketing concerns, such as estimating latent demand in foreign markets.

In recent months, our support team has been receiving more & more inquiries about provisioning Cloudant accounts in Europe. But instead of trying to guess who else is interested, run complex spreadsheet models or hire a team of consultants, we figured we’d just ask you. (Yes, you: Time magazine’s 2006 Person of the Year.) To limit the noise in this poll, we’ll ask you to put your money where your mouth is: we’ll only accept reservations from paying customers (i.e., users on an “Argon” plan or above).

Having your data located a cluster close to your application layer (or other consumers of your CouchDB docs) has a big primary benefit: latency. With a cluster in Europe, your median read & write latencies to Europe should go down by approximately 200ms (or by ~500ms for SSL access), when compared to our existing East Coast cluster .

So go on ahead to our reservation page. You’ll be asked to sign up for Cloudant if you don’t already have an account; or upgrade to a paid account if you’re currently on our “Oxygen” plan.

We’ll let you know how the process is going with a handy reservation progress bar. We’ll set up our European cluster and load it with the accounts that reserved a spot, as soon as that bar hits 100%!

Go ahead and tell us you’re interested!

We are doing another iteration of our JavaScript for Designers Workshop (in German). Our designer Kristina and I (Alex) will be teaching the basics of programming in the browser for a full day. You don’t need to have any knowledge about programming. You don’t even have to be a designer

The workshop is scheduled for July 15, early bird tickets are available for €300 until June 10.

For more information visit jstraining.de.

In order to leverage existing Lucene analyzers, our Search feature (currently in beta) makes use of Java views, rather than the traditional Javascript views you’re used to in CouchDB. Recently, we wanted to do some performance comparisons between the two; would the Java view server be significantly faster (or slower) than the Javascript view server on typical indexing tasks? To test this, we wrote a simple Javascript view (in the form of a CouchApp) that creates a Cloudant Search-compatible inverted index. The code is available on github here.

To use it, clone the git repository and use the couchapp tool to push it to your database:

$ git clone git@github.com:cloudant/jsindexer.git
$ cd jsindexer
$ couchapp push http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>

With this simple push, you gain the ability to do a search against this inverted index. For example, if you want to search all the docs in that DB whose “comments” field contains “cloudant is awesome” and whose “rating” value is between 80 and 100:

$ curl 'http://<user>.cloudant.com:5984/<db_or_couchapp_you_want_to_search>/_search?q=comments:"cloudant is awesome" AND rating:[80 TO 100]&index=_design/jsindexer/_view/whitespace'

Obviously, that’s just scratching the surface. Our knowledge base has several articles about Search. You can go there to read our introduction to Search, learn more about the Search API or how to write your own Indexer.

As for the results of the performance comparison, our full text indexing speed is limited by the writing of the data to the CouchDB B-Tree, so Javascript and Java perform the same. So there you have it: full-text search from Javascript, a guilt-free geeky pleasure. Go ahead and have at it.

Iris Couch and CouchDB means easy data replication and no vendor lock-in for you. It is important to explore what that means.

Let’s walk through how you would pack up and move on. Thanks to CouchDB, it’s simple! Even if you actually like Iris Couch, this is a useful exercise:

• You’ll maintain an independent disaster-recovery or backup site.
• You’ll stay upstream. If your needs ever change, at any time, for any reason, switch couches.
• You’ll sharpen your CouchDB skills.

The Big Idea

This is a basic disaster-recovery (DR) scenario. We want to perform one simple task to completely pull the plug, jumping to a different CouchDB system.

Thus these are the main objectives:

• Have a duplicate of the Iris couch, called B-Couch.
• B-Couch syncs from Iris couch automatically or regularly.
• Be able to activate B-Couch with a single domain name change.

Quick and Dirty Sidenote

If you have no backups whatsoever, the dead-simple partial solution is to make a full document dump: /my_db/_changes?include_docs=true

This is like an incomplete mysqldump because it does not fetch document attachments. On the other hand, it’s a quick one-liner which you can run from cron, so what’s stopping you?

$ curl 'https://icuser:icsec@ex.iriscouch.com/db/_changes?include_docs=true'
{"results":[
{"seq":5,"id":"doc 1","doc":{"_id":"doc 1","_rev":"7-1f6e64...","awesome":true}, etc. },
{"seq":8,"id":"other doc","doc":{"_id":"other doc","_rev":"3-8b92cd..."}, etc. },
{ ... etc ... },
],
"last_seq":97}

This is not a complete copy of the data. Again: it will not include attachments.

However, if you have no backups in the first place, well, at least it’s a full replica of the data. Bang for buck.

Next I’ll explore a proper couch clone. Yeah. We all see what’s coming. We’re going to replicate CouchDB—no surprises there. Of course, “surprise” isn’t typically part of a good DR plan.

Preparing the Iris couch

First, you need a user account on the Iris couch. This account must have permission to read every database. In a pinch, that can be the system _admin account.

For this discussion, suppose the user name is icuser and the password is icsec, on the couch ex.iriscouch.com.

Confirm that icuser (or any of icuser’s roles) can access every database. The security object (in Futon’s “Security” section) for each database should list icuser in the “users” array, or one of icuser’s roles in the “roles” array.

In other words:

* For each database:
  * Check the _security object (the "Security" link in Futon)
  * Check the "Admins" section:
    * If Names contains icuser, ok.
    * If Roles contains any of icuser's roles, ok.
  * Check _security (the Security link in Futon) in the Readers section:
    * If Names and Roles are both empty, ok.
    * If Names contains icuser, ok.
    * If Roles contains any of icuser's roles, ok.
  * Otherwise, you must add icuser (or its role) somewhere in this mix.

You can confirm that everything works with curl.

$ curl https://icuser:icsec@ex.iriscouch.com/_all_dbs
["stuff","not_stuff"]

$ curl https://icuser:icsec@ex.iriscouch.com/stuff
{"db_name":"stuff","doc_count":92,"doc_del_count":12,... // etc.

$ curl https://icuser:icsec@ex.iriscouch.com/not_stuff
{"db_name":"not_stuff","doc_count":8,"doc_del_count":0,... // etc.

Preparing B-Couch

Obviously, you need B-Couch on-line. This can be pretty much any CouchDB server you can find or run yourself. In fact, B-Couch can be on your laptop, or a spare computer in the office. If you haven’t got any clone of your couch, by all means, start with CouchDBX, homebrew, or (if you’re sophisticated) the CouchDB that comes with your OS.

If B-Couch is private, you won’t have the DNS switchover we discuss later. But again: that’s better than nothing.

Iris Couch runs the GeoCouch plugin. An easy way to build CouchDB with GeoCouch is with Build CouchDB. (Well, you might say “an easier way.”) GeoCouch is not necessary to have a complete backup; however, if you want to use B-Couch in production with geo queries, you’ll need it.

As before, you need a Couch user with write access to the databases. You could use the _admin account. Nobody’d blame you. But a less-privileged account is better.

A good place to start is replicating the user database straightaway.

$ curl -X POST http://admin:secret@couch.b-couch.com/_replicator/      \
  -H 'Content-Type: application/json'                            \
  -d '{ "source":"http://icuser:icsec@ex.iriscouch.com/_users"}  \
      , "target": "_users"}'

Replicating the _users database will add identical account names and roles to B-Couch. If you don’t want peopla using B-Couch, then set the _security objects differently on B-Couch. See? Locked down. That’s why security objects don’t replicate.

Confirm that B-Couch responds to queries and that everything looks good.

$ curl https://icuser:icsec@couch.b-couch.com/
{"couchdb":"Welcome","version":"1.1.0"}

$ curl https://icuser:icsec@couch.b-couch.com/_session
{ "ok": true
, "userCtx": { "name": "icuser", "roles": ["some_role"] }
, "info": { "authentication_db": "_users"
          , "authentication_handlers": [ "oauth", "cookie", "default" ]
          , "authenticated": "default"
          }
}

Replicate

You did it once already and it wasn’t so bad. Do that for every database you need.

for db in admin_db _users some_other_db; do
  curl -X POST http://admin:secret@couch.b-couch.com/_replicator/    \
       -H 'Content-Type: application/json'                           \
       -d '{ "source":"http://icuser:icsec@ex.iriscouch.com/_users"} \
           , "target": "_users"}'
done

Here our paths diverge a bit. This is your back-up, DR, cold-spare, hot-spare, or whatever. The point is, you’ve got your own couch that is a clone of your Iris couch. Back it up to tape. Put it in Dropbox. Whatever.

The command above does not use continuous replication. Instead, it starts one-off replications, effective immediately. That’s easy to understand, easy to keep an eye on, and easy to back up correctly.

You might prefer a continuous replication, maintaining a real-time copy of the Iris couch. Any time, rain or shine, B-Couch will be identical to the Iris couch. If that sounds like something you want, add one more field to your replication requests:

{ "source": "http://icuser:icsec@ex.iriscouch.com/_users"}
, "target": "_users"
, "continuous": true
}

The hostname switch

Finally, the big red button!

We want one simple place where we can change one configuration or setting, and B-Couch takes over in production.

The answer is to use your own DNS domain. For example, suppose your domain is example.com. Build your applications and services to use CouchDB from couch.example.com.

; Example BIND 9 setting. This points a domain name I
; control (couch.example.com) to whichever couch I
; prefer (currently Iris Couch).
couch           CNAME   ex.iriscouch.com.

Consider the time-to-live (TTL) for your domain. When you change the CNAME, users will still connect to the Iris couch for up to TTL seconds.

Confirm that your CNAME looks good and works for queries.

$ dig couch.example.com A

;; QUESTION SECTION:
;couch.example.com.             IN      A

;; ANSWER SECTION:
couch.example.com.      712     IN      CNAME   ex.iriscouch.com.
ex.iriscouch.com.       712     IN      A       184.73.200.44

$ curl https://icuser:icsec@couch.example.com/
{"couchdb":"Welcome","version":"1.1.0"}

You're done!

Now, if you ever need to change which couch you use for production, just update the CNAME record in your DNS domain. No software changes, no downtime, no lock-in.

; Modified BIND 9 setting. This points a domain name I
; control (couch.example.com) to whichever couch I
; prefer (now B-Couch).
couch           CNAME   couch.b-couch.com.

When that pushes and propogates, the client-side (that is, your web app or iPhone app) doesn’t change.

$ dig couch.example.com A

;; QUESTION SECTION:
;couch.example.com.             IN      A

;; ANSWER SECTION:
couch.example.com.      811     IN      CNAME   couch.b-couch.com.
couch.b-couch.com.      811     IN      A       1.2.3.4

Note that the couch clients require no change at all.

$ curl http://couch.example.com/
{"couchdb":"Welcome","version":"1.1.0"}

Cloud, everybody wants it, some actually use it. So what's my take away from AWS' recent outtage?

Background

So first off, we had two pieces of our infrastructure failing (three if we include our Multi-AV RDS) — both of which involve EBS.

Numero uno

One of those pieces in my immediate reach was a MySQL server, which we use to keep sessions. And to say the least about AWS and in their defense, the instance had run for almost 550 days and had never given us much or any reason to let us down.

In almost two years with AWS I did not magically lose a single instance. I had to reboot two or three once because the host had issues and Amazon sent us an email and asked us to make sure the instance survive a reboot, but that's about it.

Recovering the service, or at least launching a replacement in a different region would have been possible if not by coincidence we would have hit several limits on our AWS account (instances, IPs and EBS volumes), which apparently take multiple days to lift. We contacted AWS immediately and the autoresponder told us to email back if it was urgent, but I guess they had their hands full and apparently we are not high up the chain enough to express how urgent it really was.

I also tried to reach out to some of the AWS evangelists on Twitter which didn't work since they went silent almost all the way through this outtage.

All in all, it took roughly five hours to get the volume back and another 4-5 to recover the database. As far as I can tell, nothing was lost.

And in our defense — we were well aware of this SPOF and had already plans to move on a more redundant approach — I have another blog post in draft about evaluating alternatives (Membase).

Numero due

The second critical piece of infrastructure which failed for us is our hosted BigCouch cluster with Cloudant.

We managed to (manually) failover to their cluster in us-west1 later in the day and brought the service back up. We would have done this earlier, but AWS suggested it would be only a few hours which is why we wanted to avoid the hassle of having to sync up clusters later on.

Sidenote: Cloudant is still (day three of the downtime) trying to get all pieces back online. Kudos to everyone from Cloudant for their hard work and patience with us.

Lesson learned for myself: When things fail which are not within your reach, it's pretty hard to do anything and stay calm. A good thing is to keep everyone busy so our team tried to reach out to all customers (we average about 200,000 users per day) via Twitter and Facebook and it looks like we've tackled that well.

Un trio!

Well, I don't have much to say about Amazon RDS (hosted MySQL in the cloud). Except that it didn't live up to our expectations: it costs a lot of money, but we learned that apparently that doesn't buy us anything.

Looking at the CloudWatch statistics associated with our RDS setup (or EBS in general), I'm rather weary and don't even know if they can be trusted. In the end, I can't really say for how long RDS was down or failed to failover, but it must have been back before I got a handle on our own MySQL server.

The rest?

The rest of our infrastructure seems fine on AWS — most of the servers are stateless (Shared nothing, anyone?) and no EBS is involved.

And with the absense of EBS, there are no issues to begin with. Everything continued to work just as expected.

Design for failure.

This is not really a take away, but a no-brainer. It's also not limited to AWS or cloud computing in general.

You should design for failure when you build any type of applications.

In terms of cloud computing it's really not as bad as ZOMG MY INSTANCE IS GONE!!!11, but it certainly can happen. I've heard people claim that EC2 instances constantly disappear and with my background of almost two years on AWS, I know that's just not true.

Designing for failure should take the following into account:

Services become unavailable

These services don't have to run in the cloud, they can become unavailable on bare metal too. For example, a service could crash, there could be a network partition or maintenance involved. The bottom line: How does your application deal with it?

Services run degraded

For example, higher latency between the services, slower response time from disk — you name it.

The unexpected

Sometimes, everything is green, but your application still chokes.

While testing for the unexpected is of course impossible, validating what comes in is not.

Recovery

I'm not sure if a fire drill is necessary, but it helps to have a plan on how to troubleshoot issues to be able to recover from an outtage.

In our case, we log almost anything and everything to syslog and utilize loggly to centralize logs. loggly's nifty console provides us with great input about the state of our application at any time.

Add to the centralized logging, that we have a lot of monitoring using ganglia and munin in place. Monitoring is an ongoing project for us since it seems like once you start, you just can't stop. ;-)

And last but not least: We can launch a new configured EC2 instance with a couple mouse clicks using Scalarium.

I value all these things equally — without them, troubleshooting and recovery would be impossible or at least a royal PITA.

Don't buy hype

So to get to the bottom of this (still ongoing) event, I'm not particulary pissed that there was downtime. Of course I can live without it, but what I mean is: Things are bound to fail.

The truth is though, that Amazon's product description is not exactly honest, or at the very least provides everyone with a lot of room for interpretation. You're asking for how much interpretation? I'm sure you could put ten cloud experts into one room and come away with 15 different opinions.

For example the use of attributes to a service such as highly available may cause different expectations for different people.

Let me break it down for you: highly available in AWS speak, means, "it works most of the time".

What about an SLA?

Highly available is not an SLA with those infamous nine Erlang nines.

On paper a multi-az deployment of Amazon RDS gets pretty close to what generally people expect from highly available: MySQL master-master replication, backups — in multiple datacenters. As of today we all know: even these things can fail.

And speaking of SLAs: It looks like none of the services failing are covered by it: AWS' track record remains clean. This is because EBS is not explicitely named in it and by the way neither is RDS. Amazon's SLA — as far as EC2 is concerned — covers some but not all network outtages. Since I was able to access the instance the entire time none of this applies here.

Multi-Zone

On Twitter people are quick to suggest that everyone who complaines now should have had a setup in multiple availability zones setup.

Here's what I think about it:

• I find it rather amusing that apparently everyone on bare metal runs in multiple datacenters.

• When people suggest multi-zone (or multi-az in Amazon-speak), I think they really mean multi-region. Because a zone is effectively us-east-1a, us-east-1b, us-east-1c and us-east-1d. Since all datacenters (= availability zones) in the us-east1 region failed on 2011/04/21, your multi-zone setup would not have covered your butt. Even Amazon's multi-az RDS failed.

• Little do people know, but the zone identifiers — e.g. us-east-1a, us-east-1b — are tied to customer accounts. So for example, Cloudant's version of us-east-1a may be my us-east-1c, it may or may not be the same. This is why in many cases AWS never calls out explicit zones in outtages. This also makes it somewhat hard to plan ahead in a single region.

• AWS sells customers on the idea that an actual multi-az setup is plenty. I don't know too many companies who do multi-region (maybe SimpleGeo?). Not even NetFlix does multi-region, but I guess they managed to sail around this disaster because they don't use EBS.

• In the end it shouldn't be necessary to do a multi-region setup (and deal with its caveats) since according to AWS the different zones inside a region are different physical locations (let's call them datacenters) to begin with. Correct me if I'm wrong, but the description says different physical location, this is not just another rack in the same building or another port on the core switch.

Communication

Which brings me to the most important point of my blong entry.

In a nutshell, when you build for the AWS platform, you're building for a blackbox. There are a couple papers and blog posts where people try to reverse engineer the platform and write about its behavior. The problem with these things is that most people are guessing, though often (of course depending on the person writing) it seems to be a very well educated guess.

Roman Stanek blogged about communication between AWS and its customers, so head on over, I pretty much agree with everything he has to say:

• http://roman.stanek.org/2011/04/21/mr-jassy-tear-down-this-wall/

Fin

So what exactly is my take away? In terms of technical details and as far as redundancy is concerned: not so much.

Whatever you do to run redundant on AWS, applies to setups in your local colocation or POP as well. And in theory, AWS makes it easier though to leverage multiple datacenters (availability zones) and even achieve somewhat of a global footprint by distributing in different regions.

The real question for anyone to ask is, Is AWS fit to host anything which requires permanent storage? (I'm inclined to say no.)

That's all.

Two weeks ago I had the chance to give a talk about GeoCouch and MapQuery at the FOSSGIS 2011. Most of the people who read this Blog are probably aware of GeoCouch, but not so much of MapQuery. For me these two projects are tightly connected and therefore deserve a quick introduction/update.

GeoCouch

GeoCouch, a spatial index for CouchDB gains, more and more attention. One of the reason is that the installation recently got way easier for developers as well as for normal users. You now can install GeoCouch as an extension right next to your already existing CouchDB instance. You may also download a binary of Couchbase-Server, which already includes GeoCouch. And finally there's the brand new Iris Couch hosting as well (previously known as the CouchOne hosting). So getting started with GeoCouch is easier than ever before.

Some people might have wondered about the state/future of GeoCouch, especially after the merger of CouchOne with Membase to Couchbase. I will keep on developing GeoCouch at Couchbase and it is (as it always was) fully open source licensed under the Apache 2.0 License.

The new home for the latest source is the Couchbase Github repository.

OpenStreetMap

The FOSSGIS was also about OpenStreetMap. The idea to put OpenStreetMap data into GeoCouch is very sensible, but wasn't really done (AFAIK) in a big fashion. Luckily Jochen Topf from Geofabrik told me about his Projekt Osmium, which makes it possible to process OSM data with JavaScript. There is already a script to output a Shapefile, so it should be really easy to output GeoJSON, which could be consumed by GeoCouch. So if you (who are currently reading this) have some spare time, please give it a go :)

MapQuery

MapQuery is a web mapping framework that builds on OpenLayers and jQuery. The goal is a framework that is just as easy to use as jQuery combined with the power of OpenLayers. It's meant for people that just want to get started with web mapping, but also for those who have already knowledge about OpenLayers and want to have easy integration into their jQuery application.

I was able to show a quick demo of the MapQuery API at the FOSSGIS. I won't publish it here, as things are about to move fast. After over one year of discussions about MapQuery and only little code contributions, it seems that we are finally getting somewhere. That feels so good :)

The wonderful EduGIS is build on an early version of MapQuery (source code), but will be merged with the most recent version of my fork.

Other big news is that the WhereGroup hired Christian Wygoda, who is a committer of the MapQuery project. This also means that Mapbender 3 will use MapQuery.

And finally I've also met a developer of a another company that was building a big mapping application based on OpenLayers and jQuery. I don't want disclose it here, as the code isn't open source yet, but the developer told me that it should be easily possible. I will keep in touch with them and hope they will contribute their code to MapQuery.

To get to a conclusion about MapQuery. If you want to stay in touch with the project, please subscribe to the official mailing list, this is where things are happening (there's also the little attended IRC channel #mapquery on freenode). If you want to be a user of MapQuery, you should be patient and wait a bit. If you plan to contribute, you can start now. The currently biggest item is moving the EduGIS MapQuery code base over to the MapQuery version of my fork. The "documententation" are the demos.

FOSSGIS

As people started to asked about the slides from my presentaion at FOSSGIS, here they are.

FOSSGIS was a really awesome event, where I met a lot of new people, but also a lot of friends I haven't seen in a while. I'm really looking forward to next year's conference, but also hope that I might see many of the people at this year's FOSS4g in Denver.