Planet CouchDB

January 27, 2015

The Offical CouchDB Blog

CouchDB Weekly News, January 27, 2015

Major Discussions

Offer to donate Nano and futoncli to CouchDB (see thread)

CouchDB got the offer to receive Nano and futoncli as donations. Many questions and details have been discussed in the thread. Legal questions around it have been raised separately towards the ASF Incubator this week as well.

Releases in the CouchDB Universe

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • CouchDB has a new wiki. Help us move content from the old to the new one!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Job Opportunities for people with CouchDB skills

Time to relax!

  • “To avoid being stuck in a local maxima, we need to periodically look across boundaries. This means both looking below familiar abstractions, and above to see what others are building upon your work.” – Coevolution of Software
  • “How about let’s have 2015 be the year that we stop looking for ‘rockstars’ and stop tolerating ‘rockstar’ behavior like people who constantly interrupt and speak over others because their ‘rockstar’ opinion is just that … important. If we all mindfully work to interrupt this interrupt culture, all voices can be heard, and everyone wins.” – On interrupting interrupt culture
  • Philosophers and scientists have been at war for decades over the question of what makes human beings more than complex robots. – Why can’t the world’s greatest minds solve the mystery of consciousness?
  • “Fuelling the mind isn’t easy, it requires rigour and discipline. Put in that discipline, however, and the returns will be considerable. Your ideas will flow and, the more you apply yourself to priming the brain, the more you’ll see connections in the content you encounter.” – Designing a Mind

Filed under: News

by Lena Reinhard at January 27, 2015 07:12 PM

January 20, 2015

The Offical CouchDB Blog

CouchDB Weekly News, January 20, 2015

Major Discussions

Limitation on number of databases? (see thread)

Question: A user was wondering if CouchDB was imposing any limit on the number of databases which a single instance can serve (and replicate).

Answer: There are two limits:

  1. Operation system limit on file descriptions user allowed to manage (see uliimit).
  2. CouchDB inner limitation. See the CouchDB Docs for details.

Configuring these options right allows you to serve thousands active
databases without any problem.

CouchDB to csv file (see thread)

Question: a user is working on a script in Java to convert a whole CouchDB into a csv file and wants to know about the easiest way to do that.

Approaches:

  • using `/{db}/_all_docs?include_docs=true` (see docs)
  • Couch has an HTTP API, so you have to pull the right URL in Java (i.e. open a java.net.Connection to the right URL). You can try the URL in the browser before to check whether all the right data is there. So, you can point your browser to http://{server}:{port}/{db}/_all_docs?include_docs=true and check if you have all info in it. Then you can use something like (new java.net.URL(“http://{server}:{port}/{db}/_all_docs?include_docs=true”).openConnection()) to grab the URL connection and work from there. Using something like this is certainly useful
    when moving forward.
  • using a list and fetching it as a csv file.

Releases in the CouchDB Universe

Use Cases, Questions and Answers

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Job Opportunities for people with CouchDB skills

Time to relax!

  • “Turns out great minds don’t think alike. Discover how some of the world’s most original artists, writers and musicians structured their day … .” – The Daily Routines of Famous Creative People
  • “As developers, designers, and content creators we cannot control what users are carrying into their experiences with our work. They’re carrying past and present pain. We cannot pretend our users are perfect automatons, single data points on a web log devoid of emotion. We have to be kind. After all, we have our own past and present pain we’re carrying into the interaction.” – The Year of Empathy
  • “Ah, meritocracy! The great solution to all of society’s problems, at least if the white, heterosexual, cis-male hacker is to be believed. If just we’d had meritocracy, then we could actually solve the important problems, and everyone would be happy!” – There’s no merit in meritocracy

… and also in the news


Filed under: News

by Lena Reinhard at January 20, 2015 04:13 PM

January 13, 2015

The Offical CouchDB Blog

CouchDB Weekly News, January 13, 2015

Major Discussions

CouchDB Day 2015 is coming very soon (see thread)

CouchDB Day 2015 will take place in Hamburg, Germany, on February 7, 2015. You can still register a free ticket here. The schedule is already online here. The Hoodie people are giving away four travel grants for visiting CouchDB Day, you’ll find all details in this post.

For you people in Berlin: there’s also an opportunity to get a ride to Hamburg with someone visiting CouchDB day.

React.js and Fauxton (see thread)

The Fauxton team wants to move from using backbone.js for Fauxton development to using React.js and a Flux architecture. There’s already a Pull Request for the initial work and they’re looking for feedback. Generally, any feedback you will have to improve this branch is very welcome, also feedback on whether they’re adhering to the Flux pattern correctly and more questions which you can find in detail here. It would be great to have your support in this!

Open PRs which need some review and merge (see thread)

There’s a number of open PRs which need some review and help – you’ll find the list here.

Proposal: CouchDB tests organisation (see thread)

Proposal for slight change in CouchDB’s test layout, the full text is here.

Releases in the CouchDB Universe

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet:

  • Stack Overflow: How to GET all doc from CouchDB database and show then in a HTML <div>?

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Job Opportunities for people with CouchDB skills

Time to relax!

  • “Empathy is the most important skill you can practice. It will lead to greater success personally and professionally and will allow you to become happier the more you practice.”Your most important skill: Empathy
  • “One of the most popular, enduring, and irritating myths about depression is that it means depressed people are sad all the time – and that by extension, people who are happy can’t be experiencing depression, even if they say they are.” – Depression doesn’t make you sad all the time
  • “I love Open Source. It has taught me many things going beyond programming — better collaboration, patience, compassion and mentoring. But it has a very dark side too, that often gets swept under the carpet of shiny benefits of a new career and so-called fame.”The Dark Side of Open Source

… and also in the news


Filed under: News

by Lena Reinhard at January 13, 2015 07:45 PM

December 19, 2014

The Offical CouchDB Blog

CouchDB Weekly News, December 19, 2014 and CouchDB Highlights in 2014

CouchDB IRC Meeting (see summary)

  • CouchDB Day 2015, Hamburg, Germany – if you attend and would like to talk about CouchDB 2.0 or something please get in touch with awenkhh (andywenk@apache.org)
  • CouchDB 2.0 – an email will be sent to dev@ mailing list with a first plan about dependencies reorganisation
  • Cloudant Query (Mango) Contribution
  • Fauxton – good progress, still a few blockers for 2.0
  • CouchDB Branding (logo, slogan) – ongoing discussion on marketing@ mailing list

Major Discussions

Stream CouchDB changes in PostgreSQL (see thread)

A user wrote about their setup and use case and this Node libary to stream CouchDB changes into PostgreSQL, there’s also a round of feedback and more ideas from the community in the thread.

Soft launch: CouchDB Advocate Hub now live (see thread)

In order to broaden the number of people contributing to CouchDB and give people more opportunities to support the project and our marketing efforts, we’re soft launching CouchDB’s Advocate Hub this week. There’s a basic set of challenges set up, some basic badges, and some basic rewards, which can already be tried, and which you can give feedback to (either in the community forum, or in the email thread). There’s more planned for next year, but it would be great if you could already take a first look and give us some feedback.

Deprecation of erlang@ mailing list (see thread)

As for inactivity, the erlang@ mailing list has been deprecated this week. Goodbye, Erlang mailing list, and thanks for the great times we had with you!

Releases in the CouchDB Universe

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Time to relax!

  • “One of the most consolidated misconceptions about programming, since the early days, is the idea that such activity is purely technical, completely exact in nature, like Math and Physics. Computation is exact, but programming is not. The first is a result of a machine operation, and the latter is still a human activity.” – The Art of Programming
  • “If you’re feeling anxious, breathe from your diaphragm in order to keep your sympathetic nervous system from revving up too much. And remember that it can be good to be keyed up: the right amount of nervousness will enhance your performance.” – The relationship between anxiety and performance
  • “It had long been tradition that operating calculating machines was “women’s work”; it was thought to be just keypunching, like typing. Women programmed and operated the punchcard machines to produce calculations for the Manhattan Project.” – Margaret Hamilton, Lead Software Engineer, Project Apollo
  • “I decided to start celebrating in a public way so that more people may be inspired to find their own ways of marking their career achievements. These are those donuts.” – The importance of donuts

CouchDB Highlights in 2014

The year is almost over, and it’s a great time for a quick look back – back at an exciting year full of work and great news from CouchDB, its community and CouchDB’s universe. And here’s to only a few of them (in random order):

  • CouchDB 2.0 Developer Preview is released.
  • CouchDB 1.6.1 released.
  • The CouchDB community established its Code of Conduct and Diverstiy Statement, which were later adopted by the Apache Software Foundation as well
  • In 2014, 9 new Committers and 2 new Project Management Committee members joined the community in addition to everyone who’s already part of the project. CouchDB now has 42 committers and 12 PMC members. We’re proud and honoured to have you all on board.
  • IBM acquired Cloudant.
  • Cloudant Local released (the first CouchDB-on-your-premises-with-enterprise-support product).
  • In March, the first issue of the CouchDB Weekly News has been published, and this post is the 38th issue of these Weekly News.
  • CouchDB got a new blog (it’s this one that you’re looking at).
  • Cloudant Query being donated to CouchDB.
  • npm’s continued success, building on CouchDB and the founding of npm, inc.
  • Postgres & MySQL continue to work on native JSON support + HTTP interfaces.
  • Several major PouchDB releases.
  • Release of Fauxton.
  • … as well as many, many more. Thanks to everyone who helped making all of this possible!

Editors’ Note

Dear reader, we want to thank you for reading this, our last issue of the CouchDB Weekly News in 2014. Both CouchDB and its Weekly News are from the community and for the community, and we’re glad that you’re with us. We wish you great last days this year and hope your new year will be off to a good start.

Stay with us, and: relax.

… and also in the news


Filed under: News

by Lena Reinhard at December 19, 2014 11:02 AM

December 11, 2014

The Offical CouchDB Blog

CouchDB Weekly News, December 11, 2014

Releases in the CouchDB Universe

  • nano-repository 1.0.0 – makes using CouchDB via Nano even easier
  • broccoli-couchdb-versioning 0.1.1 – define couchdb design documents on the file system, and automatically push them to couchdb when they change
  • boxspring 0.0.20 – a collection of Backbone Model classes for interacting with CouchDB compatible with Browser and Server-side execution
  • express-pouchdb 0.8.1 – express submodule with a CouchDB style REST interface to PouchDB
  • cradle 0.6.9 – the high-level, caching, CouchDB library
  • couchbeam 1.1.5 – a CouchDB client library for Erlang applications
  • npm have relaunched their website

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet:

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Time to relax!

  • “The idea that the software industry benefits from an unwritten law of unconditional and mutual respect is an extension of meritocratic thinking: it’s as unrealistic as the meritocracy itself.” – Codes of Conduct: when being excellent is not enough
  • “The world is not acting in our long-term benefit. Imagine you walk down the street and every store is trying to get your money right now; in your pocket you have a phone and every app wants to control your attention right now. Most of the entities in our lives really want us to make mistakes in their favor. So the world is making things very, very difficult.” – How to be efficient

… and also in the news


Filed under: News

by Lena Reinhard at December 11, 2014 02:43 PM

December 04, 2014

The Offical CouchDB Blog

CouchDB Weekly News, December 04, 2014

Major Discussions

Infinite resets hanging views (see thread)

Question: a user is trying to run a temporary view on a big-ish db (> 1000 docs), and the view seemed to hang forever, and they were not sure if there was something wrong with their view or settings. The view looked like this:

# map function
function(doc) {
emit(doc, doc.tweets);
}

Answers / approaches:

  1. Think of `map` as a way to create secondary indexes.
    1. An index on the whole document doesn’t make much sense.
    2. Moreover, at query time, if you want to get data that are in the same document, need map/reduce isn’t needed at all. Just create a `show` function.
  2. I would go further and say: Never emit the full doc in a map function.
    You can get the full doc by using the include_docs flag when querying the
    view. If you need a key that only refers to a single doc, use the _id of
    the doc.
  3. Don’t write your own reduce functions, use couch’s built in, Erlang functions (See http://wiki.apache.org/couchdb/Built-In_Reduce_Functions). You will save yourself a lot of headaches and you will not match the performance of these functions. In your case, _count is probably what you want, though you probably don’t even need a reduce function with such an index (as Aurelien essentially pointed out). Handle the case where doc.tweets doesn’t exist in your map function:
    emit(doc._id, doc.tweets || []);

Allow user-defined views (see thread)

Question: a user was thinking about a platform based on CouchDB, where each set of users would get their own CouchDB Database, to store and query data. They want to allow them to define their own custom queries to query the data and want to create a form which allows to build a query and translates it to a JS view, and, on top of that, defining custom views directly in JS – basically their custom map/reduce functions. The users would also be able to import different sets of data. Still, they were not sure about possible DoS attacks with endless loops inside the function, or attacks by emiting too much data.

Approaches:

  • os_process_timeout applies to view servers (default is 5 seconds) so you can configure the timeout for view queries, not sure about CPU/HDD Space (although inspecting _stats might shed some light).
  • If viable for the application, it may also be possible to initially have a process where new queries written are vetted by a human before they are run. The advantage of this is two-fold, namely:
    • i) you’ll be able to move on a prove your concept quickly
    • ii) while doing this, you may learn enough (and things may change enough) for you to automate the vetting process.
  • The worst case will be always RCE since you’re going allow everyone
    execute arbitrary code on your server. JavaScript query server is only safe while SpiderMonkey sandbox is. However, if you want to use some custom query servers when things will go bad since nor Python, Erlang, Clojure and other I know servers supports sandboxing which means you can do anything.
  • The idea may work for Mango – Cloudant views query DSL – since it’s very limited by allowed operations, but then you’ll face another problem: disk space will run out very quickly since a single index file may be much more bigger then database itself.
  • For now the most simple and secure way to allow custom users view is to let users replicate your database to their CouchDB instance where they can do anything whatever they need.
  • Another approach: use CouchDB for whatever you need on the server side, and when a user needs to run a custom query, run that query on a client-side copy of the data in PouchDB. This wouldn’t work with very large data sets, but would work on smaller data sets.
  • Alternatively, don’t allow the user to write the view. Build some Ui that helps creating the view function, specify some things that can be run on the data and construct the view with that. This is essentially a query builder for CouchDB.

Releases in the CouchDB Universe

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet:

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Time to relax!

… and also in the news


Filed under: News

by Lena Reinhard at December 04, 2014 05:42 PM

November 27, 2014

The Offical CouchDB Blog

CouchDB Weekly News, November 27, 2014

Release

  • CouchDB 2.0 Developer Preview has been released last week. You can download it here, its features and more are described here

Major Discussions

Offline Replication / Synchronisation (see thread)

Question: due to infrastructural reasons, online synchronisation is impossible in a project. Thus, a user asked if it were possible to create some sort of file with the delta-update that could be transported to a place with better Internet access or to the next hub (e. g. country coordination) where it is imported in the instance running there, or anything similar that could be feasible.

Approaches:

  • If connections are bad, but work sometimes, and the use case permits that data is only synchronised sometimes, then CouchDB’s current functionalities are sufficient.
  • When connections are just always bad, the user can copy the .couch file (“the CouchDB database”) to something that they can bring to a connected hub. Then they put the .couch file into a place that can run CouchDB, start CouchDB, sync from there. CouchDB will figure out the delta. In this, the full DB will carried, but only the delta will be synchronised. Just carrying the delta is currently not really an option. If that is a concern, hacking around with pouchdb-replication-stream, pouchdb-dump-cli or pouchdb-load may help.

Releases in the CouchDB Universe

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet:

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Time to relax!

… and also in the news


Filed under: News

by Lena Reinhard at November 27, 2014 11:02 AM

November 20, 2014

The Offical CouchDB Blog

CouchDB Weekly News, November 20, 2014

Release

  • CouchDB 2.0 Developer Preview has been released. You can download it here, its features and more are described here

Major Discussions

Guidelines for creating a CouchDB Event (Work in Progress; see thread)

New guidelines for creating a CouchDB Event have been set up, you’ll find them here. Your feedback is highly appreciated.

The Apache Software Foundation adopts CouchDB’s Code of Conduct and Diversity Statement (see thread)

On November 18, the ASF has adopted the CouchDB Code of Conduct and Diversity Statement into an official ASF-wide policy.

Releases in the CouchDB Universe

  • couchdb-python 1.0 – issue fixes, improvements and more; download from PyPI | source
  • Mango – a MongoDB inspired query interface; Open Sourced by Cloudant this week, with an intent to donate it to CouchDB; more details on it are here
  • sofa – easy R interface to CouchDB

Opinions and other News in the CouchDB Universe

Use Cases, Questions and Answers

No public answer yet:

For more new questions and answers about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:

  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Can you help with Web Design, Development or UX for our Admin Console? No Erlang skills required! – Get in touch with us.
  • Do you want to help moving the CouchDB docs translation forward? We’d love to have you in our L10n team! See our current status and languages we’d like to provide CouchDB docs in on this page. If you’d like to help, don’t hesitate to contact the L10n mailing list on l10n@couchdb.apache.org or ping Andy Wenk (awenkhh on IRC).

We’d be happy to have you on board!

Events

Time to relax!

  • “It seems the mere mention of tools that support and encourage marginalised people to be a part of the tech industry incites a frothing flame war from the privileged.” – Tech Freedom vs. Feminism
  • “Keep cake moist by just eating the entire thing in one sitting.” – … and 12 more amazing Food and Life Hacks you need to know right now
  • “As a tech community, we are increasingly recognizing that diversity is fundamental to building a just, inclusive and beneficial industry, and that diverse teams perform better, are more innovative, and can build better products for a diverse world. In order to be successful with diverse recruiting, tech companies must invest in analysis and improvement at every stage of the hiring process.” – 25 Tips for Diverse Hiring
  • Kitten Therapy: the Prescription for Stress
  • Sloth Life

… and also in the news


Filed under: News

by Lena Reinhard at November 20, 2014 06:41 PM

June 10, 2014

The Old Official CouchDB Blog

Apache CouchDB 1.6.0 Released

Apache CouchDB 1.6.0 has been released and is available for download.

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

Grab your copy here:

http://couchdb.apache.org/

Pre-built packages for Windows and OS X are available.

CouchDB 1.6.0 is a feature release, and was originally published on 2014-06-10.

These release notes are based on the changelog.

Upgrade Notes

The Proxy Authentication handler was renamed to proxy_authentication_handler to follow the *_authentication_handler form of all other handlers. The old proxy_authentification_handler name is marked as deprecated and will be removed in future releases. It’s strongly recommended to update the httpd/authentication_handlers option with the new value if you have this handler configured.

What's New

  • COUCHDB-2200: support Erlang/OTP 17.0 #35e16032
  • Fauxton: many improvements in our experimental new user interface, including switching the code editor from CodeMirror to Ace as well as better support for various browsers.
  • Add the max_count option (UUIDs Configuration) to allow rate-limiting the amount of UUIDs that can be requested from the /_uuids handler in a single request (CVE 2014-2668).
  • COUCHDB-1986: increase socket buffer size to improve replication speed for large documents and attachments, and fix tests on BSD-like systems. #9a0e561b
  • COUCHDB-1953: improve performance of multipart/related requests. #ce3e89dc
  • COUCHDB-2221: verify that authentication-related configuration settings are well-formed. #dbe769c6
  • COUCHDB-1922: fix CORS exposed headers. #4f619833
  • Rename proxy_authentification_handler to proxy_authentication_handler. #c66ac4a8
  • COUCHDB-1795: ensure the startup script clears the pid file on termination. #818ef4f9
  • COUCHDB-1962: replication can now be performed without having write access to the source database (#1d5fe2aa), the replication checkpoint interval is now configurable (#0693f98e).
  • COUCHDB-2025: add support for SOCKS5 proxies for replication. #fcd76c9
  • COUCHDB-1930: redirect to the correct page after submitting a new document with a different ID than the one suggested by Futon. #4906b591
  • COUCHDB-1923: add support for attachments and att_encoding_info options (formerly only available on the documents API) to the view API. #ca41964b
  • COUCHDB-1647: for failed replications originating from a document in the _replicator database, store the failure reason in the document. #08cac68b
  • A number of improvements for the documentation.

by djc at June 10, 2014 05:17 PM

June 06, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, June 06

Weekly CouchDB meeting – summary

  • 1.6.0 release status: the vote had passed last week, binaries for Mac and Windows are ready for testing. The release will be very soon, stay tuned!
  • BigCouch merge: significant process has been made; testing is highly important now. We want to encourage everyone to help with testing COUCHDB-1843 branch, especially :5984 API iface. Info: to run the bigcouch from merge branch: clone couchdb.git, checkout 1843-feature-bigcouch, ./configure && make && dev/run. Please report any errors you find during testing on dev@ Mailing list or on IRC #couchdb-dev.
  • Bylaws, Code of Conduct and Diversity Statement: all PMC members are asked for reviewing and commenting the Bylaws and the Code of Conduct and Diversity Statement. A thread about approval models and vetoes will be started on the mailing list.

Releases in the CouchDB Universe

Opinions

Use Cases, Questions and Answers

no public answer yet:
For more new questions about CouchDB, see these search results.

Get involved!

If you want to get into working on CouchDB:
  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at June 06, 2014 04:04 PM

May 30, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, May 30

Major Discussions

Vote on release of Apache CouchDB 1.6.0 rc.5 (will be released as Apache CouchDB 1.6.0 — see thread)

The vote passed.

Limiting view access (see thread)

Use case: sharing docs between users with different roles and according access. Goal is to make sure via adding according CouchDB settings that no user can see who the other users of the doc are.

Answer: Natively, CouchDB doesn’t provide this functionality. Thus, the canonical answer is to use a proxy in front of CouchDB that can limit access to certain URLs, or to deploy a middleware layer written in your favourite language that can broker the access.

CouchDB minimum revs_limit and Compaction (see thread)

Use case: limiting the number of revisions stored for a specific document in a database. Setup: setting the revs_limit to a specific number and then running the compaction. This did not work, the number of revisions could not be limited this way, and the enquirer asked why.

Answer: _revs_limit does not do what one may think it does, – CouchDB is not a revision control system. After completion only the latest revision of each document is preserved. _revs_limit refers only to how many old _rev values are retained. Lowering this value will save some storage but can also make replication less efficient as a failure to find a common ancestor will cause CouchDB to copy the document afresh.

Releases in the CouchDB Universe

Special topic: Content Management Systems based on CouchDB

  • Kleks – a pure CouchDB based CMS written as a CouchApp using Kanso, Spine.js, CoffeeScript and Stylus. Supports multi-site setup and Markdown authoring
  • Couchpress – a super lightweight and modular CMS built on NodeJS, Express and CouchDB
  • CMS – a Content Management System written with Ruby on Rails and CouchDB
  • Cmtool – a rails 3.2+ CMS as engine for a CouchDB backend
  • CCMS – a Backend-free, client-side JavaScript and CouchDB based CMS
  • Falkland CMS (Website) – a Curation Management System written in Clojure, ClojureScript and CouchDB

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • We have an infinite number of open contributor positions on CouchDB. Submit a pull request and join the project!
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at May 30, 2014 04:17 PM

May 22, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, May 22

Weekly CouchDB meeting – summary

  • 1.6.0 release: no new issues were found during voting on CouchDB 1.6.0 rc5
  • BigCouch merge: significant progress has been made. Everyone is still encouraged to continue helping with building and testing COUCHDB-1843 branch. Further items for discussions will be sent to the mailing list.
  • CouchDB Meetup Hamburg: the CouchDB user group in Hamburg has its own Google Plus Community and will have their next meetup will be on June 10, 2014.

Major Discussions

Running views return nothing with CouchDB 1.5.1 (ongoing discussion; see thread)
Setup: code that creates database, documents and design documents with views and lists. When views are run, nothing is returned under CouchDB 1.5.1. When the same code is run against another CouchDB server (CouchDB 1.5.0), all views return the right responses and results. Approach: the problem described here could be an error from a library not found while running an external process (CouchJS in this case). It turned out libmozjs.so was either not installed or not in a path searched by the dynamic linker. Once it is added, it should be ensured that CouchJS can be run, afterwards, CouchDB should be able to build the views without restarting CouchDB.
Discussion: Project by-laws (ongoing discussion; see thread)
The discussion about the by-laws is still ongoing, more updates have been made. They can be found here, your feedback is still very welcome.
Release Apache CouchDB 1.6.0 rc5 (see thread)
The testing and voting for all systems is still in progress.

Releases in the CouchDB Universe

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at May 22, 2014 04:06 PM

May 16, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, May 16

Blog Posts

Weekly CouchDB meeting – summary

  • BigCouch merge: significant process has been made. We want to encourage everyone to check out the COUCHDB-1843 branch and help with testing. Info: to run the bigcouch from merge branch: clone couchdb.git, checkout 1843-feature-bigcouch, ./configure && make && dev/run. Please report any errors you find during testing on dev@ Mailing list or on IRC #couchdb-dev.
  • 1.6.0 release: 1.6.0 rc5 is still open vor testing and voting. There are now Windows binaries against Erlang 17.0, – Windows users out there, feel free to test them. On the issues with Erlang 17.0, please also see this post on what it takes to use it in production. There'll be a follow-up item on this which will be sent to the mailing list this week.
  • CouchDB Meetups: there have been two CouchDB Meetups in Hamburg and Berlin, Germany, in the last week, and both were amazing. The recap for Hamburg can be found here.

Major Discussions

Discussion: Project by-laws (ongoing discussion; see thread)

The by-laws have been discussed and updated according to the community's comments. They can be found here, your feedback is very welcome.

Release Apache CouchDB 1.6.0 rc5 (see thread)

The testing and voting for all systems is still in progress.

Releases in the CouchDB Universe

  • couch-daemon – helps you write CouchDB workers efficiently that can be implemented and managed by CouchDB as os_daemons; with a Highland streaming interface
  • couchdb2s3 (0.3.0) – export CouchDB databases to line-oriented json files on s3
  • sails-couchdb-orm (0.9.0) – CouchDB adapter for Sails.js and Waterline ORM
  • dimensionist – CouchDB daemon to extract dimensions from image attachments

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • Help us testing Apache CouchDB 1.6.0 rc5! You'll find all important information on release artefacts and test procedure here.
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
We'd be happy to have you!

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by dch at May 16, 2014 12:30 PM

May 15, 2014

The Old Official CouchDB Blog

Recap CouchDB Meetup Hamburg No. 1

This is the recap from our first CouchDB User Group Hamburg Meeting at May 13th 2014.

First of all thanks to everyone for attending. Also a big thank you to Ubilabs for hosting the meetup at their awesome office in Hamburg "Schanzenviertel".

Six people have been attending at this first meeting. Although it was a quite small meeting, we have discussed many things. At the beginning, everybody introduced themselves and dropped some info about what they are doing - especially with CouchDB. Andy spoke about the CouchDB history, the Apache Software Foundation and the ASF CouchDB project. After that, Klaus was giving some insights about coding in CouchDB's core and explained various internals. And Robert showed the new awesome Fauxton webinterface for CouchDB.

Julius attended accidentally, because he originally was thinking about founding a "Hamburg Beamers" - an Erlang user group. He got in touch with Dave Cottlehuber, who is a member from the "Vienna Beamers", pointing Julius to our first meetup (thanks Dave :) ). When digging a bit deeper into CouchDB, Julius was excited about what CouchDB offers.

We were also thinking about putting the Beamers and Couchers together in one user group because we thought that one of the main topics will be Erlang. This is not decided though. We did also look into Elixir and Klaus and Julius gave some basic insights. Elixir looks really cool and promising.

After nearly three hours, we finished our beers (thanks to Ubilabs for sponsering the drinks). Everybody was happy with the first meeting and we are looking forward to the next one. We plan to have the meeting every month. So the next meeting will be in mid June. Klaus will give the first talk where he is introducing a web service he is creating for measuring software complexity with Elixir and CouchDB.

We have now created a G+ Community called CouchDB Meetup Hamburg. We will use this community to announce upcoming events. This is the easiest way at the moment.

Finally, we decided to meet again at June, 10th and again at Ubilabs. Thanks a lot in advance.

Looking forward to see also new interested folks. So please spread the word.

by andywenk at May 15, 2014 07:26 AM

May 14, 2014

The Old Official CouchDB Blog

Board Report (May 2014)

Apache CouchDB is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API.

Releases

1.5.1 (2014-04-08) http://www.apache.org/dist/couchdb/notes/1.5.1/apache-couchdb-1.5.1.html

Recent Activity

  • Started with weekly news with great success. Also available at https://blogs.apache.org/couchdb/
  • The community is in the process of creating a CoC and bylaws
  • Ongoing work on the release 1.6.0. Actually voting on Apache CouchDB 1.6.0-rc.4
  • Working on the BigCouch merge from Cloudant
  • Good progress in reviewing the rcouch merge
  • Community work on migrating content to the new wiki started
  • Translation work going well

Community

Including the following additions, CouchDB has 35 committers and 12 PMC members.

New committers:

  • Robert Kowalski

New PMC members:

  • Joan Touzet

Mailing list stats:

announce

  • 175 subscribers (+24)
  • 1 message since February (1)

user

  • 1386 subscribers (-19)
  • 750 messages since February (-336)

erlang

  • 167 subscribers (+13)
  • 1 messages since February (-13)

dev

  • 599 subscribers (-3)
  • 2630 messages since February (+1306)

commits

  • 104 subscribers (0)
  • 2655 messages since February (-80)

marketing

  • 31 subscribers
  • 312 messages since February

l10n

  • 36 subscribers (+5)
  • 16 messages since May (-191)

replication

  • 54 subscribers (+7)
  • 53 messages since February (+28)

Issues: None.

by Jan at May 14, 2014 11:32 AM

May 08, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, May 08

Weekly CouchDB meeting – summary

  • 1.6.0 release: 1.6.0-rc.5 is out for testing, so for all systems go. 1.6.x CI is all green, a formal vote will be coming this week
  • rcouch merge status: testing up to now showed it's functional, it will be suggested for integration branch merge
  • BigCouch merge status: a massive progress has been made this week. Soon branches will be rewritten to clean up the history, afterwards the community will be asked for help with testing.
  • Fauxton: agreement that Fauxton replaces Futon in CouchDB 2.0 (post merge) and that resolution of COUCHDB-1954 prior to that is critical. There'll be a thread on dev@ mailing list about this and release preparation.
  • Bylaws, Code of Conduct, Diversity Statement: everyone in the community is still invited to read the by-laws draft and join the discussion on dev@ mailing list (see thread); an initial Code of Conduct and Diversity draft will be posted next week
  • Project release artefacts: the current state of what the project release artefacts are will be documented and publicized

Major Discussions

Discussion: Project by-laws (ongoing discussion; see thread)

The first draft of the by-laws has been modified and is still being discussed. It can be found here, your comments are very welcome.

How to get results in descending order by date and without ID field (see thread)

The docs for result orders can be found here. To remove the ID field, users can use a list function.

Release Apache CouchDB 1.6.0 rc5 (ongoing testing and discussion; see thread)

The testing for all systems is still in progress. If you want to test, please follow this test procedure. The changes since last vote round can be found here.

Releases in the CouchDB Universe

  • PouchDB 2.2.0 – including persistent mapreduce, eventemitter api, replication works for huge datasets
  • highland-couchr 1.0.0 – streaming style API to CouchDB, using couchr under the hood
  • massage-couch 0.1.0 – massage CouchDB documents with an OS daemon worker.

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • Help us testing Apache CouchDB 1.6.0 rc5! You'll find all important information on release artefacts and test procedure here.
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

  • DevOps Engineer, New York, USA
  • Work for any of JS, UI/UX, Node, CouchDB experience (see tweet), can work remote

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at May 08, 2014 06:52 PM

May 02, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, May 1

Weekly CouchDB meeting – summary

  • 1.6.0 release: we are making progress, next is checking if all bugs have been fixed
  • BigCouch merge status: Cloudant will now focus on the BigCouch merge, visible progress should happen in the next couple of weeks
  • rcouch merge status: the testing is still in progress, feedback will be provided to the mailing list; merge to the new integration branch can be done after that feedback
  • Code of Conduct: the first version will now be drafted based on the discussion on the mailing list (for details, see "Major Discussions" section below)
  • By-laws: everyone in the community is invited to read the draft and join the discussion (see thread); please note that the bylaws now include a new idea about PMC chair rotation
  • Translation update: German translation is making good progress (see status here)

Major Discussions

Apache CouchDB Diversity Statement (see thread)

This discussion was initiated with the suggestion to establish a project-wide Diversity Statement. First, the idea was brought up to turn this document into a PMC charter, a document that sets out what the PMC values, their commitment to diversity, and their pledge to the project. The second idea was about how merit is being recognised and it was brought up to broaden this around the four areas community, project, documentation and code. "This is what we value. This is what we will recognise. Our promise to the community." (Noah Slater) It was agreed on creating two documents, one for the PMC charter, a second one for diversity.

Discussion: Project by-laws (ongoing discussion; see thread)

Proposal to vote in set of project by-laws that define the specific roles in this community and the decision making procedures that are used. The first draft of the by-laws has been discussed and modified accordingly and can be found here. Comments are very welcome.

Apache CouchDB Developer Code of Conduct (ongoing discussion; see thread)

The PMC initiated a thread on moving forwards establishing an Apache CouchDB Developer Code of Conduct (CoC for short). Example Codes of Conduct from Debian, Python, Django , Bantik, Node.JS and the Node.JS IRC channel policy were as well considered and discussed as transparency, the consequences of infractions and the differentiation between minor offences and major offences, how to deal with profanity and how this applies to IRC and the moderation policy there.

HIPAA (Health Insurance Portability and Accountability Act) / HIPAA 5010 / PIPEDA compliance (see thread)

Question on building applications using CouchDB while having to follow HIPAA compliance. As for the legal topic this is, the community strongly recommended the user get in touch with a lawyer and getting proper compliancy and regulatory guidance.

CouchDB load spike (see thread, see also the same question on StackOverflow)

Setup: CouchDB 1.5.0, database with < 10 GB of data in it and continuous replication. Every few hours (3-4 times per day) they recognise a huge spike that floors the load to around 1.5 and memory usage to close to 100%. It turned out there was a similar issue that was already discussed (link to gist). The user filed issue COUCHDB-2231.

Tips for general CouchDB monitoring and graphs drawing tools to check what's causing huge spikes (from Alexander Shorin, see this reply):

  • Munin plugin for CouchDB monitoring – It doesn’t handle system metrics for the CouchDB process yet, but that will be added soon; users should make sure they have a similar plugin for their monitoring system.
  • Skyline, a detector for spikes and other anomalies
  • Oculus, a metrics correlation tool that makes it very easy to compare multiple graphs for an anomaly period with it.
Release Apache CouchDB 1.6.0 rc3 (ongoing testing and discussion; see thread)

Some issues have already been fixed, the testing is still ongoing. Find all release artefacts we are voting on in this list. If you want to test, please follow this test procedure. The changes since last vote round can be found here.

Releases in the CouchDB Universe

  • yama – mail app, work in progress; syncs CouchDB instance with an email inbox for RESTful email
  • tornado-couchdb 0.2.3 – Blocking and non-blocking (asynchronous) clients for CouchDB using Tornado's httpclient
  • loveseat 0.0.11 – very limited CouchDB interface
  • follow-db-updates 0.0.2 – easy CouchDB _db_changes follower, with older couch pollyfill
  • seneca-couchdb-changes 0.1.0 – emit seneca.js events for all events in a couchdb _changes feed
  • changes-stream 1.0.3 – simple module that handles getting changes from couchdb
  • neuropil 3.0.2 – a lighter and faster registry client for CouchDB based npm server
  • nano 5.8.0 – minimalistic CouchDB driver for node.js
  • couch_tap 0.0.2 – providing a DSL that allows complex CouchDB documents to be converted into rows in a RDBMS' table

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • rcouch Merge: Erlang hackers and CouchDB users, we need your help with testing and review of the rcouch merge. It's easy! Find the how-to in this post.
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
  • You want join us for the updates of CouchDB-Python for Python 3? Take a look at issue 231.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at May 02, 2014 04:06 PM

April 24, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, April 24

Weekly CouchDB meeting – summary

  • 1.6.0 release: voting for 1.6 rc3 is open; Erlang R15 and R16 are fine, but several unexpected issues for testing with Erlang R14BX were reported; any help with testing and diagnosing the issue is welcome
  • rcouch merge status: the recent call for testing led to progress on COUCHDB-1994.The Windows build and some other little bugs could be fixed. Still, help with testing is very welcome!
  • BigCouch merge status: large progress was made last week, next steps will be code review and testing
  • BigCouch branch and the new multirepo design: how to edit individual repos and split big CouchDB.git into single and finally run the merge

Major Discussions

Error when installing CouchDB on Windows 7 (see thread)

Installation of CouchDB 1.5.1_R16B02 binary on Windows 7 lead to an issue caused by a problem with the Windows R16B02 installation file. This file has been replaced quickly, everything works now. The installation file can be downloaded here.

Compaction of a database with the same number of documents is getting slower over time (see thread)

A database containing a relatively stable number of documents (in CouchDB: "doc_count"), the documents themselves change frequently (including many insertions and deletions (CouchDB: "doc_del_count")). The person asking expected the compaction time to stay the same, but numbers showed that it took longer to compact the database over time, thus they asked why this was the case, and how it could overcome.

The reply is that compaction time in CouchDB scales with the sum of "doc_count" and "doc_del_count". As explained by Adam Kocoloski, this can currently be controlled by "1) [Purging] the deleted docs (lots of caveats about replication, potential for view index resets, etc.); 2) [Rotating] over to a new database, either by querying both or by replicating non-deleted docs to the new one. Neither one is particularly palatable. CouchDB currently keeps the tombstones around forever so that replication can always work. Making changes on that front is a pretty subtle thing but maybe not completely impossible. Also, there's a new compactor in the works that is faster and generates smaller files."

Data replication: replicating only non-deleted documents (same discussion as above; see thread)

The approach could be to either run a filtered replication or block deleted documents with a validate_doc_update function on the target database (see example function here).

How to handle terabyte databases (see thread)

This was a request about improving performance, insert time, compaction and replication for terabyte databases with billions of documents (setup: CouchDB 1.2 and CouchDB 1.4). Some approaches to handle a write-heavy workload:

  • Replication: Supply parameters to allocate more resources to a given job (code example here).
  • Insert time: Ensure that e.g. compaction only runs in the background and does not impact the throughput of interactive operations; this is work in process at the moment.
  • Compaction: A new, significantly faster compactor that also generates smaller post-compaction files that also eliminates the exponential falloff in throughput observed by the inquirer is in the works. – This problem can be solved by BigCouch as it tries to partition databases.
  • Ensure to not exceed the write capacity of the RAID (this effect is amplified for partitioned databases).
Vote: Release Apache CouchDB 1.6.0 rc3 (ongoing testing and discussion; see thread)

We encourage the whole community to download and test these release artefacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release, so get stuck in on our dev@couchdb.apache.org mailing list! Find all release artefacts we are voting on in this list. If you want to test, please follow this test procedure. The changes since last vote round can be found here.

Releases in the CouchDB Universe

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • rcouch Merge: Erlang hackers and CouchDB users, e need your help with testing and review of the rcouch merge. It's easy! Find the how-to in this post.
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
  • You want join us for the updates of CouchDB-Python for Python 3? Take a look at issue 231.
We'd be happy to have you!

Events

  • April 26, Delhi, India: CouchDB meetup - Understanding how design documents work
  • June 16, 17, San Francisco, CA: CloudantCON

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at April 24, 2014 07:14 PM

April 17, 2014

Joan Touzet

Understanding race-induced conflicts in BigCouch

Distributed databases with a near-real-time multi-master configuration – such as BigCouch, coming soon to Apache CouchDB – must deal with the potential of simultaneous modifications of a single resource. While the approach taken by multiple single-machine Apache CouchDB servers using regular HTTP replication is well understood, the situation changes a little bit when dealing with BigCouch-style internal replication inside a cluster.

I think it’s time to have a better understanding of what this means, and what impact this has on you as an application developer. Most of the time, there’s no change – to your app, a BigCouch-style cluster looks and feels like a single Apache CouchDB node. But when making near-simultaneous writes to the same document from different clients, you may experience document conflicts that you wouldn’t have with an Apache CouchDB 1.x single server.

How does this happen? Bear with me – this gets a bit complex. Hopefully this diagram will help.

The sequence diagram below depicts a scenario where two very closely spaced writes to the same document in a 3-node BigCouch cluster will lead to a document conflict.

Sequence diagram showing Sequence diagram illustrating a document conflict introduced by near-simultaneous writes to a 3-node cluster.

Sequence diagram illustrating a document conflict introduced by near-simultaneous writes to a 3-node cluster.

In this example, the database cluster is a 3-node cluster, with default settings of n=3, r=2 and w=2. (This means that a successful write to a document must write 2 copies to disk before an HTTP 201 status code is returned.) Client 1 and Client 2, both external client processes talking to the cluster, are both trying to update /db/doc, which is currently at revision 3. Client 1 is trying to write rev 4-1, and Client 2 is trying to write rev 4-2.

For the purposes of this example, we are going to state a specific ordering for document writes, and treat the write as being processed serially in a specific order. In reality, writes are issued in parallel with no coordination of writes. The scenario as shown is simply one of the possible event sequences you may run into, in the wild.

Client 1’s write is being mediated by Node A. For this example, Node A issues writes to nodes A, B and C, in that order. Client 2’s write is being mediated by Node C. For this example, Node C issues writes to nodes C, B and A, in that order. Both Clients’ first writes succeed (Client 1/Node A and Client 2/Node C) and return the internal equivalent of an HTTP 201 status code.

Arbitrarily, we say Client 1’s request to Node B arrives prior to Client 2’s request to Node B. (One will always be processed before the other; it doesn’t matter which one gets there first.)  Node B makes the write of revision 4-1, and returns a 201 equivalent to Node A.

At this point, Node A has 2 successful write responses back. Since the cluster is configured with w=2, a 201 is returned to Client 1 and the client disconnects; its work finished. The third write, already issued by Node A to Node C, will eventually be handled, the conflict recorded to disk and a 409 sent back to Node A. Note that both copies of the document are kept, capturing the document conflict on Node C when this write occurs. Node A’s work is now done.

Just after completing its write of revision 4-1, Node B then processes Node C’s write attempt of rev 4-2 from Client 2. This results in the conflict being written to disk, and returns the 409 equivalent to Node C. The same happens when Node C’s write to Node A is processed. Node C now has a non-conflict 201 response from itself, and the 409 responses from Node B and Node A, so it sends the client a 202 status.

At the end of the process, all 3 nodes have both versions of the document recorded, fulfilling CouchDB’s promise of eventual consistency.

Still with me? Good.

So which document “wins”? By design, the document with the higher hash value (the second part of the _rev token, i.e. _rev=##-hash) will win. If 4-1 and 4-2 were the actual _rev values, 4-2 would win. As such, there is no guarantee that the write with a 201 response will be the ‘winner.’ [1]

The closer together writes to the same document occur, the more likely it is that the cluster may still be processing a previous write when the subsequent write comes in. Even with resolution and DELETEs of the losing conflicts, document “tombstones” will be left behind on these leaf nodes to ensure replication results in eventual consistency (CouchDB’s guarantee! [2])

The best approach is to avoid these kinds of document conflicts via an access pattern where simultaneous writes are as unlikely to occur as possible. There are a number of resources out there on how to design apps this way, but one of my favourites is to never re-write a document, store all operations as new documents, and use a view or _all_docs and the startkey/endkey parameters to retrieve the latest state of a given resource.

Barring an application redesign, your CouchDB client should look for 202s and conflicts in documents consistently, and provide evidence of this result to the application layer. [3] You can also create a conflicts view to review all conflicts in a given database.

Resist the temptation to try and resolve the conflict in the CouchDB library! Only at the application layer can you best decide how to deal with a document conflict. You might initially choose to ignore conflicts, but probably it’s in your best interest to perform some sort of manual resolution and write a new, merged version based on data in all the conflicted versions.

If you’re still lost, post in the comments or on the couchdb-user mailing list and I’ll do my best to explain.


[1] https://cloudant.com/for-developers/faq/data/, “My write returned a 201, and yet I have a conflicted document -  what gives?”

[2] Doing better is a provably difficult problem. For reference, start with  L. Lamport’s Time, Clocks and the Ordering of Events in a Distributed System, Communications of the ACM 21, 7 (July 1978), 558-565.

[3] Take note: some CouchDB libraries, such as python-couchdb, do not differentiate between a 201 and a 202 response!

by Wohali at April 17, 2014 05:01 PM

The Old Official CouchDB Blog

CouchDB Weekly News, April 17

Blog Posts

Weekly CouchDB meeting – summary

  • Bigcouch Merge: work on branches has started again and will make good progress
  • rcouch Merge: code reviews have been restarted; more help with testing is needed; discussion around Windows support, will be continued on Dev-Mailing list
  • Advocate Hub: onboarding at the moment, next steps will be sent to marketing mailing list soon
  • 1.6.0 release: release candidate 3 (rc3) is being prepared now
  • COUCHDB-1986: the issue is being investigated, will be updated soon if its status changes

Major Discussions

1.6.0 changelog review (see thread)

A discussion around what exactly should be included in 1.6.0 (see changelog and documentation for details).

Handling the Welcome Request (see thread)

How to handle the welcome message that appears from a root request and direct the "/" request to another URL (setup: version 1.5 on a Windows machine).

authentication_redirect is not working (see thread)

How to handle the redirection of users after login to other databases that service page requests (and avoid errors; setup: public application with CouchDB-based authentication).

Vote: Release Apache CouchDB 1.6.0 rc2 (see thread)

The vote was aborted because of an issue brought up by the test suite.

Modeling Relationships and providing Transactional Integrity (see thread; ongoing discussion)

… and the question if _bulk_docs is transactional (hint: it's not)

Releases in the CouchDB Universe

Opinions

Use Cases, Questions and Answers

Get involved!

If you want to get into working on CouchDB:
  • rcouch Merge: Erlang hackers and CouchDB users, we need your help with testing and review of the rcouch merge. It's easy! Find the how-to in this post.
  • Here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
  • You want join us for the updates of CouchDB-Python for Python 3? Take a look at issue 231.
We'd be happy to have you!

Events

Job opportunities for people with CouchDB skills

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at April 17, 2014 10:03 AM

April 16, 2014

The Old Official CouchDB Blog

Merging rcouch

rcouch is a variant of CouchDB that is derived from the main Apache CouchDB code base. One way to look at it is as a “future port” of CouchDB, that is for two reasons:

  1. The internals are greatly improved. In simple terms, if CouchDB would be started to day, it would look a lot like rcouch.
  2. It comes with a number of nice new features.

So whether you are a CouchDB user or a CouchDB Erlang developer, rcouch is good news for you!

Since CouchDB has been around for a while, the internal codebase is a little crufty. We are constantly working on improving that, but one of the larger efforts have been done within the rcouch project by Apache CouchDB committer Benoit Chesneau.

At the same time, Benoit added a number of nice little features that benefit CouchDB users, just to name a few:

  • _changes for views.
  • Replication from views.
  • Write-only “Drop-Box” databases.
  • Request validation on-read.
  • _db_changes (actually, we nicked that one for Apache CouchDB 1.4 already :).

For more information, see the rcouch website.

Benoit has generously donated the rcouch project back to Apache CouchDB and has worked on merging the rcouch and the CouchDB codebases. Now it is time to review the merged code and get it into Apache CouchDB proper and put it into a new release.

With that, we need your help.

How you can help, you ask? Easy!

If you are a CouchDB user, and are not afraid to build CouchDB yourself, give the rcouch merge branch a spin:

git clone https://github.com/apache/couchdb.git
cd couchdb
git checkout -b 1994-merge-rcouch origin/1994-merge-rcouch

# Read the DEVELOPERS file to install all dependencies

# test regular build:
make

# test build with static icu library
make icu=static

# test build with shared libraries
make libs=shared

# run test suite
make check

# make a release
make rel

# run:
./rel/apache-couchdb/bin/couchdb start

# now CouchDB is running at http://127.0.0.1:5984/_utils/, as usual, you just won’t see any output.


If you are an Erlang developer, we could use your help with code reviews. Any small chunks will help!

Please see the associated JIRA ticket for more information.

Thanks so much!


FaQ: how does this relate to BigCouch?

It doesn’t really, the BigCouch merge is ongoing and we will announce updates and requests for review when the time is right. Well, actually, if we can convince you to help with the review of the rcouch merge, we can put even more resources on the BigCouch merge :)

by Jan at April 16, 2014 08:50 PM

April 10, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, 10 April

Releases

Apache CouchDB 1.5.1 released
This is a security release. Download link | Release notes

Blog Posts

Major Discussions

Writing many docs at once: bench marking a 10k write to a CouchDB server (see thread here)
A discussion around handling and measurements of bulk input in CouchDB.
Question about "complex" range queries (see thread here)
Requirements for sort orders of data views.
Is the revision field deterministic? (see thread here)
Quoting Robert Samuel Newson's reply: "Yes, it’s deterministic. The same document with the same history will have the same _rev value. This is an optimization over the previous algorithm where _rev was a random UUID. […] The advantage is that two servers receiving the same update can more optimally replicate. They still have to check that the target has all _id/_rev pairs but will usually be able to skip actually transmitting document and attachment content."

CouchDB Universe

  • Apache Con has taken place from April 7-9. You can find the slides of the talks given here

Releases in the CouchDB Universe

  • New Erlang Version is out
  • mbtiles2couchdb, for using CouchDB as a simple tiles server
  • RCouch has a new home: announcement | the new home
  • PPnet, a minimal social network to drop into websites based on PouchDB
  • sabisu, a sensu web UI, got open sourced by Cloudant last week
  • Janus, an online storage for offline Web Apps built using PouchDB
  • couche 0.0.2, a couchdb client for node, with specific apis
  • mock-couch, an http server pretending to be couchdb, for unit testing
  • couchdb-sync 0.1.1 - a generic couchdb replicator which is basically an eventemitter
  • hackoregon-couch 0.0.2 which exports database information from PostgreSQL and imports into CouchDB
  • overwatch 0.2.7, a deterministic couchdb replication watcher

Opinions

Use Cases, Questions and Answers

Get Involved!

If you want to get into working on CouchDB:
  • here's a list of beginner tickets around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help.
  • Your help on updating CouchDB-Python for Python 3. Wanna join? Please participate here.
We'd be happy to have you!

New PMC Member

  • Joan Touzet joins the Apache CouchDB Project Management Committee today. Joan has made outstanding, sustained contributions to the project. Welcome to the Couch, Joan!

Events

Job opportunities

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at April 10, 2014 10:59 AM

April 09, 2014

The Old Official CouchDB Blog

Apache CouchDB 1.5.1 Released

Apache CouchDB 1.5.1 has been released and is available for download.

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

Grab your copy here:

http://couchdb.apache.org/

Pre-built packages for Windows and OS X are available.

CouchDB 1.5.1 is a security release, and was originally published on 2014-04-09.

These release notes are based on the changelog.

Changes

  • Add the max_count option (UUIDs Configuration) to allow rate-limiting the amount of UUIDs that can be requested from the /_uuids handler in a single request.

by djc at April 09, 2014 05:39 PM

Apache CouchDB 1.5.0 Released

Apache CouchDB 1.5.0 has been released and is available for download.

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

Grab your copy here:

http://couchdb.apache.org/

Pre-built packages for Windows and OS X are available.

CouchDB 1.5.0 is a feature release, and was originally published on 2013-11-05.

These release notes are based on the changelog.

Changes

  • COUCHDB-1781: The official documentation has been overhauled. A lot of content from other sources have been merged, and the index page has been rebuilt to make the docs much more accessible. #54813a7
  • A new administration UI, codenamed Fauxton, has been included as an experimental preview. It can be accessed at /_utils/fauxton/. There are too many improvements here to list them all. We are looking for feedback from the community on this preview release.
  • COUCHDB-1888: Fixed an issue where admin users would be restricted by the public_fields feature.
  • Fixed an issue with the JavaScript CLI test runner. #be76882, #54813a7
  • COUCHDB-1867: An experimental plugin feature has been added. See src/couch_plugin/README.md for details. We invite the community to test and report any findings.
  • COUCHDB-1894: An experimental Node.js-based query server runtime has been added. See Experimental Features for details. We invite the community to test and report any findings.
  • COUCHDB-1901: Better retry mechanism for transferring attachments during replication. #4ca2cec

by djc at April 09, 2014 05:37 PM

April 08, 2014

The Old Official CouchDB Blog

CouchDB and the Heartbleed SSL/TLS Vulnerability

You may or may not have heard about the Heartbleed SSL/TLS vulnerability yet. Without much exaggeration, this is a big one.

What does this mean for CouchDB?

1. If you are using CouchDB with the built-in SSL support, you are at the whim of Erlang/OTP’s handling of SSL. Lucky for you, while they do use OpenSSL for the heavy lifting, they do the TLS/SSL handshake logic in Erlang (Source). That means you are not affected by this issue.

2. If you are using CouchDB behind a third-party proxy server you are at the whim of the SSL library it uses. For the big three Apache, nginx and HAProxy it’s all OpenSSL. So if they are using OpenSSL 1.0.1-1.0.1f with heartbeat support (RFC6520) enabled (the default), you need to take action. As far as I can tell now:

  • 0. Check if you are vulnerable
  • 1. Stop your service.
  • 2. Upgrade to OpenSSL 1.0.1g or recompile OpenSSL without heartbeat support.
  • 3. Request new cert from your SSL cert vendor.
  • 4. Revoke your old cert.
  • 5. Invalidate all existing sessions by changing the CouchDB couchdb_httpd_auth/secret configuration value to a new UUID.
  • 6. Restart your service.
  • 7. Invalidate all your user’s passwords and/or OAuth tokens.
  • 8. Notify your users that any of their data and passwords are potentially compromised.

by Jan at April 08, 2014 03:53 PM

The Little Things(1): Do Not Delete

CouchDB takes data storage extremely seriously. This usually means we work hard to make sure that the CouchDB storage modules are as robust as we can make them. Sometimes though, we go all the way to the HTTP API to secure against accidental data loss, saving users from their mistakes, rather than dealing with hard drives and kernel caches that usually stand in the way of safe data storage.

The scenario:

To delete a document in CouchDB, you issue the following HTTP request:

DELETE /database/docid?rev=12345 HTTP/1.1

A common way to program this looks like this:

http.request('DELETE', db + '/' + docId + '?rev=' + docRev);

So far so innocent. Sometimes though, users came to us and complained that their whole database was deleted by that code.

Turns out the above code creates a request that deletes the whole database, if the docId variable isn’t set correctly. The request then looks like:

DELETE /database/?rev=12345 HTTP/1.1

It looks like an honest mistake, once you check the CouchDB log file, but good old CouchDB would just go ahead and delete the database, ignoring the ?rev= value.

We thought this is a good opportunity to help users not accidentally losing their data. So since late 2009 (yes, this is an oldie, but it came up in a recent discussion and we thought it is worth writing about :), CouchDB will not delete a database, if it sees that a ?rev= parameter is present and it looks like that this is just a malformed request, as database deletions have no business requiring a ?rev=.

One can make an easy argument that the code sample is fairly shoddy and we’d agree. But we are not here to argue how our users use our database beyond complying with the API and recommended use-cases. And if we can help them keep their data, that’s a win in our book

Continuing down this thought, we thought we could do one better. You know that to delete a document, you must pass the current rev value, like you see above. This is to ensure that we don’t delete the document accidentally without knowing that someone else may have added an update to it that we don’t actually want to delete. It’s CouchDB’s standard multi version currency control (MVCC) mechanism at work.

Databases don’t have revisions like documents, and deleting a database is a simple HTTP DELETE /database away. Databases, however, do have a sequence id, it’s the ID you get from the changes feed, it’s an number that starts at 0 when the database is created and increments by 1 each time a document is added, updated or deleted. Each state of the database has a single sequence ID associated with it.

Similar to a rev, we could require the latest sequence ID to delete a database, as in:

DELETE /database?seq_id=6789

And deny database deletes that don’t carry the latest seq_id. We think this is a decent idea, but unfortunately, this would break backwards compatibility with older versions of CouchDB and it would break a good amount of code in the field, so we are hesitant to add this feature. In addition, sequence IDs change a little when BigCouch finally gets merged, so we’d have to look at this again then.

In the meantime, we have the protection against simple coding errors and we are happy that our users keep their hard earned data more often now.

by Jan at April 08, 2014 01:59 PM

April 03, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, April 3

Major Discussions

Vote on release of Apache CouchDB 1.5.1-rc.1 (will be released as Apache CouchDB 1.5.1 — see thread)

The vote passed.

Importing CSV data into a CouchDB document using Python-Cloudant module (discussion still open — see thread)

Approaches brought up: (1) Transforming a CSV file into a JSON file with Python; (2) rc_scv, a direct rcouch extension; (3) using Google Refine and Max Ogden's refine uploader (see visual example here); (4) CSV2Couch. Further information can also be found in this blog post on "Using Python with Cloudant" and the newer Cloudant-Python interface

CouchDB 1.6.0 proposals (see thread)

Discussion around the open blocker plus how to deal with it and around re-cutting 1.6.x from master. Releasing 1.5.1 will take precedence.

Poll around Erlang whitespace standards (see thread; the poll is still open)

Joan Touzet: "I know many of you are fed up with not being able to auto format in your favourite editor and match the CouchDB Erlang coding standards, or receiving pull requests that are formatted poorly. I'd like to fix that with an appropriate whitespace standard, and supplementary plugins for vi and Emacs that will just Do The Right Thing and let us all stop worrying about whitespace corrections in pull requests."

There's currently a poll around this topic which is still open.

Multiple Concurrent Instances of CouchDB on Mac OS X 10.7.5 (see thread)

Approaches: (1) for a powerful enough machine Vagrant could be used to spin up a few CouchDB VMs, e.g. with CouchDB-Vagrant. Other options could be: (2) using Docker, (3) Node-Multicouch (used in Hoodie) or (4) this script to configure isolated instances, it should be possible to point couchdb to the CouchDB commands inside the .app.

CouchDB Universe

Releases in the CouchDB Universe

  • Wilt 3.0.0 – a browser/server based CouchDB API library based on the SAG CouchDB library for PHP
  • contentful-2-couchdb – a proof of concept for easy data export to CouchDB and replication use
  • Availability of MariaDB 10 announced
  • PouchDB released a new website
  • PouchDB 2.1.0 release includes e.g. (all release notes here):
    • Support optional leveldown builds
    • Replication performance improvements
    • Fix for localStorage detection in Chrome Apps
    • Improved error reporting from replicator et al.

Opinions

Use Cases, Questions and Answers

Getting involved into CouchDB

If you want to get into working on CouchDB: here's a list of beginner tickets you can get started with. These are issues around our currently ongoing Fauxton-implementation. If you have any questions or need help, don't hesitate to contact us in the couchdb-dev IRC room (#couchdb-dev) – Garren (garren) and Sue (deathbear) are happy to help. We'd appreciate having you!

New Committers and PMC Members

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at April 03, 2014 10:19 AM

March 27, 2014

The Old Official CouchDB Blog

CouchDB Weekly News, March 27

Welcome to the Apache CouchDB weekly news! From today on, we’ll bring you the highlights of what happened in the CouchDB Universe each thursday.

Major Discussions

BigCouch vs. Couchbase
A comparison between the two, around conflict solving, revision docs, nodes, replicas and hashing. Find the thread here.
Healthcare projects running on CouchDB
Albin Stigö asked for healthcare companies / projects using CouchDB. In the thread, a collection of examples and reasons why this is a good fit was discussed. CouchDB works well in healthcare due to its relative ease of configuration and deployment, easy synchronization, backups, attachment-handling and notifications and mobile-readiness. Some examples of projects / companies using CouchDB: CommCare HQ, Mobius, Neurofoundation. Find his summary and the entire thread here.

Weekly CouchDB meeting – summary

  • 1.6.rc-2: discussion around blocker; will be collated and sent to the mailing list
  • Marketing: there'll be a weekly CouchDB news (which you are currently reading)
  • Fauxton - status: Futon will be removed as soon as Fauxton-implementation is done; following release will be marked as beta
  • Merges: discussion around BigCouch and rcouch merges will also be collated and sent to the mailing list

CouchDB Universe

Opinions

… and also in the news

Posted on behalf of Lena Reinhard.

by Noah Slater at March 27, 2014 03:29 PM

March 24, 2014

Joan Touzet

Hello world!

This is a test post for integration with Planet CouchDB, the aggregator for all things bloggy and CouchDB-related. I’m excited to be a part of the group!

by Wohali at March 24, 2014 01:02 PM

March 13, 2014

The Old Official CouchDB Blog

On The CouchDB Community

Jan Lehnardt sent a link to the CouchDB marketing@couchdb.apache.org mailing list where Ari Najarian used CouchDB in a five part course to teach beginners how to program. It’s a great read and I invite you to visit the page.

I would like to slice out one quote from the article where Ari described the CouchDB community:

“The personalities and voices in the CouchDB community are, hands down, the most helpful, generous, welcoming, insightful, ethical, principled, future-minded group of developers I’ve ever encountered in one place. As far as online communities go, you can’t find a safer, more encouraging group of advocates. These people are role models that will help beginners to understand what “open-source” can really be about: social justice, diversity, inclusion, and collaboration.”

This is the reason why I became a part of the Apache CouchDB project - because the community is so awesome.

by andywenk at March 13, 2014 04:49 PM

February 24, 2014

The Old Official CouchDB Blog

The State of CouchDB

This is a rough transcript of the CouchDB Conf, Vancouver Keynote.

Welcome

Good morning everyone. I thank you all for coming on this fine day in Vancouver. I’m very happy to be here. My name is Jan Lehnardt and I am the Vice President of Apache CouchDB at the Apache Software Foundation, but that’s just a fancy title that means I have to do a bunch of extra work behind the scenes. I’m also a core contributor to Apache CouchDB and I am the longest active committer to the project at this point.

I started helping out with CouchDB in 2006 and that feels like a lifetime ago. We’ve come a long way, we’ve shaped the database industry in a big way, we went though a phoenix from the ashes time and came out still inspiring future generations of developers to do great things.

So it is with great honour that I get to be here on stage before you to take a look at the state of CouchDB.

Numbers

I’d like to start with some numbers:

  • In 2013 we added 15 committers to the project, up to a total of 30. Thats 2x the number of people regularly contributing to CouchDB!
  • The year isn’t yet over, but these committers already created 3x the commits of 2012. And they have committed more than in any other year in CouchDB’s history.
  • We have shipped eight releases: 1.0.4 1.1.2, 1.2.1, 1.2.2, 1.3.0, 1.3.1, 1,4.0 and 1.5.0 just this year, that is up from one(!) last year.
    • thanks to our new release schedule we are getting more features to more people faster by focusing on small iterative changes forward.
  • 20% more JIRA tickets and 50% more GitHub issues

We have made a lot of changes in 2012 to make 2013 a great year for CouchDB and it sure looks like we succeeded and that 2014 is only going to trump that.

I’d like to thank everyone on the team for their hard work.

Currently

We’ve just shipped CouchDB 1.5.0 last week and it comes with a few exciting new things as previews, for you to try out and play with and report any issues with back to us. And that is on top of all the regular bug fixing and other improvements.

  1. A completely new developed admin UI, nicknamed Fauxton, that is poised to replace the much-loved, but increasingly dated Futon. I’d like to personally thank the Fauxton team: Sue “Deathbear” Lockwood, Russell “Chewbranca” Branca, Garren Smith and many more volunteers for their work as well as the company Cloudant for sponsoring a good chunk of that work. Great job everyone! Fauxton is going to be replacing Futon in one of the next few releases and will give us the foundation for the next stage of CouchDB’s life.

  2. Plugins. While it was always possible to write plugins for CouchDB, you kind of had to be an expert in CouchDB to get started. We believe that writing plugins is a great gateway drug to getting more people to hack on CouchDB proper, so we made it simpler to build plugins and to install plugins into a running instance of CouchDB. It is still very early days, we don’t even have a plugin registry yet, but we are surely excited about the prospects of installing GeoCouch with a single click of a button in Futon or Fauxton. We also included a template plugin that you can easily extend and make your own, along with a guide to get you started.  

    The plugins effort also supports a larger trend we are starting to follow with the CouchDB core codebase: decide on a well-defined core set of functionality and delegate more esoteric things to a rich plugin system That means we no longer have to decline the inclusion of useful code like we’ve done in the past, because it wasn’t applicable to the majority of CouchDB users. Now we can support fringe features and plugins that are only useful to a few of our users, but who really need them.

  3. A Node.JS query server. CouchDB relies on JavaScript for a number of core features and we want to continue to do so. In order to keep up with the rapid improvements made to the JavaScript ecosystem we have tentative plans to switch from a Spidermonkey-driven query server to a V8-driven one. In addition, the Node.js project has a really good installation story, something that we had trouble with in the past, and includes a few utilities that make it very easy for us to switch the query server over.

    All this however is not to blindly follow the latest trends, but to encourage the community to take on the query server and introduce much needed improvements. The current view server is a tricky mix of JS, Erlang and C and we are not seeing many people daring to jump into that. In a second step we expect these improvements to trickle down to the other query server implementations like Python or PHP and make things better for everyone. For now this is also a developer preview and we are inviting all Node.js developers to join us and build a a better query server.

  4. Docs landed in 1.4.0, but 1.5.0 is seeing a major update to the now built-in documentation system. With major thanks to Alexander Shorin, Dirkjan Ochtmann and Dave Cottlehuber who were instrumental in that effort, CouchDB now has “really good docs” instead of a “really crappy wiki”, that are shipped with every release and are integrated with Futon and Fauxton.

Beyond

The immediate next area of focus for the CouchDB project is the merging of two forks: BigCouch and rcouch.

BigCouch is a Dynamo implementation on top of CouchDB that manages a cluster of machines and makes them look as a single one, adding performance improvements and fault tolerance to a CouchDB installation. This is a major step in CouchDB’s evolution as it was designed for such a system from the start, but the core project never included a way to use and manage a cluster. Cloudant have donated their BigCouch codebase to the Apache project already and we are working on an integration.

rcouch is a what I would call a “future port” of CouchDB by longtime committer and contributor Benoit Chesneau. rcouch looks like CouchDB would, if we started fresh today with a modern architecture. Together with BigCouch’s improvements, this will thoroughly modernise CouchDB’s codebase to the latest state of the art of Erlang projects. rcouch also includes a good number of nifty features that make a great addition to CouchDB’s core feature set and some great plugins.

Finally, we’ve just started an effort to set up infrastructure and called for volunteers to translate the CouchDB documentation and admin interface into all major languages. Driven by Andy Wenk from Hamburg, we already have a handful of people signed up to help with translations for a number of different languages.

This is going to keep us busy for a bit and we are looking forward to ship some great releases with these features.

tl;dr

2013 was a phenomenal year for Apache CouchDB. 2014 is poised to be even greater, there are more people than ever pushing CouchDB forward and there is plenty of stuff to do and hopefully, we get to shape some more of the future of computing.

Thank you!

by Jan at February 24, 2014 05:53 PM

Cloudant and IBM: Our Commitment to Apache CouchDB

IBM have announced that they are acquiring Cloudant.

Adam Kocoloski is co-founder and CTO of Cloudant as well as serving on CouchDB's Project Management Committee. And with his permission, I am re-posting an email he sent to the CouchDB developer list a few hours ago.

In his words:

"Apache CouchDB means a great deal to me, and to Cloudant as a company. Cloudant and CouchDB have grown alongside each other over the past several years in one of the more authentic vendor/community collaborations I can think of in Apache history. Today marks the next step in Cloudant’s growth as we enter into a definitive agreement to become part of IBM.

"What does this mean for CouchDB? I would not have agreed to this transaction if I had any concerns about Cloudant’s ability to continue its contributions and collaboration with Apache CouchDB. IBM has a strong track record in open source software and a productive relationship with Apache; in fact, IBM was instrumental in bringing CouchDB to the ASF many years ago. IBM is fully supportive of our efforts here, and I’m looking forward to bringing increased resources to bear in support of the project.

"CouchDB has the potential to shape the future of distributed data management and computing. 2013 was a year of tremendous progress, as we doubled our committer base and shipped no fewer than eight releases. Already in 2014 we’ve seen amazing progress on long-standing initiatives to enhance the core of the system. The timing is right — in the market and for the community — to take the next big step forward. With your help, that is exactly what we will do.

"Truly, the future of CouchDB is CouchDB."

Thanks Adam!

I'd like to extend my hearty congratulations to everyone at Cloudant.

The future for CouchDB looks very interesting...

by Noah Slater at February 24, 2014 05:47 PM

February 14, 2014

The Old Official CouchDB Blog

Board Report (Februrary 2014)

Description

CouchDB submits a board report every quarter. Here's a copy of our February board report.

Releases

No releases in this time period.

Release has been blocked on a number of issues, but is actively being worked on.

Recent Activity

  • Number of the core devs met in Vienna to discuss the pending merges.
  • Finalising the merge of Benoit Chesneau’s rcouch fork.
  • Working on the BigCouch merge from Cloudant.
  • Created a replication list to discuss specific replication topics.
  • New marketing list about to be created for new marketing team.
  • Confluence wiki set up, and migration being planned.
  • Review Board instance set up, and discussion about review ongoing.
  • Translation work going well.
  • Community-provided packages added for Ubuntu Precise and upcoming Trusty release.

Community

Including the following additions, CouchDB has 31 committers and 9 PMC members.

New committers:

  • Nick North

No new PMC members. Most recent PMC addition Nov 9th, 2012.

Mailing list stats:

  • announce
    • 151 subscribers (+20)
    • 1 message since May (-1)
  • user
    • 1405 subscribers (-26)
    • 1086 messages since May (+280)
  • erlang
    • 154 subscribers (+8)
    • 14 messages since May (-8)
  • dev
    • 602 subscribers (-4)
    • 1977 messages since May (+1097)
  • commits
    • 104 subscribers (-1)
    • 2735 messages since May (+1694)
  • l10n
    • 31 subscribers (+10)
    • 207 messages since May (+104)
  • replication
    • 47 subscribers
    • 35 messages since May

Issues

None.

by Noah Slater at February 14, 2014 05:24 PM

Board Report (November 2013)

CouchDB submits a board report every quarter. Here's a copy of our November board report.

Description

Apache CouchDB is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API.

Releases

Recent Activity

  • Finalising the merge of Benoit Chesneau’s rcouch fork.
  • Preparation of CouchCamp Vienna, a community run weekend hackathon.
  • Successful run of CouchDB Conf Vancouver with ~80 attendees, 13 speakers. Videos will be available.
  • Created an I18N list and translation team to drive translation efforts.
  • Passed a proposal on using Influitive to boost our community engagement.

Community

Including the following additions, CouchDB has 29 committers and 9 PMC members.

New committers:

No new PMC members. Most recent PMC addition Nov 9th, 2012.

Mailing list stats:

  • announce
    • 131 subscribers (+37)
    • 2 message since August (+1)
  • user
    • 1431 subscribers (-10)
    • 806 messages since August (+310)
  • erlang
    • 146 subscribers (+25)
    • 22 messages since August (15)
  • dev
    • 606 subscribers (+7)
    • 880 messages since August (-130)
  • commits
    • 105 subscribers (-2)
    • 1041 messages since August (+344)
  • l10n
    • 21 subscribers (+21)
    • 103 messages since August (+103)

Issues

None.

by Noah Slater at February 14, 2014 05:20 PM

February 13, 2014

The Old Official CouchDB Blog

Packages for Ubuntu 12.04 Precise

I’m delighted to announce the availability of Apache CouchDB Ubuntu Precise (12.04) packages, with integrated upstart support.

These have been contributed with support from Mobius Medical Systems, LP, who use Apache CouchDB in their product Mobius3D/FX for oncology and radiology, and funded the development of this work through the Couch Firm.

This would not have been possible without the assistance of Jason DeRose (who has lead the 1.5.0 package going into the next Ubuntu Trusty release), and Noah Slater, who was surprised on reviewing the package to find so much of his original work there!

Installation should be straightforward, usual caveats around making backup before upgrading precautions apply, noting that between 1.0.1 and 1.5.0 there are a number of key changes, including the on-disk data format used by the .couch files was upgraded. Although it should keep your dbs intact and do the right thing for your local.ini files, always have a backup - or two!

sudo apt-get install python-software-properties -y
sudo add-apt-repository ppa:couchdb/stable -y
sudo apt-get update -y
# remove any existing couchdb
sudo apt-get remove couchdb couchdb-bin couchdb-common -yf
# see my shiny 1.5.0 goodness
sudo apt-get install -V couchdb
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
       couchdb-bin (1.5.0-0ubuntu2)
       couchdb-common (1.5.0-0ubuntu2)
       couchdb (1.5.0-0ubuntu2)
     … Y …

# manage via upstart
sudo stop couchdb
    couchdb stop/waiting
sudo start couchdb
    couchdb start/running, process 3541
…

You can of course modify or duplicate the /etc/init/couchdb.conf script, to support running multiple CouchDB instances or users on the same box.

Note there's also a couchdb/dev PPA which will be used for testing and updating the next release, e.g. the upcoming 1.6.0, before switching it over to the stable branch once it's had enough eyes on it.

by dch at February 13, 2014 12:19 PM

December 11, 2013

The Old Official CouchDB Blog

CouchDB Conf Videos

Videos from CouchDB Conf are now available.

CouchDB everywhere with PouchDB
Dale Harvey, Mozilla
PouchDB is CouchDB written in JavaScript, that means it runs everywhere, I mean everywhere. Find out about why this changes everything for CouchDB and app developers, as well as the latest updates and status of the project.

CouchDB Writ Large
Mike Miller, Cloudant
In this talk, Mike Miller describes how CouchDB fits into Cloudant's software stack as one of many open source projects the company uses to run distributed database clusters around the world.

Replication, FTW!
Benjamin Young, Cloudant
Jan's "call to arms" earlier this year made replication the central focus of Apache CouchDB. Replication is the feature that makes CouchDB so distinctive in any marketplace (open or otherwise) and sets it apart from the NoSQL pack. We'll take a look at the various replicating databases, how we can extend the reach of replication, and how we can help others find their way into this world of JSON juggling.

10 Common Misconceptions about CouchDB
Joan Touzet, Atypical
I've consulted with hundreds of people who use CouchDB, and the same sorts of questions keep coming up. Come to this talk if you want to know more about the kinds of mistakes many users make when thinking about how to use the database in their application. I'll talk a bit about the "rough edges" of CouchDB, and how to work them to your advantage.

Experiences using CouchDB inside Microsoft's Windows Azure team
Will Perry and Brian Benz
Will Perry, a developer on the Windows Azure team, is part of a small team at Microsoft that has been using CouchDB extensively on an internal project. In this talk, he'll share motivation for using CouchDB, experiences using and running CouchDB in Azure, design and architectural patterns the team used and a quick run-down of some of the things they love and struggle with on a team more used to relational SQL DBs.

Deep Dive into a Shallow Write Pool
Jason Johnson, IBM SoftLayer
When storing data safely, disk I/O is almost always a major bottleneck. In this talk, we'll examine some of CouchDB's write patterns and explore how we can tune underlying storage media to complement those patterns.

Say Hello to Fauxton, the New Web Dashboard for CouchDB
Russell Branca, Cloudant
In this talk, Cloudant Engineer Russell Branca discusses the new Web dashboard that will soon be released for Apache CouchDB.

Making CouchDB Plugins
Jason Smith, Nodejitsu
CouchDB now has plugin support! This talk explains the design of the plugin system and demonstrates extending CouchDB by building example plugins. From "my first Erlang project" to major CouchDB changes, plugins have something for everybody.

Personal Web Apps
Nathan Vander Wilt, freelance product developer
CouchDB is great at keeping track of all sorts of personal data — everything from notes and contacts, to sharing a live stream of photos from a balloon 250 feet above a construction site. I'll demo of some of the more interesting apps I've been building for my "personal cloud" and talk about the CouchDB features which make them easy. We'll also explore what CouchDB *doesn't* do and how to get those things done anyway.

How CouchDB is powering the next-generation cloud storage
Sascha Reuter, doctape / tape.io GmbH
This talk will give you an overview on how we leverage CouchDB to be the heart of our cloud-storage service. It will cover a whole bunch of fascinating real-life experiences, as well as some nifty tricks on how to unleash CouchDB's full potential in production. I'll show off how the mighty changes-stream helps us interconnecting different components of our service, as well as pushing realtime UI updates & notifications to our users. I'll also tell you about things to avoid, to guarantee full performance, data consistency and high availability in production. To the end, I'll show off how we use CouchDB even as our global queue for processing & conversion jobs and why this is actually a pretty good idea.

Open Data with Couch, Pouch, and Cloudant: Liberating the Laws of Massachusetts
Calvin Metcalf
As a member of Code for Boston, the Boston Code for America Brigade I scrapped the complete laws of Massachusetts to JSON, put them into a CouchDB database on Cloudant, and then used Cloudant to implement full-text search and PouchDB to make a demo app for people to browse the laws. Learn how couch can help distribute open data when that data is too big to just dump into github.

War stories from npmjs.org
Charlie Robbins & Jason Smith, Nodejitsu
Q&A with Nodejitsu and npmjs.org, the package manager for the Node.js server-side JavaScript platform — running CouchDB on the back end.

Check out the full playlist on YouTube.

Thank you to Cloudant for recording and making these videos available.

by Noah Slater at December 11, 2013 03:53 PM

November 09, 2013

The Old Official CouchDB Blog

CouchDB will go multilingual - looking for contributors!

The Apache CouchDB project is pleased to announce the creation of a localization team. The l10n@couchdb.apache.org mailing list has been set up to cover everything l10n- and i18n-related in Apache CouchDB.

The first steps will be to set up some tools to enable easy translation of the documentation and (at a later point) the administration interface (Futon/Fauxton). We are looking for people who want to help out. Would you like to read the CouchDB documentation in another language, and have some time to spare? Please help out! Bonus points if you have experience with translation or localization-related tools and best practices.

We’re just getting started: we have some ideas on how to get started, but we need more help. If you’re interested, please subscribe to l10n@couchdb.apache.org and let us know!

Cheers from the Apache CouchDB I10n team

by andywenk at November 09, 2013 10:09 PM

November 05, 2013

The Old Official CouchDB Blog

CouchDB Conf is Next Week

It’s almost time for CouchDB Conf, held in Vancouver on November the 13th. That’s the day before Cascadia JS. If you don’t have your ticket to CouchDB Conf yet, there’s still time secure a place. Come meet the community, and learn about the future of CouchDB. The ticket includes a reception at Cascadia JS, which starts the day after CouchDB Conf.

Purchase your ticket for CouchDB Conf now

The conference will include sessions that span Apache CouchDB and related technologies — covering everything from getting started to advanced features as well as CouchDB internals, application development, and best practices. Attendees will also get updates on other projects in the community such as PouchDB, a Web browser-based database capable of replicating data with Apache CouchDB and other compatible databases. Featured speakers include Jan Lehnardt, Hoodie; Joan Touzet, Atypical; Brian Benz, Microsoft; and Dale Harvey, Mozilla.

Featured Discussions:

  • Apache CouchDB is the database that replicates. CouchDB-style replication is now supported in a wide range of databases that live in browsers (PouchDB), mobile (TouchDB), and the cloud. Many of the talks will cover various aspects of using replication and modeling your data to work with it.
  • Fauxton is the new Futon, CouchDB's Web-based administration console. Fauxton brings a fresh, modular approach to UI for Apache CouchDB. It is available at /_fauxton in Apache CouchDB 1.5, and will eventually become the new /_utils UI.

Apache CouchDB Conf is the event to attend to expand knowledge, meet project committers and contributors, and find out where Apache CouchDB is headed.

Visit the conference website for more information.

We hope to see you there!

by Noah Slater at November 05, 2013 05:04 PM

October 10, 2013

The Old Official CouchDB Blog

Registration Is Open for CouchDB Conf, Nov 13th

CouchDB Conf is a one-day conference in Vancouver, Canada on November 13, 2013. I am excited to let you know that registration is now open! Early bird tickets are $150, and are available until October 13th. (So get yours quick!)

Purchase your ticket for CouchDB Conf now

CouchDB Conf will convene 200 developers and technology evangelists. They will come together to learn about CouchDB, related technologies, and new projects in the community. The agenda will cover everything from getting started to advanced features and internals, app development, and best practices.

View the confirmed speakers

We expect the conference to sell out quickly, so register today to avoid disappointment!

by Noah Slater at October 10, 2013 05:41 PM

Apache CouchDB 1.4.0 Released

Apache CouchDB 1.4.0 has been released and is available for download.

CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can even serve web apps directly out of CouchDB. And you can distribute your data, or your apps, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.

Grab your copy here:

http://couchdb.apache.org/

Pre-built packages for Windows and OS X are available.

CouchDB 1.4.0 is a feature release, and was originally published on 2013-09-03.

These release notes are based on the changelog.

Upgrade Notes

  • We now support Erlang/OTP R16B and R16B01; the minimum required version is R14B.
  • User document role values must now be strings. Other types of values will be refused when saving the user document.

Changes

  • COUCHDB-1684: Support for server-wide changes feed reporting on creation, updates and deletion of databases. #917d8988
  • COUCHDB-1139: it’s possible to apply list functions to _all_docs view. #54fd258e
  • Automatic loading of CouchDB plugins. #3fab6bb5
  • Allow storing pre-hashed admin passwords via _config API. #c98ba561
  • COUCHDB-1772: Prevent invalid JSON output when using all_or_nothing _bulk_docs API. #dfd39d57
  • Add a configurable whitelist of user document properties. #8d7ab8b1
  • COUCHDB-1852: Support Last-Event-ID header in EventSource changes feeds. #dfd2199a
  • Much improved documentation, including an expanded description of validate_doc_update functions (commit:ef9ac469) and a description of how CouchDB handles JSON number values (#bbd93f77).
  • COUCHDB-1632: Ignore epilogues in multipart/related MIME attachments. #2b4ab67a
  • Split up replicator_db tests into multiple independent tests.

by djc at October 10, 2013 12:07 PM

October 04, 2013

Till Klampäckel

SQL MAX() and GROUP BY for CouchDB

While re-writing a couple SQL statements into CouchDB we got stuck when we wanted to do a SELECT MAX(...), id ... GROUP BY id in CouchDB.

MySQL

Imagine the following SQL table with data:

mysql> SHOW FIELDS FROM deploys;
+-------------+-------------+------+-----+---------+-------+
| Field       | Type        | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| project     | varchar(10) | NO   |     | NULL    |       |
| deploy_time | datetime    | NO   |     | NULL    |       |
+-------------+-------------+------+-----+---------+-------+
2 rows in set (0.01 sec)

In order to get the latest deploy for each project, I'd issue:

mysql> SELECT MAX(deploy_time), project FROM deploys GROUP BY project;
+---------------------+----------+
| MAX(deploy_time)    | project  |
+---------------------+----------+
| 2013-10-04 22:01:26 | project1 |
| 2013-10-04 22:02:17 | project2 |
| 2013-10-04 22:02:20 | project3 |
+---------------------+----------+
3 rows in set (0.00 sec)

Simple. But what do you do in CouchDB?

CouchDB

My documents look like this:

{
  "_id": "hash",
  "project": "github-user/repo/branch",
  "deploy_time": {
     "date": "2013-10-04 22:02:20",
     /* ... */
  },
  /* ... */
}

So, after more than a couple hours trying to wrap our heads around map-reduce in CouchDB, it's working.

Here's the view's map function:

function (doc) {
  if (doc.project) {
    emit(doc.project, doc.deploy_time.date);
  }
}

This produces nice key value pairs — in fact, multiple — for each project.

And because the map-function returns multiple, we need to reduce our set.

So here is the reduce:

function (doc, values, rereduce) {
  var max_time = 0, max_value, time_parsed;
  values.forEach(function(deploy_date) {
    time_parsed = Date.parse(deploy_date.replace(/ /gi,'T'));
    if (max_time < time_parsed) {
      max_time = time_parsed;
      max_value = deploy_date;
    }
  });
  return max_value;
}

The little .replace(/ /gi,'T') took especially long to figure out. Special thanks to Cloudant's seńor JS-date-engineer Adam Kocoloski for helping out. ;-)

Step by step

  • iterate over values
  • fix each value (add the "T") to make Spidermonkey comply
  • parse, and compare
  • return the "latest" in the end

A note of advice: To save yourself some trouble, install a local copy of Spidermonkey and test your view code in there and not in your browser.

Open the view in your browser: http://localhost/db/_design/deploys/_view/last_deploys?group=true

{
  "rows":[
    {
      "key":"testowner/testname/testbranch",
      "value":"2013-09-13 11:41:03"
    },
    {
      "key":"testowner/testname/testbranch2",
      "value":"2013-09-12 16:48:39"
    }
  ]
}

Yay, works!

FIN

That's all for today.

by Till Klampaeckel (till@php.net) at October 04, 2013 08:40 PM

February 19, 2013

EasyBib

Cloudant's NYC meet-up & webinar!

Cloudant’s NYC meet-up will take place at our office this Wednesday (tomorrow)! Please RSVP using the following link:

http://cloudantimagineeasy.eventbrite.com/

The office’ address is 12 West 31 St:

<iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="https://maps.google.com/maps?f=q&amp;source=s_q&amp;hl=en&amp;geocode=&amp;q=12+East+31+Street,+NY,+United+States&amp;aq=0&amp;oq=12+West+31+St,+NY&amp;sll=37.0625,-95.677068&amp;sspn=38.826758,78.662109&amp;t=h&amp;ie=UTF8&amp;hq=&amp;hnear=12+E+31st+St,+New+York,+10016&amp;z=14&amp;ll=40.745971,-73.985233&amp;output=embed" width="425"></iframe>
View Larger Map

If you can’t make the meet-up, make sure to check out the webinar on Thursday:

https://www1.gotomeeting.com/register/648768041

by tillinside at February 19, 2013 07:23 PM

February 12, 2013

EasyBib

EasyBib and Cloudant

Slides from a talk I gave at the CouchDB conference in Berlin last month. As always, the slides are on speakerdeck:

https://speakerdeck.com/till/easybib-and-cloudant

— Till

by tillinside at February 12, 2013 02:44 PM

February 06, 2013

IrisCouch

Announcing Premium Server

We’re excited to announce our latest product—a Premium Server option for Iris Couch users. Premium Server gives serious users an option to increase the stability and response time of their couches. It costs $75/month (in addition to the regular, metered fees), and buys you oh-so-much-more peace of mind... and performance.

A Premium server is more responsive and reliable than a standard CouchDB server. Premium servers run on isolated, dedicated clusters.

Interested? Log in and enable Premium Server in our billing page.

Once you have a credit card on file, you will see a Premium Server option. If for any reason you’re dissatisfied with the service, you can cancel at any time.

More Details

What, exactly, is a Premium server?

Premium servers run permanently in specialized clusters, with dedicated CPU and memory. Premium servers run on a common physical server, with dedicated CPU, memory, and i/o resources. Nothing is “oversold.”

Premium servers never pause when idle. Typically, if a couch goes unused for too long, we pause it, freeing resources to share with others. Unfortunately, infrequent usage suffers high latency. Premium servers do not have this issue.

Premium servers are monitored. Our automated tools monitor everything. But people watch every premium server 24/7. If we see any server issues, we respond immediately.

Premium servers are isolated. Each cluster runs in a dedicated DNS zone and uses dedicated HTTP infrastructure. Issues in one cluster do not affect another.

Premium servers work. Thanks to many of our users for months of beta testing. Premium service already supports many couches, such as the Node.js npm registry, a mission-critical CouchDB service, processing over 100 million monthly requests.

by Jeff (jeff@iriscouch.com) at February 06, 2013 02:08 PM

February 01, 2013

Jan Lehnardt

The State of CouchDB

Slides to go with the headings.


Hi, welcome to CouchDB Conf.

CouchDB means a lot to me and it makes me very happy that you are all here today.

Who am I

My name is Jan Lehnardt and I am the elected project management committee chair of the Apache CouchDB project. If you don’t know what this means, don’t worry, it means I sign up to deal with politics and paperwork voluntarily.

I have been with CouchDB since 2006. My first contribution was the initial UNIX build system to CouchDB. I stuck around ever since.

My main job back then was PHP/MySQL consulting and Open Source work as my own 20% project. In 2007, I started giving talks about CouchDB at user groups and conferences and quickly became addicted. People called me the CouchDB Chief Evangelist, although I don’t really like the title. From 2008 I started getting contracting jobs specifically for CouchDB and things blew up quickly that it became clear soon, that an organisation of some sort was needed to handle this. I eventually joined forces with CouchDB’s original inventor and J Chris Anderson to form the company Couch.io. We initially weren’t keen on taking venture capital, but when the whole NoSQL thing blew up, it was very hard to say no and on January 1st 2010 we were in business.

Couch.io turned into CouchOne and was eventually merged with Membase into Couchbase. Unfortunately, at this point, there isn’t much Couch-y about Couchbase aside from some underlying technology, so the name can be confusing.

I parted ways with Couchbase a week ago in order to concentrate and focus more on Apache CouchDB and related technologies.

CouchDB isn’t Dead

In late 2011 two major supporters of CouchDB withdrew their support: Canonical’s UbuntuOne service, on paper a poster-child use-case for CouchDB and Saucelab’s Selenium platform publicly made clear that they were moving to other technology. It wasn’t a big deal if you looked closely, the one project didn’t have enough resources to integrate with CouchDB the right way and the other diverged so far from the original system design, that CouchDB was no longer a good fit. We were and we remain good friends with the respective developers, so we didn’t feel like pointing that out specifically. The result though was that people who didn’t look closely saw that CouchDB had popular failures where it hadn’t. This wasn’t all bad, but set the stage for another incident a few weeks later.

CouchDB’s original inventor publicly announced that he’d move on from Apache CouchDB to Couchbase fully. That new technology would only go into Couchbase. If you are in the know, this wasn’t a big deal again, he hadn’t contributed to CouchDB in two years, Couchbase was a different thing and he wanted to rally support for that. The casual reader though took away one lesson: “CouchDB is dead”.

With the added confusion of Couchbase’s name, and continuing their CouchConf conferences, things looks dire.

CouchDB 2012

With this in the wild, the active CouchDB committers got together and put out an excellent 1.2.0 release that was accompanied by a new, modern website design. We thought instead of writing counter-statements and start unnecessary he-said-she-said discussions, we’d rather concentrate on shipping good software. 1.2.0 was well received and the whole thing lead the team to come together as a whole in Boston and two weeks later in a smaller group in Dublin in April 2012.

We came up with a long list of of features that we wanted to get into CouchDB. Some work happened over the summer, but I personally took a bit of a break to avoid burn-out from CouchDB. In fall, things started moving faster again when more community members started pushing things along. New contributors came and we finally came up with a way to go forward.

We decided to not be insane. The definition of insanity is trying the same thing over and over again and expecting different results. While it wasn’t all bad in CouchDB land we could do better and to do better we had to change a few things.

CouchDB Reboot

We determined that CouchDB needs a fresh mission statement. I signed up to come up with a few drafts, but nothing happened yet, so this one is still out there cough.

We decided to release more often. With the upcoming 1.3.0 release we will switch to a quarterly release cycle. A new minor version will come out approximately every 90 days, even if it is just a few bug fixes that a small number of people direly need.

CouchDB 1.3.0 will ship with top-notch API documentation right inside of Futon, the built in admin interface.

We will start to set up teams where anyone can sign up to work on a specific thing: docs, our JavaScript components, the Erlang core, the admin interface (built in HTML+JS). We want to make sure anyone who has the right skills can contribute in their area of expertise and not be scared away by the Erlang.

We put some emphasis on QA, automated testing, building binaries and packages for releases as well as in-development versions, so people can try new features before they come out more easily and as a result report issues early.

A few of the core team will put some focus on encouraging contributions rather than coding themselves. We hope to get more people on board with CouchDB development.

We also started doing roughly quarterly in-person meetings (at least as smaller groups) and weekly status meetings on IRC to give everything a little more focus and transparency.

The next few slides show a long list a few features that we’ll be working on. If you think anything is missing, let us know!

The more prominent things are: Merging rcouch, BigCouch, the new Futon, the plugin system and v8/node integration.

rcouch is a cutting edge distribution of CouchDB with many experimental features. A lot of that work is useful for Apache CouchDB and we plan to fold those back into the project.

BigCouch is a industry-strength clustering solution for CouchDB to allow seamless scaling and storing of lots of data. BigCouch is going to be merged into mainline CouchDB as well.

The admin interface Futon is getting a complete refresh with modern web development technologies.

There are many features hidden in some developer’s git repository because, while useful, it wasn’t generally applicable to CouchDB. With the plugin system, we hope that we can fuel a repository of plugins to CouchDB that are easily installed (think browser extensions, right from Futon). For example, you could install GeoCouch with the click of a button.

Finally, we are currently discussing the move from SpiderMonkey and to V8 or even Node. The jury is still out on which one it is going to be, but we hope that this is going to open up a lot more opportunities for CouchDB in terms of features and new contributors.


Community

CouchDB would be nothing without you, the community. People who know little about CouchDB might like it or not, but people who really get to know CouchDB are extremely passionate about it.

CouchDB’s core idea, the promise that data can be anywhere it is needed, to make applications easier, more approachable and effective for people in modern computing environments, that idea is a big one. It’s the one that brings this community together.

Thanks again for coming, you are all awesome. CouchDB means a lot to me and I get a little teary seeing you all here.

If you like any of the above todos, or have some of your own, please help out, get in touch about how.

Friends of CouchDB

The CouchDB ecosystem is full of exciting projects that support CouchDB in one way or another.

PouchDB

PouchDB is the newest project on the scene. PouchDB is a pure JavaScript implementation of CouchDB targeted at the browser.

Let me say that again: CouchDB in JavaScript, for the browser.

PouchDB is the oldest idea, but it took the longest to realise because browsers weren’t ready for it. Now is the time though that we can build and ship a CouchDB compatible data store and replication engine that lives entirely in your browser. If you want to build web apps that can seamlessly synchronise data between a laptop, a phone and a server and not treat one as a “master” and the other as “slaves”, PouchDB is your solution.

TouchDB

TouchDB-iOS and -Android are Objective-C and Java implementations of the same idea, but for mobile phone platforms. They natively integrate into your iOS and Android applications and allow synchronisation between themselves, multiple devices, CouchDB’s on the server and even PouchDB in a browser. Again, a milestone in fulfilling the CouchDB vision: data where it is needed, not on some server, far away over a slow and unreliable or even unavailable connection.

Node.js

This might not be too obvious to people who don’t do Node, but CouchDB is big, if not essential here, too. As any programming language & platform, Node.js comes with a package manager called NPM. As far as package managers go (and I’ve used them all except I think ant) NPM is the one that sucks least, or rather, it is pretty good.

If you are invested in Node.js programming NPM isn’t just a tool you can use, it is essential to the development and deployment workflow of your applications. It is that good.

And here is the kicker: NPM’s (de)-central registry is build on CouchDB. NPM is effectively a CouchApp. The Node elders love CouchDB.

I suggest, and plan to turn this into an opportunity for CouchDB. Say, for example, we need benchmarking tools for CouchDB. The Node folks are great at that. If we were to go and say, “hey, do you want to help make NPM faster? Help us build benchmarking tools, so we can speed up CouchDB for you.” — And that’s just one idea, the potential collaboration is immense. Add to that that we maybe get native Node support inside CouchDB, we can do all sorts of crazy shit like writing core add-ons to CouchDB in JavaScript.


Finally, I’d like to mention Hoodie, a project that Gregor, Alex, I and a few friends have been working on on and of over the past year. It is frontend web app framework that uses Node and CouchDB. We are happy to show that to you on the weekend.

The Node Firm

I am going to mention Node.js one more time. Some of you might know The Node Firm, but for those who don’t, think of it as a company made up of the core contributors to Node.js that come together as a collective of professionals to offer services around Node.js that are complementary to the open source project. Things like consulting, technical vetting, training, things that an open source project rarely can do itself.

The key part for me to like The Node Firm (I joined recently) is that, and I repeat, it is complementary to the open source project. It doesn’t try to extract any value from the project, but merely satisfies demand for services that the project can provide. Finally, proceeds are going right back into the community, in the form of conference support, swag, stickers, t-shirts, and so on.

I think this is a brilliant set up for an open source company and I hope many will fork this model.

With that as a preamble, I am happy to announce that together with Dave Cottlehuber and the preliminary support of a few community members, we are launching The Couch Firm, modelled after The Node Firm.

It is early days for us, but expect the same setup, a business complementary to the open source project.

Conclusion

Above all, Open Source is about transparency and community, and I hope you feel you have an accurate picture of where we’re at right now. We had a hard year, lost our traction, and we still came out on top.

With this, I’ll leave the stage for a day of exciting speakers with all sorts of crazy cool topics.

Thank you all for being here!

February 01, 2013 12:00 AM

CouchDB Conf Recap

On January 25-27th 2013 we ran CouchDB Conf Berlin. A one-day conference and two-day CouchHack event dedicated to the Apache CouchDB community and technology.

Why

CouchDB Conf was born out of the desire to run an Apache CouchDB Technology conference. Couchbase’s CouchConf used to be that, but has moved away, while sadly keeping the name.

The CouchDB community is a busy bunch and any excuse to get together is taken up with enthusiasm. CouchDB Conf was sold out with ~70 participants in a wintery Berlin.

Conferences in general are great for various reasons:

  • the core developers get some face time to have a good time, but also to get neck deep into discussions and hacking that require in-person communication. Lots of future plans can be hashed out quickly as well. We made good use of that.

  • the user community gets to hang out with the core developers, learn about how they work and most importantly, get to ask all the fun, hard, silly and important questions they never dared ask before. We also made good use of that.

  • Everybody gets to show of what they are working on and get everybody else excited about it. We made extensive use of that. Exictement was rampant.

Eventify captured a few good impressions.


Talks

The slides for the talks are all linked up on the conference site.

One common thread through all presentations that CouchDB’s replication feature is most the popular thing CouchDB has to offer. HTTP and JSON are nice and simple, but replication is the one thing all would go out of their way to point out that it is the thing that makes CouchDB most attractive.

CouchDB’s vision early on was to be able to run on small clients, big servers, with lots of data and very little data, mobile phones and really any device that one can carry around, and crazily enough also in the browser. For the longest time though this was a bit of a pipe dream, as the Erlang CouchDB implementation wasn’t really suited to run on low-power devices and definitely not in a browser.

This now changed. BigCouch makes the start, as it handles CouchDB’s BigData and clustering aspirations. It has been around for a while, but is now finally being integrated into Apache CouchDB proper.

Hot on the heels is TouchDB, which comes in two flavours: iOS and Android, or Objective-C and Java respectively. Both are native reimplementations of CouchDB for mobile devices. TouchDB-iOS is production ready (a few apps are already live), the Android version is a little behind and both are open for contributions.

The newest player on the scene is PouchDB, a pure JavaScript implementation of CouchDB targeted at modern desktop and mobile browsers. Albeit being the newest member of the CouchDB family, it has the highest activity and was recently selected to act as a sample project in the Stanford Computer Science course. It is on the verge of being production ready.

All of this finally brings CouchDB’s vision together. Lots of data can live on BigCouch clusters, regular data can live on regular servers, user-data can live on mobile handsets, tablets and even in the browser and data can float freely between all of them. Application developers no longer have to worry about how to get a user’s data in front of them, handle concurrent updates, sync with offline devices and online servers. Even peer-to-peer sync is possible.

Towards the end of the day, when the whole vision inadvertently came together, the energy levels were very high. And nothing fuels developers better than seeing users getting excited.

Shit is getting real, as the kids say.

CouchHack

The following weekend was dedicated to hands on hacking, some more cool show & tell type talks and a Skype-powered video talk from Bangkok.

Among the highlights are: - CouchDB/rcouch running on the RaspberryPi with Zeroconf auto-networking and everything. - A preliminary port to BlackBerry 10. - Wallaby, a native QT-based app framework that uses CouchDB for data storage & sync. - Hoodie, a pet project of ours. - CouchDB Operations. - Hands on hacking with PouchDB. - Hands on hacking with TouchDB. - Hands on hacking with PouchDB and TouchDB. - Clearing out 30+ issues in preparation for the 1.3.0 release. - Adding GeoCouch functionality to PouchDB.

Use the Time Slider feature to see the topics for all days.

Thanks

As the organiser, I couldn’t be more happy about how CouchDB Conf has turned out. Of course, I couldn’t have it all done by myself, so I’d like to thank the following people and companies:

The Speakers

Katrin Apel, Dale Harvey, David Zuelke, Martin Hilbig, Nuno Job, Volker Mische, Oliver Kurowski, Jason Smith, Robert Newson, Joan Touzet, Stephan Strohmeier, Till Klampaeckel and Randall Leeds.

The Helpers

Gregor Martynus, Dave Cottlehuber and Jan Friedensreich.

The Companies

Co.Up for hosting us, Tito.io for the flawless ticketing, GitHub & Cloudant for the drinks, Nodejitsu, FreshX, Mozilla, EasyBib and IrisCouch for sending people or donating company time.

Next

There are preliminary plans to set up a CouchDB Conf in Seattle later this year. We’ll keep you posted.

Members of the CouchDB Community will be attending the following conferences:

You are welcome to join us and nerd out about CouchDB :)

Thanks!
Jan

February 01, 2013 12:00 AM

Changes and New Beginnings

About two weeks ago I parted ways with Couchbase’s day to day business. Not that I was involved much in the end, but it was time to move on. I remain a co-founder and shareholder, but my regular involvement is over.

This wasn’t an easy decision, and as those go, it took me a good while to make it.

About a year ago I started noticing a change. Things I’ve always cared about intensified manyfold and my work situation didn’t leave me with much time and energy to pursue them.

Couchbase made every attempt to accommodate me, let me work on Apache CouchDB for most of the past year, and help with an arrangement where I could spend some of my time at my discretion (with no pay, but enough work to keep me afloat) and I am very grateful for that opportunity.

I spare you the details (ask me over a drink), but in the end I had to answer the question of whether I am a CouchDB person or a Couchbase person. I had to answer that for my working relationship with Couchbase and ultimately I had to answer that for myself.

My heart knew, of course, but leaving a company that you helped founding isn’t the easiest of things to consider. I carried that conflict through most of 2012 and I tried to make things work in a compromise, but ultimately failed, realising that I am, at heart, a CouchDB person.


2012 wasn’t the best of years for CouchDB. That helped me realise how much I believe in the idea and the community behind it.

I learned a lot about myself, turned 30 in December. Around that time, the introspective period of the year turned into action. I was done contemplating what to do with newfound revelations and had a vague plan of the direction things should go towards.


There are a number of distinct moments that helped me figure out what is important to me.

Rails Girls

The Rails Girls movement teaches non-programmers web programming for free. Volunteers from the Rails community would spend their weekends teaching, regular week-nights coaching development groups. All stuff I thought was honourable, but I didn’t see how I could fit that into my schedule, along my regular work and have enough energy to do it all. Yet I saw others doing just that. This created an urgency for changes, so I could, while not teaching Rails Girls, do things I deem important. More on that later.

In the spirit of Rails Girls we started a JavaScript workshop JSFAB (JavaScript for Absolute Beginners) which, along with other similar initiatives eventually culminated in OpenTechSchool, a roof organisation for volunteer learning and teaching. The idea spreads like fire, still, into other tech communities, and other cities and countries. This is truly a tech education innovation. I am very proud to be a part of this.

Gleicherlohn.de

Around the US Equal Pay Day Gina Trapani put up Narrow the Gapp. Anne suggested there should be something like that for Germany, so I spent a total of four or so hours to rip off the project’s UI (open source) and made it so Anne could put in the hard data into a JSON file on GitHub.

Gleicherlohn.de was a mini project with a large effect. The lesson for me though was that web technology, while incredibly accessible already, is still not easy enough to handle for regular people, even technically inclined non-programmers struggle. Why, I thought, would Anne not be able to put this up all by herself.

Then I realised what my original motivation for working on CouchDB and eventually founding Couch.io was: empowering people to use technology to their advantage. I wanted to get back to that.

Giving Back

While talking to friends last winter I made the realisation that we technology people are a rather privileged bunch. Not that there is anything wrong with that, but for me that means finding ways to give back to the communities that enable me to live such a privileged life. Teaching and making technology more accessible fit right in there, so it wasn’t hard to figure out what to do. Instead of working five or seven days a week, I tried to find a way to work as much as I need to live comfortably, while having enough time and energy to pursue pro-bono work. Again, Couchbase was very accommodating here.

Diversity

One last topic that found its way to me is diversity in tech communities. The numbers don’t look particularly good, so I tried to find ways to get this improved. This lead to helping Tiffany to get We Are All Awesome set up, start our diversity-friendly Call for Papers for JSConf EU and the result that we were able to organically get 25% women speakers, an absolute novelty for hardcore tech conferences.

More importantly though is that we inspired many more events to follow in our footsteps, improve on what we did and share the results. I lost count of how many conferences took after us, but is easily 20+. The resulting cultural change is immense. I couldn’t be more proud of that.


I want to make sure I can continue to do this work and I hope I can inspire more people to do the same in the way that works best for them. In terms of work, I want to be able to work on things that matter and get paid fairly, but also not over-do it. Leave enough time to work outside of work.

To that end, I am doing three things:

The Node Firm is a complementary company to the Node.js open source project. A collaboration of professionals to support the open source projects in areas it can’t provide itself.

The Couch Firm is exactly that, but for CouchDB. It is early days, I’ll keep you posted.

Hoodie is a thing to be revealed at a later time, but it is not exactly a secret, if you know where to look.


2013 looks mighty exciting from where I am sitting.

I can’t wait to get started!

February 01, 2013 12:00 AM

June 16, 2012

Volker Mische

OSGeo Code Sprint Bolsena 2012

The OSGeo Code Sprint in Bolsena/Italy was great. Many interesting people sitting the whole day in front of their laptops surrounded by a beautiful scenery. This year I spent most of my time on GeoCouch, CouchDB, Couchbase, deegree, Imposm and GeoNetwork.

Already on the first hacking day we had a result, a Couchbase backend for the deegree blob storage. This means that you can now store your rich features (e.g. from INSPIRE) in Couchbase and serve them up as a WFS with deegree. In case you wonder what rich features are, it's the new, shiny and more descriptive name for complex features.

In the following days I worked together with Oliver Tonnhofer on a CouchDB/GeoCouch backend for Imposm. You are now able to store your OpenStreetMap data in CouchDB and make queries on it through GeoCouch. I've created a small demo that displays the some data import from Andorra with directly with MapQuery, without the need of any additional server/service. The CouchDB backend should be easily adaptable to Couchbase, if you want to help, let me know.

I've then spent some time on the GeoNetwork project and helped translating the language file to German. I cleaned it up a bit and fixed major wrong translations. It's not perfect yet, as I've only spent little time on it, but at least it should be way more understandable (and sadly less funny) than the original version which was generated by Google Translate.

When it was time for presentations, I give a quick overview over the Couch* ecosystem. From CouchDB to GeoCouch, BigCouch, PouchDB, TouchDB (TouchDB-iOS, TouchDB-Android), Couchbase Syncpoint and Couchbase. You can find the slides as PDF here.

On the last day I've spent my time on polishing GeoCouch a bit and getting it ready for the Couchhack in Vienna. I've backported all changes from Couchbase to the CouchDB 1.2.x branch and also ported the geometry search into an experimental branch. You can now search your GeoCouch with any Geometry GEOS supports.

The event was fun as always and I also get to know some new people (hello B3Partners guys). Thanks Jeroen from GeoCat for organizing it, and thanks all other hackers that made it such a awesome event. Hope to see you all next year!

by Volker Mische at June 16, 2012 08:27 PM

April 11, 2012

IrisCouch

Announcing paid cloud CouchDB hosting

April 12 is Iris Couch’s one year birthday. Today, we offer paid cloud CouchDB hosting plans.

How it Works

Our pricing plan has the details. It is relaxing and RESTful. Your log file is your receipt.

This month is completely free for everybody. May is the first month under the new policy.

  • People who don’t accrue $5/month of usage don't owe anything for that month
  • The vast majority of users don’t accrue $5/month in usage charges.
  • Thus, by the transitive property of service pricing, most of you won’t pay anything
  • Your account shows your historical usage since you signed up

Below, co-founders Jeff and Jason reflect on our milestone.

Jeff’s thoughts

I’m really proud of how dialed-in this service has become with Jason’s expertise. I’m excited about this launch and new features we have in development. Mostly, though, I’m really looking forward to not feeling so sad every month signing huge checks to pay our hosting bills.

We spent a lot of time mulling over how to price this so that beginners and tinkerers can still have a free sandbox, and I think what we’ve come up will protect that. Thus Iris Couch will remain both a platform and a tool to spread the word about how relaxing CouchDB is.

Jason’s thoughts

I am pleased with our first year. We have built technology I am proud of. Many people (Jeff in particular) have asked:

Why the delay for paid stuff? Aren't you hungry? Are you lazy?

We are both hungry and diligent. But despite being an Apache CouchDB developer, I am a sysadmin first. Before I take your money to steward your data, I have an ethical duty to produce a proven, reliable service.

  • We proved our mettle, in production, for two years (CouchOne plus Iris Couch)
  • We have had zero data loss incidents.
  • We have zero single points of failure.
  • We have a confirmed, tested backup process
  • We have a confirmed, tested off-site disaster recovery process

Is it perfect? No, it is better than “perfect.” It is very good. We built something befitting a proper database company.

by Jeff (jeff@iriscouch.com) at April 11, 2012 05:00 AM

April 07, 2012

IrisCouch

Upgrade to Apache CouchDB 1.2.0

Just now, the Apache CouchDB project released version 1.2.0.

I am happy to announce that all Iris Couch accounts are now running this release. You needn’t do anything, the upgrade is automatic. Just relax on the Couch!

Notable improvements:

  • Persistent cookies—users remain logged in through a browser restart
  • More secure _users database
  • Better, faster replication
  • Reduced (and faster) disk usage

We also upgraded to the new Ocasta Labs Facebook Authentication plugin. It is now easy and relaxing to build a Couch app letting users log in via Facebook.

Please email us if you have questions or issues, or visit our support forum. Thanks!

by Jason (jhs@iriscouch.com) at April 07, 2012 12:49 AM

January 11, 2012

IrisCouch

The future of awesome databases

Damien Katz’s blog post, The Future of CouchDB deserves a response.

Damien tackles difficult problems, thinking clearly and executing well. If he is focusing on Couchbase, then Couchbase will be outstanding.

Both CouchDB and Couchbase are domain-specific databases. Moreover, these domains intersect minimally. The projects represent non-overlapping magisteria; and one might say instead that the future of awesome datatabases is Couchbase.

We are interested in CouchDB’s domain, web applications modeled after the web itself: standard, decentralized, and skewed toward empowering regular people.

by Jason (jhs@iriscouch.com) at January 11, 2012 03:19 AM

January 09, 2012

Till Klampäckel

The future of CouchDB

… is not Damien Katz.

TL;DR

The blog post Damien Katz wrote earlier today, doesn't mean much or anything for the Apache CouchDB project (or memcache project for that matter). If anything it's a public note that Damien Katz acknowledged that he moved (on) from CouchDB to Couchbase.

Short story, long

I'm not a contributor to CouchDB by means of code, (but) I blog a lot, I maintain the FreeBSD port, wrote a book and have an opinion on many things CouchDB. I've been a CouchDB-user for something like four years (since pre-0.8 times) and a BigCouch-user (and Cloudant customer) of about 1.5-two years.

I am not sure what Damien Katz tried to achieve when he posted his message to the community and while I personally find it ignorant (to say the least), it worries me how it is perceived by the general public.

Talk is cheap

Of course it may sound like the end of the world when the creater of CouchDB quits his own project, but truth to be told, Damien Katz left CouchDB a long time ago. Couchbase moved past CouchDB long before they announced it. Basically when they started integrating Membase, though there are all kinds of notable contributions from Couchbase employees (e.g. Filipe and Jan).

Damien himself hasn't (actually) commited in over a year to CouchDB. Which makes his move no real surprise, just the way he decided to communicate surprised me. Especially since he said to have no regrets, I find the tone and statements in his blog post rather questionable. That is both from a personal and professional perspective.

I just attended a Couchbase event in Berlin last year where talks about CouchDB were given along with newer Couchbase developments. So personally, while I welcome clarity, all too sudden changes in strategy don't make me happy. If I was new to CouchDB and/or Couchbase, this would look like a headless chicken (excuse the image) and way too much drama to get into.

And on a professional level, Damien's posts invalidates the efforts of many people who both contribute and work with Apache CouchDB on a daily basis.

Then later today Damien shared this:

TIL, if you create an open source project, you should stick with it forever and ever. Family can live off unicorns and stardust. — Damien Katz

Real talk

First off, the most people in this discussion (excluding HN, of course ;-)) are actually active Open Source contributors one way or the other. Many of us have other projects (plural) besides CouchDB. It's not that we troll about something we have no idea about.

Secondly, it's not the fact that Damien left, it's how he left.

No one blames people for moving on: it happens all the time. I do it all the time — write code, push it out, move on.

If code is good enough it'll be picked up, if not, it'll rott on Github for forever. It happened to other projects and it happened to CouchDB. But why would anyone pronounce a project dead where he is not anymore invested in?

Anyway: I wish Damien good luck in the future.

So where is CouchDB at?

I wrote a blog post about the current state of CouchDB last year (2011) in early December.

A few things have changed since then:

Overall, I still see contributions from notable community members all around. Not sure if it's my own perception, but there has been more activity as of late. A new release of Apache CouchDB is just around the corner. Overall, good times for Apache CouchDB indeed.

The other thing that happened after my blog post is that Couchbase said in their 2011 review (which was published in late December) that it would officially step off Apache CouchDB and contribute documentation and OSX builds (aka CouchDBX or Couchbase Single Server) to the Apache CouchDB project.

This is great news and announcing that they will step off the turf is fine too since it clears up a couple misconceptions people may have about Apache CouchDB and the former Couchbase Single Server.

And all in, this makes Damien's blog post even more unnecessary and confusing to many users out there. Especially confusing for those who are not neck-deep in Apache CouchDB — and by that I mean: they are neither subscribed to a mailing list, take part on IRC, read the CouchDB planet or know any of the contributors directly.

In the end his blog post confuses people because it contains absolutely nothing but fear, uncertainty and doubt (FUD).

Outlook

I can't reveal too much because it's not my business to announce anything — let me just say that there are good times ahead for Apache CouchDB.

Fin

I'd like to get past all the drama and re-focus on what is important: CouchDB and our data. I don't care for the rest, I want to see exciting things from Apache CouchDB in 2012.

by Till Klampaeckel (till@php.net) at January 09, 2012 10:51 PM

January 06, 2012

Volker Mische

The future of GeoCouch and CouchDB

The CouchDB world is currently full of “The future of CouchDB” blog posts. It started with the blog post from Damien Katz the creator of CouchDB. Of course people were also concerned about the future of GeoCouch. No worries, it will be good.

The future of Apache CouchDB

The reactions were quite different. People who are not deeply involved with the CouchDB community think that this means the end of Apache CouchDB. My reaction was positive, I tweeted:

“It’s good to see the Damien is so open to [the] world”

The reason was, that for me it was pretty clear that it would happen, and I was just happy that Damien officially made the cut.

The reactions from CouchDB community members where pretty much what Till Klampäckel describes in his blog post. You could see it comming after Couchbase announced that they are not the CouchDB company and that their product won’t be Apache CouchDB compatible.

I agree with Till here, the way Damien wrote his blog post, isn’t the best imaginable. For outsiders, it really seems to be the end of Apache CouchDB, but it is not. For me it just shows, why foundations like the Apache Foundation are such a great idea. Even if the original creator leaves the project, it still lives on.

Apache CouchDB has a lot of contributers and the mailing lists and IRC channel is busy as always. That CouchDB has a future is also shown by the blog post from Cloudant. They will keep supporting Apache CouchDB.

The future of GeoCouch

After this quick recap what happened so far, it’s time to talk about the future of GeoCouch. As you may know, I work for Couchbase on the integration of spatial functionality into their product.

Currently the overlap between Apache CouchDB and the version Couchbase uses internally is still quite huge, but it will diverge more and more in the future. Thus it will get harder and harder to maintain a single version that supports Apache CouchDB and Couchbase.

The good news is, that GeoCouch is pretty much a data structure only. It's an R-tree that stores JSON documents. This can easily be used by CouchDB and Couchbase. Perhaps small wrappers will be needed, but those should be minimal.

The easiest way to understand how the future looks like is in a small illustration:

Illustration of GeoCouch and its relation to CouchDB and Couchbase

GeoCouch's core is the R-tree, it's the same code for CouchDB and Couchbase. On top of it there will be code that is specific to either CouchDB or Couchbase.

This means that the majority of the devlopment I do for Couchbase will also improve the GeoCouch you can use for CouchDB.

Conclusion

The future of all three, Apache CouchDB, Couchbase and GeoCouch looks bright.

by Volker Mische at January 06, 2012 08:27 PM

December 21, 2011

Till Klampäckel

Quo vadis, CouchDB?

Update, 2011-12-21: Couchbase posted their review of 2011 (the other day) — TL;DR: Couchbase Single Server (their Apache CouchDB distribution) is discontinued and its documentation (and its buildtools) will be contributed to Apache CouchDB.


When Ubuntu1 dropped CouchDB two weeks ago, there were a couple things which annoy (present tense) me a lot. Add to that the general echo from various media outlets blogs which pronounced CouchDB dead and a general misconception how this situation or CouchDB in general is dealt with.

Some people said I am caremad about CouchDB and that is probably true. Let me try to work through these things without offending more people.

Ubuntu1

What annoy[ed,s] me about this situation is that I wrote a chapter about Ubuntu1 in my CouchDB book. And while I realize that as soon as a book is published the information is outdated, I also want to say that I could have used the space for another project.

I talked to a couple of people about CouchDB at Ubuntu1 on IRC and no one made it sound like they are having huge or for that matter any issues.

Of course I neither work for Canonical or Couchbase. I haven't signed any NDAs etc. — but looking back a week or two my well-educated guess is that not even the people at Couchbase knew there were fundamental issues with CouchDB and Ubuntu1.

The NDA-part is of course an assumption: don't quote me on it.

Transparency

Scumbag Ubuntu1 drops CouchDB and doesn't say why. — myself on Twitter

First off: I'm not really sorry. I was abusing a meme and if you read my Twitter bio, you should not take things personal.

I also should have known better since it's not like I expect anything transparent from Canonical. (Just said it.)

When people are compelled to write a press release and put it out like that, they should expect a backlash. The reason why I reacted harsh is that Canonical didn't share any valuable information on why they discontinued using CouchDB except for: it doesn't scale.

And I'm not aware of anything concious to date.

Helpful criticism — how does it work?

Please take a look at the following email: https://lists.launchpad.net/u1db-discuss/msg00043.html

This email contains a lot of criticism. And it's all valid as well.

CouchDB feedback

Other examples:

These are great emails because they contain extremely valuable feedback.

Deal with it!

In my (humble) opinion, these kind of emails are exactly what is necessary in CouchDB-land, and many other open source projects: criticism and a little time to reflect on not so awesome features. And then moving on to make it better. If the feedback cycle doesn't happen, there's no development or evolution — just stagnation.

And in retrospect I wish more people would share their opinion on CouchDB and this situation more often. Since I'm personally invested in CouchDB, it's hard to say certain things. Honesty is sometimes brutal, but it's necessary.

In summary, a CouchDB user like Ubuntu1 (or Canonical) doesn't have the civic duty to give feedback, but to desert a project while pretending to be an Open Source vendor, and not talking to the community of the project or sharing your issues in public, that is extremely unhelpful.

Overall it strikes me that the only thing to date known about Canonical's collaboration with CouchDB is the support for OAuth in CouchDB. And most people don't even know about that (or wouldn't know how to use it). It worries me personally to not know the kind of problems Canonical ran into because they seem so messed up that they couldn't be discussed in public.

CouchDB doesn't scale

One thing I was able to extract is: CouchDB doesn't scale.

Thanks! But no thanks.

I wrote a book on CouchDB and I pretty much used it all, or at least looked at it very, very closely. I also get plenty of experience with CouchDB due to my job. Indeed, there are many situations where CouchDB doesn't scale or where it becomes extremely hard to make it scale. Situations where the user is better of putting data somewhere else.

Myself (and I'm assuming others) enjoy to learn the reasons why things break, so we can take this experience and use it going forward. If this doesn't happen we might as well all subscribe to the koolaid of a closed source vendor and purchase update subscriptions, install security packs and happily live ever after.

A patch to make CouchDB scale?

Another piece of information I gathered from the various emails written is that Canonical maintained CouchDB-specific patches for Ubuntu1. However, it's unknown what the purpose of these patches were. For example, if these patches made CouchDB scale (magically) for Ubuntu1 or if the patchset added a new feature.

What I'd really like to know is why these patches were not discussed in the open and why no one worked with the project on incorporating them into upstream. The upstream is the Apache CouchDB project.

This is another example of where communication went horribly wrong or just didn't happen.

A CouchDB company

I'm a little torn here and I don't want to offend anyone (further) especially since I know a couple Couchbase'rs or original CouchOne'rs (Hello to Jan, JChris and Mikeal) in person, but seriously: a lot of people realized that CouchOne stopped being The CouchDB company a long time ago.

This is not to say that the CouchDB project members who are employed by CouchOne/Couchbase are not dedicated to CouchDB. But if I take a look at the mobile strategy and the more or less recent integration of CouchDB with Membase/Memcache, I must notice that these strategies are far away from Apache CouchDB. Big data (whatever that means), to mobile and back.

The conclusion is that the majority of work done will not be merged into Apache CouchDB and this is one of the reasons why the Apache CouchDB project hasn't evolved much in a long time.

Not all changes can go upstream

I realize that when a company has a different strategy, not everything they do can be send upstream. After all, most if not all companies operate in a world where money is to be made and goals are to be met. Nothing wrong there.

But let's take a look at the one project which could have been dedicated to Apache CouchDB: the documentation project.

CouchOne hired an ex-MySQL'er to write really great documentation for CouchDB. The documentation made sense, it was up to date with releases, contained lots examples and what not. But it was never contributed to the open source project. The documentation is still online today, though it's now the documentation of the Couchbase Server HTTP API.

Wakey, wakey!

So in my opinion the biggest news is not that Canonical stopped using CouchDB and it's also not outrageous to think that there can be one CouchDB company. The biggest news is that Couchbase officially said: "It's not us!".

Having said that and also not knowing much about Canonical's setup and scale, I still fail to even remotely understand why they didn't work with Cloudant who spezialize in making CouchDB scale all along.

CouchDB and Evolution

Of course it is unfair to single them (Couchbase employees) out like that. For the record, there are pretty vivid projects such as GeoCouch which are also funded by Couchbase and while being devoted to the project, these guys also have to meet goals for their company.

Add to that, that other CouchDB contributors involved have not driven sustantial user-facing changes in Apache CouchDB either. CouchDB is still a very technical project with a lot of technical issues to solve. The upside to this situation is that while other NoSQL vendors add new buzzwords to each and every CHANGELOG, CouchDB is very conservative and stability driven. I appreciate that a lot.

User-facing changes on the other side are just as important for the health of a project. Subtle changes aside, but today's talks on for example querying CouchDB are extremely similar to those talks given a year or two ago. Whatever happens in this regard is not visible to users at all.

Take URL rewriting, virtualhosts and range queries as examples for features. I question:

  • the usefulness for 80% of the users
  • the rather questionable quality
  • the state of their documentation

Users need to have the ability to grasp what's on the roadmap for CouchDB. There needs to be a way for not so technical users to provide feedback which is actually incorporated into the project. All of these things aside from opening issues in a monster like Jira.

Since no one bothers currently, this is not going to happen soon.

Pretty candid stuff.

Marketing

In terms of marketing and with a lack of an official CouchDB company, the CouchDB project has taken a PostgreSQL-attitude in the last two years.

In a nutshell:

We don't give a damn if you don't realize that our database is better than this other database.

This is a little dangerous for the project itself because when I look at the cash other NoSQL vendors pour into marketing for their NoSQL database, I realized quickly that with the lack of support this project can go away pretty soon.

CouchDB being an Apache project doesn't save me or anyone either: clean intellectual property, deserted, for forever.

The various larger companies (let's say Cloudant and Meebo) are basically employed with their own forks with maybe too little reason to merge anything back to upstream yet. There are independent contributors Enki Multimedia who contribute to core but also sub projects like CouchApp.

And then, there's Couchbase which is trying to tie CouchDB behind Memcached. And from what I can tell pretty much abondens HTTP and other slower CouchDB principals in the process.

Is CouchDB alive and kicking?

You saw it coming: it depends!

Dear Jan, I'm still thinking about the email you wrote while I write my own blog entry. And honestly, that email and the general response raised more questions for myself and others than it answered.

I'd like to emphasize a difference I see (thanks, Lukas):

Core

Is the core of Apache CouchDB alive? — It's not dead.

  • Yes, because some companies drive a lot of stability into CouchDB.
  • No, because there's little or no innovation happening right now.

Ecosystem

There is a lot of innovation going on in CouchDB's ecosystem.

Most notable, the following projects come to mind:

  • BigCouch
  • Couchappspora
  • CouchDB-lucene
  • Doctrine ODM in PHP (and I'm sure there are similar projects in other languages)
  • ElasticSearch's river
  • erica
  • GeoCouch
  • Lounge (and lode)
  • various JavaScript libraries to connect CouchDB with CouchApps or node.js
  • various open data projects (like refuge.io)

Need more? Check out CouchDB in the wild which I think is more or less up to date.

Hate it or love it — there is plenty of innovating going on. And many (if not all) CouchDB committers are a part of it.

The innovation just doesn't happen in CouchDB's core.

Fin

My closing words are that I don't plan on migrating anywhere else. If anything, we have mostly migrated to BigCouch.

For Apache CouchDB, I think it's important that someone fills that void. That can be either a company, a BDFL or more engaging project leaders (plural). I think this is required so the project continues vividly.

Because I would really like to see the project survive.

by Till Klampaeckel (till@php.net) at December 21, 2011 04:15 PM

December 05, 2011

Klaus Trainer

couchapp-compress

If you've ever wondered whether there's a tool that will automatically compress JavaScript files when you're pushing your CouchApp somewhere, the answer is that there exists at least one: couchapp-compress.

Basically, couchapp-compress is a small Ruby script that wraps the couchapp command line tool. It compresses a CouchApp's JavaScript files and puts them altogether into one file, and temporarily changes the CouchApp so that instead of all the single uncompressed JavaScript files the compressed one is used. It pushes the CouchApp and then restores the previous state, so that again everything looks like before couchapp-compress was executed.

Check out the README for more details if you're curious and want to give it a try.

by Klaus at December 05, 2011 11:00 AM

November 11, 2011

Chris Strom

CouchDB and Backbone-relational

‹prev | My Chain | next›

I ended last night's exploration of Backbone-relational making quite a bit of headway, but also a bug. The bug resulted in a duplicate set of empty models in a has-many relationship.

In my case, my Backbone.js calendar application has appointments. Each appointment has many people that are invited ("invitees"). When viewing an appointment on the 17th, there should be 3 invitees. And indeed they show up, but so do three uninvited, no-name phantoms:


To figure this out, I add debugger statements to my calendar application. Lots of debugger statements.

Eventually, I add a debugger statement to my collection's parse() method:

    Invitees = Backbone.Collection.extend({
model: Models.Invitee,

url: function( models ) {
return '/invitees?' + ( models ? 'ids=' + _.pluck( models, 'id' ).join(',') : '' );
},
parse: function(response) {
debugger;
return _(response.rows).map(function(row) { return row.doc; });
}

});
There is nothing actually wrong with that parse statement. It converts the results of a CouchDB query into a list of attributes that can be used to build individual Invitee models. As can be seen from the parse() method, the CouchDB response contains a "rows" attribute with the actual invitee documents / attributes. The "rows" in that response contain several attributes. The only one that is needed to create an Invitee Backbone model is the "doc" attribute, which contains the entire JSON representation of a Person/Invitee:
{
"_id": "6bb06bde80925d1a058448ac4d006f6e",
"_rev": "3-231654f6914afe2e20eb57a41ec8497a",
"firstName": "Black",
"lastName": "Francis",
"type": "Person"
}
Checking one of the objects in the response in the debugger, I find that, indeed, the person is included in the list of results:


So far so good. This is exactly what I expect and my parse method should work fine. And it does. The problem is that the existing models in the collection look like:


That is, they look like:
attributes: Object
id: "6bb06bde80925d1a058448ac4d006f6e"
That is just the model placeholder until the real thing can be fetched from the server. But, when the response is fetched from CouchDB, it comes back as:
doc: Object
_id: "6bb06bde80925d1a058448ac4d006f6e"
_rev: "3-231654f6914afe2e20eb57a41ec8497a"
firstName: "Black"
lastName: "Francis"
type: "Person"
Do you see the problem there? I did not. Not for quite some time. The problem is that the existing placeholder has an "id" attribute, but the replacement from CouchDB has an "_id" attribute. CouchDB puts an underscore in front of the ID to indicate that it is meta data. Far be it for me to argue the wisdom of doing so, but it sure screws things up for me here.

The problem is that, when the model is fetched, it tries to replace the existing model, but there is no existing model. Although "id" and "_id" point to the same ID, they are two different attributes. And so Backbone simply adds the new model to the collection, retaining the placeholder. Hence the duplicates.

It took me a long time to track that down and to figure it out. As is usually the case, the solution is simple. In parse() I copy the "_id" attribute to "id":
    Invitees = Backbone.Collection.extend({
// ...
parse: function(response) {
return _(response.rows).map(function(row) {
var doc = row.doc;
doc['id'] = doc['_id'];
return doc;

});
}
});
And with that, I have no more phantom invitees:


That was a pain to track down. It is more a function of using CouchDB than anything else, but I do wish I could have figured that out quicker.


Day #200

by Chris Strom (noreply@blogger.com) at November 11, 2011 05:31 AM

November 10, 2011

Chris Strom

Replacing Homespun "Has Many" with Backbone-Relational

‹prev | My Chain | next›

I got started with backbone-relational last night. It took me a bit, but I ended up making some progress. Tonight, I hope to actually get it working in my Backbone.js calendar application.

The specific use case for which I have need of backbone-relational are the calendar appointments in my application. Calendar appointments have many people invited to them:


In JSON, the invitees attribute of this appointment is simply a list of IDs:
{
"_id": "6bb06bde80925d1a058448ac4d004fb9",
"_rev": "2-7fb2e6109fa93284c19696dc89753102",
"title": "Test #7",
"description": "asdf",
"startDate": "2011-11-17",
"invitees": [
"6bb06bde80925d1a058448ac4d0062d6",
"6bb06bde80925d1a058448ac4d006758",
"6bb06bde80925d1a058448ac4d006f6e"
]

}
And, to get that invitees attribute loading actual Invitee models, I had to add a relations attribute to my Appointment relational-model:
    Appointment = Backbone.RelationalModel.extend({
// ...
relations: [
{
type: Backbone.HasMany,
key: 'invitees',
relatedModel: 'Invitee',
collectionType: 'Invitees'
}
],
// ...
});
At this point, I can load the invitees in the Javascript console, but they are no longer showing up in appointment dialog. For that, I need to replace the loadInvitees method call from my pre-backbone-relational days with the fetchRelated() method from backbone-relational:
    Appointment = Backbone.RelationalModel.extend({
// ...
initialize: function(attributes) {
if (!this.id)
this.id = attributes['_id'];

this.fetchRelated("invitees");
// this.loadInvitees();
},
// ...
});
With that change, I have an invitees collection in the "invitees" attribute of my model. To make use of that, I pass said collection to the collection view (but only if people have been invited):
    var AppointmentEdit = new (Backbone.View.extend({
// ...
showInvitees: function() {
$('.invitees', this.el).remove();
$('#edit-dialog').append('<div class="invitees"></div>');

if (this.model.get("invitees").length == 0) return this;

var view = new Invitees({collection: this.model.get("invitees")});


$('.invitees').append(view.render().el);

return this;
}
}));
Amazingly, that works! It turns out that I was not that far away from success last night after all.

Although it does work—the invitees again show up in the appointment dialog—this works because of the somewhat hackish collection fetch() model that I wrote a few nights back. Specifically, the collection overrides fetch to individually retrieve the models specified by the list of IDs, manually triggering the "reset" event when complete.

Looking through the backbone-relational documentation, I see that it recommends mucking with the collection's URL so that it can request multiple IDs from the backend.

First up, my backend. Since I am using CouchDB, I can POST from my node.js backend to CouchDB, requesting all documents with the IDs POSTed in the request body:
app.get('/invitees', function(req, res){
var options = {
method: 'POST',
host: 'localhost',
port: 5984,
path: '/calendar/_all_docs?include_docs=true'
};

var couch_req = http.request(options, function(couch_response) {
console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

couch_response.pipe(res);
}).on('error', function(e) { /* ... */ });

var ids = req.param('ids').split(/,/);
couch_req.write(JSON.stringify({"keys":ids}));

couch_req.end();
});
Admittedly, that is somewhat exotic, but it works. Now, I need to be able to GET the /appointments resource with a query parameter of a comma separated list of strings.

As suggested by the backbone-relational documentation, I do that in the url() method of my collection (url can be either property or method in Backbone):
    Invitees = Backbone.Collection.extend({
model: Models.Invitee,

url: function( models ) {
return '/invitees' + ( models ? '?ids=' + _.pluck( models, 'id' ).join(',') : '' );
},

parse: function(response) {
return _(response.rows).map(function(row) { return row.doc;});
}
});
(I also have to parse() the results coming back from CouchDB to ensure that they map into an array of Model attributes).

And (again) amazing, this works. Except... that my collections have now doubled in size with the first bunch of elements being empty:


I do a little bit of digging, but am unable to see an obvious explanation for the phantom invitees. I will pick back up here tomorrow to solve this minor mystery (and hopefully conclude my exploration of backbone-relational).

I am still undecided on backbone-relational at this point. It has certainly guided me to a cleaner solution. But the reliance on global variable definitions for the models and collections remains worrisome. Perhaps another day's exploration will ease my concerns.


Day #199

by Chris Strom (noreply@blogger.com) at November 10, 2011 05:08 AM

October 09, 2011

Chris Strom

Filtering Backbone.js Collections

‹prev | My Chain | next›

One of the things lacking in my Backbone.js calendar application is the ability to switch to different months. Currently it simply shows the current month (and does not even display the month name):


So I do a little plain-old Javascript hacking to give me a global function named draw_calendar(), which takes an ISO 8601 string that can draw the basic calendar (without the Backbone application adding appointments):
  function draw_calendar(year_and_month) {
$('.year-and-month', 'h1').html(' (' + year_and_month + ') ');
reset_calendar();
add_dates_to_calendar(year_and_month);
};
The first line in there now displays the calendar date:

And, if I set the date to September 2011 in Chrome's Javascript console:
draw_calendar('2011-09');
Then I am treated to a blank September calendar:
Now I just need to teach my Backbone application to filter its Appointments collection. But, first I need the CouchDB backend to be able to support filtering appointments by month.

In the futon admin interface, I define a temporary view that lists documents by the year and month of the appointment's startDate:

This gives me the desired results—keys of the format YYYY-MM and values of the appointment documents:

So I save the view so that I can re-use it in my application:

Since CouchDB is of the web, I can query my new view with curl to verify that I can find all startDates in the month of 2011-10:
➜  calendar git:(pagination) curl http://localhost:5984/calendar/_design/appointments/_view/by_month\?key\='"2011-10"'
{"total_rows":18,"offset":12,"rows":[
{"id":"4a5600f5b2e36fc99d24fe9b8700037d",
"key":"2011-10",
"value":{
"_id":"4a5600f5b2e36fc99d24fe9b8700037d",
"_rev":"5-181418fffeacdc08b5fdda91525978b6",
"title":"Update dialog errors",
"description":"asdf1",
"startDate":"2011-10-05"}},
{"id":"8b5c80c0211068428272af4784000451",
"key":"2011-10",
"value":{
"_id":"8b5c80c0211068428272af4784000451",
"_rev":"1-87f5be84687eada2cd178c8dab7aa34c",
"title":"Finish Beta",
"description":"Book important",
"startDate":"2011-10-31"}},
{"id":"8b5c80c0211068428272af478400f7bb",
"key":"2011-10",
"value":{
"_id":"8b5c80c0211068428272af478400f7bb",
"_rev":"2-b1ce8c6c815a11f7b78b7db412988451",
"title":"Go to bed early",
"description":"asdf",
"startDate":"2011-10-22"}},
{"id":"8b5c80c0211068428272af478400fc93",
"key":"2011-10",
"value":{
"_id":"8b5c80c0211068428272af478400fc93",
"_rev":"1-b19359520433df557ad2fa0d56165f24",
"title":"Validations",
"description":"description should be required",
"startDate":"2011-10-03"}},
{"id":"956a5c19fd866a6a024bbb4c39002e3b",
"key":"2011-10",
"value":{
"_id":"956a5c19fd866a6a024bbb4c39002e3b",
"_rev":"2-0bbb8660f7f116cc813e0fd9093cec6a",
"title":"Has Description",
"description":"asdf",
"startDate":"2011-10-13"}},
{"id":"956a5c19fd866a6a024bbb4c390031a2",
"key":"2011-10",
"value":{
"_id":"956a5c19fd866a6a024bbb4c390031a2",
"_rev":"2-0568aded9df520a0c1c8bb2ee8961156",
"title":"In-dialog errors","description":"asdf",
"startDate":"2011-10-04"}}
]}
Yay!

I update my backend server, which is effectively middleware between the browser and CouchDB, to request this design document resource rather than the _all_doc resource it had been using.

Teaching the Apppointments Backbone collection how to interact with this new resource requires almost no changes at all. Only the parse() method needs to be updated to return appointments from the value attribute of my query:
      var Collections = (function() {
var Appointments = Backbone.Collection.extend({
model: Models.Appointment,
url: '/appointments',
parse: function(response) {
return _(response.rows).map(function(row) { return row.value ;});
}
});

return {Appointments: Appointments};
})();
To initialize the collection, I now need to pass the date query parameter to the Appointments URL. This is done by passing a data option to the collection's fetch() method (just like with jQuery's ajax() call):
// ...
// Initialize the app
var appointments = new Collections.Appointments();

new Views.Application({collection: appointments});

var today = new Date(),
year = today.getFullYear(),
month = today.getMonth() + 1,
year_and_month = year + '-' + pad(month);

appointments.fetch({data: {date: year_and_month}});
With that working, all that remains is the ability to switch months.

For tonight, I will just try to get this working manually in Chrome's Javascript console. In there I set the variable september to last month's date, then draw_calendar(september) (to get the calendar itself drawn correctly). Lastly, I tell the collection to load in September's appointments via a calendar.appointments.fetch({data: {date: september}}):
Based on the length of the Calendar appointments collections, it seems that 9 appointments hath September and, indeed, they actually show up on the calendar:

Best of all, I can do all of the common operations in my Backbone application like editing:
That is pretty darn cool.

I call it a night here. I will pick back up tomorrow trying to do all of this without resorting to Javascript console hacking.


Day #167

by Chris Strom (noreply@blogger.com) at October 09, 2011 04:01 AM

October 04, 2011

Chris Strom

Set CouchDB Attributes Before Backbone.js Transport

‹prev | My Chain | next›

I am still not quite done with my conversion to faye as the persistence layer for my calendar Backbone.js application. Most of the mistakes that I have found so far have been of the basic Backbone implementation variety, not the faye-as-persistence-layer variety. Well tonight, I think that I have a legitimate faye persistence problem.

When I update newly created models from my Backbone application, I am getting HTTP 400 responses from CouchDB:
Got response: 400 localhost:5984/calendar/undefined
{ error: 'bad_request', reason: 'Invalid rev format' }
Hrm... Checking the CouchDB logs, I see that I have a least got the PUT part of the update working:
[info] 127.0.0.1 - - 'PUT' /calendar/undefined 400
The PUT is correct, but the "undefined" is a clear indication that whatever I am updating does not have an "id" attribute.

My first instinct is that my overridden Backbone.sync() is not sending the JSON representation of the model, but rather it is sending the model itself:
    var faye = new Faye.Client('/faye');
Backbone.sync = function(method, model, options) {
faye.publish("/calendars/" + method, model);
}
So adding a toJSON() ought to fix the problem:
    var faye = new Faye.Client('/faye');
Backbone.sync = function(method, model, options) {
if (model.toJSON) model = model.toJSON();
faye.publish("/calendars/" + method, model);
}
Except it has no effect whatsoever. And really, it should not have an effect. If I am always sending a Backbone model object my overridden Backbone.sync() method, then something must already be calling toJSON() on my models. Since Backbone.sync() is sending directly to faye, it must be faye that is calling toJSON(). And in fact it is. Looking through the faye source, for each of the various transports, I see Faye.toJSON:
Faye.Transport.WebSocket = Faye.extend(Faye.Class(Faye.Transport, {
// ...
request: function(messages, timeout) {
this._timeout = this._timeout || timeout;
this._messages = this._messages || {};
Faye.each(messages, function(message) {
this._messages[message.id] = message;
}, this);
this.withSocket(function(socket) { socket.send(Faye.toJSON(messages)) });
},
// ...
});
Ah, so I am lucky that Faye has that toJSON() wrapper. That or faye is just a fabulous choice for a Backbone transport. At the very least, it is not the cause of my woes.

Anyhow, I still have my original problem that some update messages do not have the "_id" or "_rev" attributes needed on the server:
client.subscribe('/calendars/update', function(message) {
// HTTP request options
var options = {
method: 'PUT',
host: 'localhost',
port: 5984,
path: '/calendar/' + message._id,
headers: {
'content-type': 'application/json',
'if-match': message._rev
}
};

// ...
});
So I finally take the advice of Recipes with Backbone co-author and put mapping of "_id" and "_rev" attributes into the Backbone.sync() method:
    Backbone.sync = function(method, model, options) {
var message = model.toJSON();
if (!message._id && message.id) message._id = message.id
if (!message._rev && message.rev) message._rev = message.rev


faye.publish("/calendars/" + method, message);
}
I assign an intermediate message variable so that setting attributes does not affect the real model.

With that, I can now update my newly created model as many times as I like:
Best of all, there is nothing but beautiful HTTP 201's from CouchDB:
Got response: 201 localhost:5984/calendar/8b5c80c0211068428272af478400df1e
{ ok: true,
id: '8b5c80c0211068428272af478400df1e',
rev: '4-2929cf0e4933a4e32474145eb3e79f02' }
That is a fine stopping point for tonight. I think that I might have resolved all of my faye transport issues (and issues that I noticed after better testing with faye than in the original). I will do some more monkey testing tomorrow and then possibly move on to other areas to explore.


Day #151

by Chris Strom (noreply@blogger.com) at October 04, 2011 04:09 AM

October 02, 2011

Chris Strom

Deleting Backbone.js Records with Faye as the Persistence Layer

‹prev | My Chain | next›

To date, I have enjoyed decent success replacing the persistence layer in my Backbone.js calendar application. With only a few hiccups, I have replaced the normal REST-based persistence layer with faye pub-sub channels. The hope is that this will make it easier to respond to asynchronous change from other sources.

So far, I am able to fetch and populate a calendar appointment collection. I can also create new appointments. Now, I really need to get deleting appointments working:
Thanks to replacing Backbone.sync(), in my Backbone application, I already have delete requests being sent to the /caledars/delete faye channel:
    var faye = new Faye.Client('/faye');
Backbone.sync = function(method, model, options) {
faye.publish("/calendars/" + method, model);
}
Thanks to simple client logging:
    _(['create', 'update', 'delete', 'read', 'changes']).each(function(method) {
faye.subscribe('/calendars/' + method, function(message) {
console.log('[/calendars/' + method + ']');
console.log(message);
});
});
...I can already see that, indeed, deletes are being published as expected:
To actually delete things from my CouchDB backend, I have to subscribe to the /calendars/delete faye channel in my express.js server. I already have this working for adding appointments, so I can adapt the overall structure of the /calendars/add listener for /calendars/delete:
client.subscribe('/calendars/delete', function(message) {
// HTTP request options
var options = {...};

// The request object
var req = http.request(options, function(response) {...});

// Rudimentary connection error handling
req.on('error', function(e) {...});

// Send the request
req.end();
});
The HTTP options for the DELETE request are standard node.js HTTP request parameters:
  // HTTP request options
var options = {
method: 'DELETE',
host: 'localhost',
port: 5984,
path: '/calendar/' + message._id,
headers: {
'content-type': 'application/json',
'if-match': message._rev
}
};
Experience has taught me that I need to send the CouchDB revisions along with operations on existing records. The if-match HTTP header ought to work. The _rev attribute on the record/message sent to /calendars/delete holds the latest revision that the client holds. Similarly, I can delete the correct object from CouchDB by specifying the object ID in the path attribute—the ID coming from the _id attribute of the record/message.

That should be sufficient to delete the record from CouchDB. To tell my Backbone app to remove the deleted element from the UI, I need to send a message back on a separate faye channel. The convention that I have been following is to send requests to the server on a channel named after a CRUD operation and to send responses back on a channel named after the Backbone collection method to be used. In this case, I want to send back the deleted record on the /calendars/remove channel.

To achieve this, I do the normal node.js thing of passing an accumulator callback to http.request(). This accumulator callback accumulates chunks of the reply into a local data variable. When all of the data has been received, the response is parsed as JSON (of course it's JSON—this is a CouchDB data store), and the JSON object is sent back to the client:
  var req = http.request(options, function(response) {
console.log("Got response: %s %s:%d%s", response.statusCode, options.host, options.port, options.path);

// Accumulate the response and publish when done
var data = '';
response.on('data', function(chunk) { data += chunk; });
response.on('end', function() {
var couch_response = JSON.parse(data);

console.log(couch_response)

client.publish('/calendars/remove', couch_response);
});
});
Before hooking a Backbone subscription to this /calendars/remove channel, I supply a simple logging tracer bullet:
    faye.subscribe('/calendars/remove', function(message) {
console.log('[/calendars/remove]');
console.log(message);
});
So, if I have done this correctly, clicking the delete icon in the calendar UI should send a message on the /calendars/delete channel (which we have already seen working above). The faye subscription on the server should be able to use this to remove the object from the CouchDB database. Finally, this should result in another message being broadcast on the /calendars/remove channel. This /calendars/remove message should be logged in the browser. So let's see what actually happens...
Holy cow! That actually worked.

If I reload the web page, I see one less fake appointment on my calendar. Of course, I would like to have the appointment removed immediately. Could it be as simple as sending that JSON message as an argument to the remove() method of the collection?
    faye.subscribe('/calendars/remove', function(message) {
console.log('[/calendars/remove]');
console.log(message);

calendar.appointments.remove(message);
});
Well, no. It is not that simple. That has no effect on the page. But wait...

After digging through the Backbone code a bit, it ought to work. The remove() method looks up models to be deleted via get():
    _remove : function(model, options) {
options || (options = {});
model = this.getByCid(model) || this.get(model);
// ...
}
The get() method looks up models by the id attribute:
    // Get a model from the set by id.
get : function(id) {
if (id == null) return null;
return this._byId[id.id != null ? id.id : id];
},
The id attribute was set on the message/record received on the /calendars/remove channel:
So why is the UI not being updated by removing the appointment from the calendar?

After a little more digging, I realize that it is being removed from the collection, but the necessary events are not being generated to trigger the Backbone view to remove itself. The events that the view currently subscribes to are:
        var Appointment = Backbone.View.extend({
initialize: function(options) {
// ....
options.model.bind('destroy', this.remove, this);
options.model.bind('error', this.deleteError, this);
options.model.bind('change', this.render, this);
},
// ...
});
It turns out that the destroy event is only emitted if the default Backbone.sync is in place. If the XHR DELETE is successful, a success callback is fired that emits destroy. Since I have replaced Backbone.sync, that event never fires.

What does fire is the remove event. So all I need to change in order to make this work is to replace destroy with remove:
        var Appointment = Backbone.View.extend({
initialize: function(options) {
// ....
options.model.bind('remove', this.remove, this);
options.model.bind('error', this.deleteError, this);
options.model.bind('change', this.render, this);
},
// ...
});
And it works! I can now remove all of those fake appointments:
Ah. Much better.

It took a bit of detective work, but, in the end, not much really needed to change.

I really need to cut back on my chain posts so that I can focus on writing Recipes with Backbone. So I sincerely hope that lessons learned tonight will make updates easier tomorrow. Fingers crossed.


Day #147

by Chris Strom (noreply@blogger.com) at October 02, 2011 09:19 PM