Projects

See also: open-source project links.

Current projects and interests

Newt DB

Newt DB combines the power and ease of use of ZODB, an object-oriented database for Python, with a document-oriented database based on PostgreSQL JSON (jsonb) support.

Newt DB addresses the biggest drawbacks of using ZODB:

  • Access limited to Python

    Newt DB makes data available as JSON in PostgreSQL.

  • Searching

    There are "catalog" frameworks built on ZODB that can be very fast, leveraging the powerful caching abilities of ZODB. However, they can greatly increase the number of objects in a database, add a lot of extra processing and application complexity.

    With Newt DB, you can leverage PostgreSQL's powerful indexing and search capabilities. In most cases, this provides faster and more powerful searches and decreases the burden of search on the application.

ZODB

ZODB is an object-oriented database for Python. It provides highly transparent transactional object persistence with a pluggable storage back end.

Last year I rewrote the networking layer of ZEO to use asyncio and add SSL support. As part of this effort I simplified the ZEO and ZODB code quite a bit. It had (and still has, but less so) some over-done abstractions.

I've been wanting a much faster ZEO server for some time and I began that effort by creating a prototype ZEO server implemented using the Rust programming language.

I've also gotten more involved in the development of RelStorage. Development, and more importantly maintenance of RelStorage had languished, because the original developer lost interest. I arranged the hand over to Jason Madden and myself. Jason is the lead developer for RelStorage, but I've been able to make some contributions that have led to significant performance improvements, especially for PostgreSQL.

Upcoming projects include:

  • Support for easy asynchronous update of external indexes for ZEO, using a transaction pub/sub capability built on ZRS. I recently released something similar for RelStorage with PostgreSQL.
  • Better garbage collection.
  • More work on the Rust-based byteserver.
  • An implementation for JavaScript (?)

Buildout

Buildout is a build automation framework for Python. It's focus is on:

Assembly of applications
Assembles application components, invoking other builders, like make or distutils if necessary. The components may be Python libraries, configuration files, or applications.
Project-oriented, rather than system package management for Python
It automatically downloads and installs Python distributions into project, as opposed to system, locations as needed to satisfy Project requirements.
Automation and reproducibility
You can check a buildout out of a revision control system and build it it one or 2 commands and get a predictable result.

It provides an alternative package-installation model, using eggs and generated Python paths, that's similar to the way Java applications are assembled using jar files and generated class paths.

I recently added Python wheel support. Wheels are becoming the dominant distribution format for Python packages. Wheels also make binary distributions more practical, because they do a better job of capturing environment differences. This makes it much easier to install harder-to-build packages like numpy.

I recently embarked on a complete rewrite of the Buildout documentation. The original documentation was a collection of doctests. This was an experiment in reusing doctests as documentation. The result was pretty horrible documentation. The new documentation is written first as documentation. (The examples in the documentation are still tested, using manuel.) The new documentation also reflects years of changes, experience, and best practices.

2-tiered Kanban

For agile project management, I strongly prefer two-tiered Kanban to Scrum. A two-tiered Kanban board has a top-level tier with nested Kanban boards for one (typically) or more states.

A team will often work on a few development projects, which are typically new features or releases for products. The outer Kanban might have a card for each project (feature/release). When a project is in development, the inner Kanban shows individual development tasks and their states (waiting, doing, review). Having the outer Kanban has a number of advantages:

  • Groups related subtasks who's value isn't recognized until the whole is deployed and in users' hands.
  • Gives visibility to projects outside of development.
  • Makes sure we don't consider projects done until they're actually delivered to users.

I find the outer tier provides the project focus that might be provided by sprints in Scrum, but with a lot less ceremony and without arbitrary time boundaries.

I haven't found a tool that supported 2-tiered Kanban, although I haven't shopped lately. In updating this document, I've found a lot more pages and images on the topic, which is encouraging.

I implemented a two-tiered Kanban on top of Asana:

https://github.com/zc/asanakanban

I gave a talk on this in 2013.

The Asana APIs became very slow and we stopped using this. Since then, the Asana APIs have likely evolved. I doubt this Kanban works any more.

I began work on a replacement with a ZODB back end that uses generational sets:

https://github.com/zc/twotieredkanban

I haven't worked on this in a while and want to get back to it soon.

Rust and Scala

In 2016, I used Rust for an experimental ZODB server. It provided significant performance improvements. Rust has an interesting run-time-free approach to safe memory management and a functional programming style that reminded me of Scala.

I used Scala actively from 2013-2015. I care about code readability. I was looking for a high-level clean language that provides better performance than Python [1]. I settled into a comfortable familiarity with it for Android development.

I wrote a Blob server for ZODB in Scala that let us (Zope Corporation) store blobs in S3. It would be fun to do more server work in Scala.

I was somewhat skeptical of the claim that statically typed languages pay for themselves (for the hassles of static typing) by reducing the need for tests, however, on one of our project, we decided to skimp on tests to move more quickly and I was pleased with the low error rates we had despite skimping on tests.

I haven't used Scala for over a year and miss it. I'm getting rather (um) rusty.

Projects I'm actively maintaining or contributing to

Zope Replication Services

Zope Replication Services (ZRS) provides replication of ZODB ZEO servers. ZRS has a fairly simple architecture and is straightforward to use. It uses Twisted for network communication.

bobo

Bobo is a very lightweight WSGI application framework.

This is the second incarnation of bobo. The first was a object publishing framework for the web, developed in 1996 that eventually grew into Zope and that inspired many other Python Web frameworks. The new bobo returns to the simplicity of the original bobo, while benefiting from more than 15 years of web-development experience.

A while back, I gave a Talk on bobo at the NoVA Python Meetup.

When I released Bobo, I made a conscious decision to not promote it, as I was weary of promotion (and counter-promotion) of web frameworks. I still am, however this was a mistake. Flask came out a year later and became the main Python micro-framework. I used Flask for almost a year and still prefer bobo because bobo is lighter-weight and provides a more pythonic API. Bobo, sadly gets very little attention and use.

I'm contemplating a Flask plugin that gives Flask some of bobo's features.

[1]Python is fast enough most of the time, and generally scaling horizontally is better than scaling vertically, but there are times where raw speed is valuable, especially when you really need to scale vertically, or where you need to minimize latency. Python executes more slowly than many other languages, and it doesn't make good use of multiple cores.