Older Projects

Various projects at Aledade

I really enjoyed my work at Aledade. I made a lot of friends and enjoyed working together to improve healthcare.

My work at Aledade was concentrated in 4 areas.

Data ingestion REST API to support ETL

A core part of Aledade's approach is to use data to drive better value-based care. They interface with a wide variety of data sources, from government data to hospital data feeds to various systems in doctors' offices. They wanted a unified interface to provide to extraction software to transform and load data into Aledade's PostgreSQL database.

I built a Flask-based REST server to accept raw data, save it in load tables and then process it into the regular database using various business rules. A simple rule engine based on zope.event was used to decompose and manage business logic.

UI/Application development

Aledade provides an application to medical professionals to help them provide value-based care. I added and updated various application features. The application stack consisted of PostgreSQL, SQLAlchemy, and Flask.

This was the first project on which I'd used an ORM. Aledade's database schema was complex. I found SQLAlchemy to be more of a hindrance than a help. I often needed to perform queries that were difficult (often requiring undocumented features gleaned from SQLAlchemy source) or impossible to express in SQLAlchemy. Even when I could come up with an SQLAlchemy incantation, I was often left with something more complicated than SQL and in a non-standard language. There were also issues that were a lot harder to approach because I had to work out what SQLAlchemy was automating.

I got to do a lot of AngularJS (1) work on this project. At first I strugged to come up with a abstraction approach that worked. Eventually, I came to understand that directives were the most useful UI abstraction mechanism, especially when the ngModel framework was used correctly. At least at the time, none of this was easily gleaned from documentation. I enjoy UI development, and once I got the right development patterns working, this was a lot of fun.

Data science support

When I joined Aledade, the data science team was mainly using SAS. This made it difficult to share data and results between the data and development teams. It was also very expensive.

We decided to try to move the data team to Python-based tools. I spent a lot of time helping the data team learn Python and see how they could convert their SAS programs to Python, especially leveraging Pandas and Jupyter notebook. An interesting aspect of this was that getting the necessary software stack working on individual computers was impractical. We ended up setting up a Jupyter server on a very large EC2 instance. In addition to making the software a lot easier to manage, users had a lot more access to compute resources, especially memory, making larger problems amenable to attack with Pandas.

The data team's expertise was mainly with health-data domain knowledge and statistics. We decided to have developers help them with data wrangling. Initially, this was me. I enjoy data wrangling. I used to do a lot of it when I was at the USGS. To my amazement, I found this easier to do mostly in PostgreSQL than in Python/Pandas.

Eventually, I integrated the data-manipulation code into the data ingestion servers as rules that fired as new data came in.

Automated testing and process improvements

One of the first things I did when I started was to write automated tests, as there were none. This was initially in the ingestion service, which was a new project and therefore relatively easy to incorporate testing into.

Automated testing was challenging because the data was very complex and sample data was hard to come by. The database schema was large and impractical to build on test startup. There was also often a lot of reference data that was also impractical to load on test startup.

I ended up leveraging Docker. I created a PostgreSQL docker image that kept it's data internally, rather than using external volumes, and populated the image with our schema, reference data and sample data. For many of our tests, we could create a container from this image, and use the container throughout the test run, using transaction tricks to reset the data for each test.

Later, we started testing the application and I set up machinery to run Selenium tests. At Zope Corporation, I'd written zc.wsgidriver to make it easier to Selenium test Python web applications. The main idea of zc.wsgidriver is to run the Python test script and the Python server in the same process. This makes controlling the server much easier. For example, you don't need to use Selenium to drive the UI to do test setup. Instead, you can just manipulate data directly. Similarly, it's easier to make test assertions about data.

A complication added by the selenium tests is that the interactions with the database became multi-threaded and the transaction tricks I used to roll back changes made by tests no longer worked. I ended up creating a container per test for these tests. This was a little heavy, but given how long selenium tests run, the (not too high) overhead of creating a Docker container for each test wasn't significant.

Having the database controlled by the tests led me to a model of mostly developing in the test runner. Having the test automate data setup was often easier than doing in manually as part of development.

Our database schema evolved rapidly and we needed to keep multiple database instances:

  • Production database
  • Staging database
  • Test/development database in the Docker image.

in sync.

I instituted using Alembic to manage our schema changes. Alembic was designed to automate schema evolution expressed as SQLAlchamy models, but most of our schema wasn't implemented that way. We ended up using Alembic at a low-level, using raw SQL execution almost exclusively. I chose Alembic because I didn't want to reinvent the wheel, but I think something much simpler would have served us better.

A major development and testing challenge was sample data. We couldn't use extracts of real data because most of it was Personal health information and had to be closely guarded. I eventually created an anonymizing framework using faker. This involved keeping a database of previous transformations, so that we could re-anonymize the same data as our schemas evolved without changing data that might already be used in tests.

I also instituted regular code reviews using pull requests both for regular development and for data science activities. I'm a big fan of code reviews, not mainly because it improves quality, but because it fosters knowledge exchange.

Automated operations at Zope Corporation

At Zope Corporations, we hosted the applications we built. Our operations were a core part of our business. We hosted content-management systems and web sites for small newspapers, as well special applications like Reacht.

I began managing the operations team a few years before the company went out of business. Over that time, I gradually increased automation. We also migrated from running on our own servers in 2 data centers, to running all of our services in Amazon Web Services (AWS). When I started, we had 4 dedicated system administrators. At the end we had zero, because there was nothing for them to do.

Some core initiatives:

  • Monitoring is part of the software. Developers, not system administrators, wrote monitors for their applications. Culturally, monitoring was in the same class as testing. It's just another part of the development effort. (We didn't have dedicated testers either.)
  • Migration to AWS
  • Automated service modeling, deployment and service discovery using ZooKeeper,. Deploying new services or service updates involved checking in changes to a high-level tree based model. Committed changes are then deployed automatically.
  • A simple monitoring framework that's compatible with our deployment philosophy,. We replaced an in-house monitoring system with CIMAA. Existing monitoring systems were either too centralized, too complex, or not powerful enough for our needs. Our approach to application development and deployment was to keep applications self-contained and have configuration changes localized on machines where applications are deployed. Monitoring systems with centralized configurations were especially problematic for us.
  • Docker. Replacing our existing RPM deployments with Docker-based would have been our next initiative. We deployed a few applications with Docker, but would have deployed nearly all services with Docker eventually.

Coordinating independent monitoring agents (CIMAA)

We were using an in-house system that had an overly complex implementation and monitoring API. I stuck with it longer than I should have mainly because it fit our deployment philosophy well. Configuration was distributed to individual hosts and installed as part of application deployment.

At the end of 2014, I decided to replace the in-house system, but was unhappy with the existing solutions I found. Many either had centralized configurations or were simple and inflexible. Some required message busses. Most systems that monitor clusters of machines have a central master that must be kept available and communicated with.

We decided to pursue a much simpler architecture that relied on existing services and frameworks as much as possible with:

  • Distributed configuration
  • No central master
  • No message bus other than a database that was managed by AWS, not us.

Our goal was to have as little to manage as possible.

The result was Coordinating independent monitoring agents.

Features:

  • Use Nagios plugins for checks and collecting metrics. Because Nagios plugins are scripts, they're easy to write and debug and there are lots of existing plugins.

  • Local configurations

  • No master. Each monitored host runs an agent. Some or all agents run meta-monitors that check that all agents are running.

    We also supported using Sentry to report errors that occur in agents. Sentry is a fantastic service. All projects should use sentry (or something like it).

  • A pluggable database interface for storing faults. We recommended using a highly redundant database. We used DynamoDB in no small part because we did't have to manage it ourselves.

  • A pluggable alerting interface for notifying support staff. We used the excellent Pager Duty service.

  • A pluggable metrics API. Our agents were configured to send metrics to Kinesis. We had a system that extracted data from the Kinesis stream and wrote it to RRD files for graphing.

AWS

At Zope Corporation, We moved our operation from physical data centers to Amazon Web Services (AWS). We got a fairly boring (stable) stage where things operate pretty smoothly.

We used a variety of services:

  • CloudFormation
  • CloudFront
  • DynamoDB
  • EC2
  • Glacier
  • IAM
  • Kinesis
  • Route 53
  • S3
  • SNS
  • SQS
  • VPC

Our deployment strategy was based on:

Reacht

Reacht was a group messaging application for facilitating communication between group leaders and members. It evolved over time. Initially it was geared toward small groups (classes, teams, clubs) leaders communicating with group members, to media (radio, podcast) hosts communicating with their audiences.

Reacht was a mobile application for Android and iOS that communicated with a Python REST server. It used push technology both for messaging and for data synchronization.

As software development lead, I led the development of the overall application and application architecture. I was the lead developer for the Android app, which is written in Scala. I also contribute to development of the Python server.

Generational sets

Reacht put a premium on very rapid updates for people asking questions. I decided to use a synchronization model for this project. Most data that a user needs are synchronized to their devices. When there's new data, we sent users push notifications. When the app gets a notification, it fetches updates from the server. This allows us to provide extremely timely information without polling. The synchronization model used Generational Sets. Generational sets model data to be synchronized as a tree of sets to be synchronized together. On the server, we effectively push data to clients by updating their generational sets on the server and notifying them that there are updates to be pulled.

Android

Zope Corporation did native mobile application development for several years. I and a colleague did the Android development. The first application we started was written in Java. As Python developers, Java is hard to take with it's cluttered syntax and poor support for abstractions. I ported parts of the first application to Scala. Even without using many Scala idioms, the increase in readability was compelling.

Our second Android project was written in Scala. While the code was a lot more readable, especially as we started making heavier use of Scala idioms, build issues were a huge distraction. Following the original Android Development Toolkit, we initially used Eclipse and Ant. These tools were pretty primitive, and Eclipse became unstable as the size of our application increased. Lack of support for installing third party packages [1] in a sane way was also problematic. We worked around this by installing third party packages using zc.buildout, which is a "build" system for Python.

We eventually converted our build to use sbt and IntelliJ. We used the Android Plugin which had a lot of problems initially, but eventually got to a usable state thanks to the work of a student who adopted it for a student project.

I gave a talk about our experience using Scala to develop Android Apps through August 2013.

For our third and last project, we used the Android SDK Plugin for SBT which was a lot more stable.

I suspect part of the improvement in stability came from our decision not to use Android tests for this project. The Android test machinery provides functional testing for Android applications. It requires some special build mechanisms that put extra stress on the build process. The tests are essentially an extra Android app that has to be built and installed along with the application under test. This made test iteration slow and cumbersome.

Instead, we switched to Selendroid [2] and Webdriver. With this approach, tests run outside the device and communicate with a generic application on the device that sends event to the application under test. We wrote our tests in Python and even used the Python debugger to explore test failures and evolution interactively. Because our server was written in Python, we were able to run the server inside the test runner and manipulate and inspect server state as we test the client.

Storing ZODB blobs in S3

Zope corporation had some ZODB databases with multiple terabytes of blob data. We've moved these databases to Amazon Web Services (AWS). Storing the data in Elastic Block Storage (EBS) poses cost and reliability issues:

  • Need space for data and for growth.
  • Need multiple copies for redundancy.
  • Problems have been reported using very large EBS volumes, requiring combining many smaller volumes.

The AWS simple storage service (S3) is an attractive alternative:

  • Less expensive than EBS,
  • Only pay for what you use, so don't need to allocate (and later reallocate) space for growth.
  • Already redundant.

Balancing the benefits is the significant downside:

  • S3 access is slow, typically 10s or 100s of milliseconds.

The performance issues are especially problematic when committing database transactions. Transaction commits are a bottleneck, and it's critical to commit transactions as quickly as possible.

To mitigate the performance issues with S3, we created a caching blob server

The database server commits blobs as usual. The caching blob server watches the committed blob directory and, when blobs reach a configured age:

  • Copy blob to S3
  • Move blob to cache.

The blob server serves blobs via HTTP. It serves from the committed, directory, the cache, or from S3, as necessary.

Clients store blobs through the database server, as usual, but load blobs from the blob server.

New ZODB server components are needed to work with the blob server.

The most interesting aspect of this project was implementing the blob server in Scala:

I gave a talk about this project at the DC Scala meetup.

Content-aware Resume-based WSGI load balancer

The resume-based load balancer (RLB) is a WSGI load balancer that sends work to workers based on what they've worked on before.

For applications with working sets too large to fit in a single worker's cache, and where you can identify subsets of the working set based on request parameters, the load balancer can greatly improve cache performance by splitting the working set across different workers.

At Zope Corporation, we ran multiple RLBs, as internal load balancers, behind Nginx or Elastic Load Balancer. The RLBs discover workers using ZooKeeper. RLBs and workers communicate over long-running connections that multiplex requests and responses [3] using a protocol that uses Python's marshal serialization format. Workers keep track of their own resumes, which they periodically send to RLBs, so all RLBs share resume information.

In addition to providing content-aware load balancing, RLBs are simple and inexpensive alternatives to other internal load-balancer solutions. They're especially attractive due to their ability to discover workers dynamically with ZooKeeper.

In the future, I'd like to make a number of enhancements to the load balancer:

  • Have workers discover and connect to load balancers. This would make deploying workers easier with Docker. Deploying workers with Docker is challenging now because they don't know their real addresses and can't register their real addresses with ZooKeeper. Update: Docker's host-based network configuration would address this issue as well.

  • Provide a non-content-aware option.

    We sometimes used RLBs as internal load balancers when there's no need or benefit to split the working set. The behavior of the RLBs is a little counter productive in this case, as it tries too hard to send work to what it thinks are skilled workers. We worked around this by adjusting some balancing parameters, but it would be nice to have an option to use a simpler strategy that treats all requests the same.

  • Use a non-Python-specific serialization for communication between RLBs and workers. Protocol Buffers are an attractive option.

  • Have workers inform load balancers of their backlog status. While resume information is shared, information about worker backlogs isn't, making it more likely for workers to be overworked.

  • Rewrite in Rust, because Rust is fast and fun.

Tree-based application deployment

IMO, tools like Puppet and Chef are too low-level.

I wanted a deployment framework that provided:

  • A concise high-level model that lets you model the forest.
  • Multiple levels of abstraction for implementing the high-level model.

My stab at this was zkdeployment. With zkdeployment, you model system architecture at a high level with text trees. Nodes in trees represent system components, like applications, databases, virtual hosts, and so on.

Each service node has a type and version. Agents on each machine install (and uninstall) software based on the type and version. Application-provided scripts add and remove deployments (service-instance configurations) based on configuration information in the service nodes of the tree model.

Typically, service deployment scripts leverage buildout to perform deployment installation and uninstallation.

As of March 2015, we'd been using this system at Zope Corporation for two years and it worked well. We had hick-ups due to issues with yum, which we used to install RPMs and with problems due to bugs in the ZooKeeper C client interface and the Python C interface to it. The later has been addressed by zc.zk 2, which uses kazoo rather than the buggy C interfaces.

Docker burst on the scene since the work on zkdeployment. Docker looks very promising and would likely radically change how (or if) I would use zkdeployment. With Docker, most application deployment configuration could be done long before deployment to a production machine.

Service registration and discovery with ZooKeeper

When solutions are broken into multiple services, managing those services and the connections between them becomes a significant challenge. I created a ZooKeeper interface for registering and discovering services and for building service models.

I added support for ZooKeeper into various of our systems. To support this, I've built support for integrating ZooKeeper with ZEO, zope.server and buildout.

Zebrareach

Zebrareach was a startup that sought to enhance the relationships between small businesses and their customers. The first approach for this was to provide a loyalty system based on QR codes. When consumers purchased from a retailer, they'd show a QR code, either using a card or their smartphone, that was scanned by a business and points were awarded to the consumer. The consumer could later redeem their points, using their QR code, to earn free merchandise or discounts. This system worked well for businesses with long transaction times, like hairdressers and sit-down restaurants, and poorly for businesses with short transaction times, like take-out restaurants. The cost of sales to us was very high and we ultimately abandoned this product.

There were 2 other products under the Zebrareach brand:

  • A member-discounts product was offered to large institutions, including 2 hospitals and a community college to catalog discounts offered by various businesses. Ultimately, we were unable to monetize this.
  • An online ordering solution. It was thought that providing on-line ordering would allow restaurants to expand their businesses, but we found that this was mostly a service that restaurants needed to provide, but didn't have enough volume to be strategic for them or profitable for us.

Zebrareach's implementation included:

  • Mobile apps for iOS and Android for the loyalty program and member discounts. There were separate apps for businesses and consumers.
  • A REST API to support the mobile apps.
  • Web sites for member discounts and online ordering.

In addition to leading the overall effort, I helped on the REST server and websites and on the Android development.

The server was implemented in Python using Bobo. The Web client software used a combination of Dojo and jQuery.

The business Android app was written in Java. As Python developers, we found Java cumbersome and prototyped writing parts of the business app in Scala. We then used Scala to develop the consumer app.

Zope

Zope is an open-source community originally formed around an open-source content-management system and application platform.

I started working on what would become Zope when I joined Digital Creations in 1996. The first piece was the Python Object Publisher (bobo). This was initially an experiment to automatically publish functionality provided by Python modules as web sites and applications. It was one of the first Web frameworks for Python. The second piece was the Bobo Persistent Object System, which was the initial version of what would become ZODB. The third piece was the Document-Template Mark-up Language (DTML). We built a number of projects for customers based on these technologies.

In 1998, I led the creation of a general-purpose content-management/application-development system called "Principia". In late 1998, we open-sourced Principia and renamed it "Zope". The initial release was Zope 1. Zope 2 was released a few months later. The main difference between Zope 2 and Zope 1 was that it supported multiple application threads. Zope 2 is still being maintained and released today.

Zope was an application that provided a through-the-web development model. Users could create content objects, templates, and Python scripts and classes through-the-web. This was extremely empowering for non-developers.

Zope 2 had a number of unique features:

  • Object Oriented Database

    Zope 2 is aggressively object oriented and oriented around the tree of objects provided by its object-oriented database.

    The ZODB provides tasty cool-aid that, for better and worse, makes Zope 2 stand apart from web-application servers built on relational data models.

  • Acquisition

    Zope implemented Environmental Acquisition as way to share information in an object hierarchy. Zope's acquisition was implicit, in the sense that an object's attributes included attributes of it's ancestors in the containment hierarchy. Implicit acquisition is very similar to the sharing of data between JavaScript objects and their prototypes. Ultimately, users decided that implicit acquisition was too implicit.

    A significant challenge to implementing acquisition in Python 1 was that, given Python 1's reference-counting garbage collector, we couldn't store parent pointers. Objects didn't store references to their containers. Rather, when an object was accessed, a proxy was created that had references to the object and to its container and provided acquisition.

  • New-style types

    Zope 2 provided "new style types" several years before Python did. A meta type, ExtensionClass allowed built-in types, written in C, to be sub-classed in Python. This was used to create the Persistent base class used by ZODB. Implementing key facilities in C provided significant performance benefits.

  • ZClasses

    Zope 2 includes a form of Python class that's persistent. This is no mean feat. Python classes aren't designed to be updated at run time.

    Something I'm very proud of is that Adele Goldberg, one of the creators of Smalltalk used Zope. I imagine that Zope 2 felt a little bit like Smalltalk.

When we released Zope, we stopped making separate releases of Bobo, BoboPOS and DTML, because they were included in Zope, and because there weren't tools for packaging Python libraries at the time. This brought people used to developing with these libraries to Zope, where they were unsatisfied. Zope was an application, not a framework. The development tools provided by Zope were designed for non-technical users rather than for Python developers. We published APIs for Python developers to extend Zope, but this was a frustrating development environment for people who weren't otherwise interested in leveraging the Zope platform. Nevertheless, Zope brought a lot of people to Python in it's early growth years. The Eighth International Python Conference had a dedicated Zope track, which brought a third of the conference attendees.

In 2001, we began identifying projects to overcome Zope shortcomings:

Zope Component Architecture
Many Zope developers wanted a lot more extensibility in Zope. We decided that Zope Should have some sort of component architecture to make Zope easier to extend, including a mechanism to make extension (called "Zope Products") more extensible.
Zope Page Templates (ZPT)

Templating systems have generally been challenging to designers. We wanted to make it possible for designers and programmers to be able to work on the same templates without interfering with each other. Zope page templates allowed HTML mockups to be annotated through custom attributes so they remained valid HTML. The attributes were ignored and retained by tools like Dreamweaver.

ZPT was implemented in 2001 and later replaced DTML as the standard templating system for Zope.

ZSync
Through-the-web developers weren't able to use file-system tools, most notably version control systems (VCSs), or to easily transfer data between Zope databases. ZSync made it possible to check data out of and into Zope much as someone would with a VCS. Having checked data out of Zope, they could check into VCS. Later, the data could be checked out of VCS and checked into Zope.

In 2002, we began the Zope 3 project. This project had 2 high-level goals:

  • Provide a component-based model suitable for Python developers.
  • Replace Zope 2 with a new implementation benefiting from lessons learned from using Zope 2.

The Zope 3 project progressed in large part through a sequence of community "Sprints", modeled loosely on Scrum sprints. Sprints were one of the more successful products of the Zope 3 effort. They provided a fantastic vehicle for community building and collaboration. Sprints have been adopted by the Python community and a number of other projects.

The Zope 3 project had some significant technical achievements:

  • The Zope Component Architecture (ZCA) is a solid system for making Python applications extensible. It includes:

    • Adapters, for allowing disparate systems to interoperate.
    • Utilities, for defining pluggable system services, and
    • Events, for extending processing behavior.
  • Proxy-based protection scheme as part of a security architecture.

    Zope ran untrusted code. If you're going to let your users add code to your system, you need a way to limit what their code could access.

    Zope 2 has a security protection scheme based on manipulation of Python abstract syntax trees. This has a number of problems, one of which was that untrusted code could call trusted code and potentially escape restrictions (similar to calling set-UID scripts in Unix).

    Security proxies are much more secure. When untrusted code calls trusted code, objects passed to trusted code are proxies, as are results returned from trusted code. Similarly, results of accessing proxy methods and attributes are proxied. Accesses are described via white lists, so proxies fail safe. In fact, proxies are so effective at limiting access that their use only makes sense if you have an application that supports untrusted code, which most applications don't need to do.

  • Consistent handling of Unicode.

    Python 2's Unicode support was added on to the language at a relatively late date. A mistake in the implementation was to provide implicit conversion between (binary) strings and Unicode. While this was seen as a convenience, it proved to be a real headache as the implicit conversions often caused bugs that were hard to debug because symptoms often arose long after the problematic conversion.

    In Zope 3, we took the approach of treating all text as Unicode and systematically encoding only on system boundaries. This approach worked remarkably well. We had very few encoding or conversion related bugs as a result.

StructuredText

In 1996, I created StructuredText as a light weight text markup for generating various forms of documentation, especially HTML documents. It was inspired by Setext. Like Python, it used indentation to provide document structure.

StructuredText was widely used in the Python, and especially in the Zope community for a few years. The extensive use of indentation was eventually recognized as a mistake.

StructuredText was ultimately replaced by the superior ReStructuredText.

USGS (1984-1996)

At the USGS, my role evolved from a hydrologist writing software for hydrologic modeling and data analysis to a full-time software developer.

A common theme of this work was data analysis. Since leaving the USGS, my work has gone in different directions, but I still find data analysis to be a lot of fun. I hope I find an opportunity at some point to use some of the modern big-data analysis tools.

Plotwat

This was an application for plotting time series data directly from an operational (ISAM) database. The ability to visualize time series data was novel at the time. I needed to better understand model inputs and outputs but others used it to guide data collection activities. I believe this application is still in use at the USGS.

This work got me involved with the team that was building the next generation hydrologic database. Unfortunately, hardware resources were very limited. The data structures had been designed such that available disk resources would be insufficient. I created a simple compression scheme for the database that made its deployment possible.

P-Stat

When I joined the USGS, they were deploying Prime minicomputers for use in the individual offices. There was a contract for statistical software won my a small company, P-Stat, in Princeton New Jersey, which was just down the road from our office in Trenton.

The selection of P-Stat was controversial. The statisticians who'd written the contract had hoped that SAS would be selected, but SAS was too expensive.

I got involved in P-Stat support, trying to help people make the best use of it possible. In doing this, I created training and eventually made extensions to the package, including the addition of charts and integration with our databases. I was fortunate to get to know the authors of P-Stat, from which I learned much.

A strength of P-Stat, like SAS, is its powerful capabilities for manipulating data in preparation for analysis.

In my experience, data analysis is something like 90% data preparation and 10% analysis.

Perl

A colleague suggested I look at Perl. At the time, it was one of the most productive languages available and I taught it widely in the USGS. I think it's still widely used at the USGS, sadly.

Later, when I discovered Python, Perl was too entrenched for me to get many people to switch. Python could have had a huge positive impact.

Still, Perl was very good to me at the time.

Rand-rdb data manipulation language

In 1988, the USGS awarded a contract to Data General for workstations. Once again, SAS wasn't included. The contact included the Ingres database, which the IT people thought would be used for data manipulation. It was wholly inappropriate for the sort of ad hoc manipulations used when analyzing data.

There was a commercial "4GL" named /rdb. It provided powerful data manipulation using a language based on the Unix Shell. There were individual programs for performing various data-manipulation operations. Programs were combined using Unix pipelines.

Walt Habbs at Rand Corporation created an open-source implementation of rdb, implementing various operations as Perl scripts. This was a great proof of concept, but wasn't implemented very well. I forked this project, adding new operators, based on my experience with P-Stat, and improved existing operators.

Perl was good at manipulating text, but it wasn't appropriate for dealing with numeric data.

I looked for a object-oriented scripting language that could be used for user-defined data transformations in RDB. This search led me to discover Python. (Tom Christianson assured me that Perl would never support objects. :-))

Python enabled me to provide a far more robust data manipulation language for rdb. It also allowed me to implement new operators more cleanly.

Python has an excellent C extension API. I eventually used this to create RDB operators for moving data out of and back into Ingres, for integrating with our time-series databases. Later, using Python Motif bindings, I created an interactive application for building rdb pipelines and viewing analysis results.

Interactive charting, G2

The USGS has very demanding (or picky, depending on your point of view) requirements for producing charts for publication. It had a lot of difficulty finding COTS software that met it's needs. Given my experience producing charts for our hydrologic data and in P-Stat, i volunteered to work on an in-house system that met our specific requirements.

The application I built was a combination drawing and charting program. A user started with a blank canvas and drew charts onto the canvas, adding features like data, axes, legends and so on. Charts could be repositioned and resized by dragging with a mouse. Any number of charts could be positioned on a page at once.

The application was implemented with GNU Smalltalk. I'm a great fan of Smalltalk, and, at the time, this was the only version I could afford. Graphic were provided using a GKS implementation. GKS was a standard API for line graphics. The initial GUI was implemented using GKS, but was later converted to use Motif.

I enjoyed working with Motif. I created a binding for GNU Smalltalk and later used a very similar binding for Python. Motif was really well suited to dynamic languages like Smalltalk and Python, much more so than C++.

I added a context-sensitive help system written in Python using the Mosaic widget for displaying help information in HTML format. This was when Mosaic was the dominant web browser. The help system used the Tk "send protocol" to communicate with the main application.

Working on G2 gave me my first exposure to the power of user testing. The flexibility and power of the application, together with the initial blank canvas initially baffled users. I got complaints that people found the application literally unusable. I spent time watching people trying to use the system and realized that the blank canvas without any initial guidance was the stumbling block. The application was unlike applications they'd seen before and required a paradigm shift. I created a getting started guide that explained the basic concepts, including how draw charts on the canvas and customize them. This made a world of difference. People who'd reported the application to be unusable now found it the easiest they'd every used and could no longer remember the initial stumbling blocks.

Email

The USGS had developed an in-house Email package for Prime minicomputers. When we moved to Unix workstations. We decided to switch to off-the-shelf tools based on Email standards like SMTP and POP. This provided challenging due to some very specific requirements as well as the distributed nature of the user base. We used sendmail as the foundation. This required creating a highly customized sendmail configuration with a highly replicated USGS email-user database. It also entailed integrating Email user interfaces, including a text-screen interface, Plum, written in Perl, and a GUI, Exmh, written in Tcl/Tk.

[1]Later, we discovered Ivy, an Ant extension for downloading and installing packages, but by that time, we'd moved on to sbt.
[2]We chose Selendroid rather than Appium because it allows us to select controls by ID and because it supports older versions of Android.
[3]This is similar to the approach taken by Fast-CGI <http://www.fastcgi.com/drupal/>, except that in addition to requests and responses, workers send messages to load balancers about their resumes.