Paul Sonnentag

Github | Twitter | Resume

Programming through graphical interfaces

This is a series of prototypes I've developed under the name Pluto. My goal was to make the development of interactive applications more direct and concrete by replacing or augmenting static code with graphical interfaces.

Demo

I was inspired by sketch.systems, a small tool that helps you reason about the underlying states of complex applications with statecharts. I liked the idea of having a single graphical representation that gives you a high-level overview of your application. With sketch.systems you are limited to building click dummies. I wanted to use this paradigm and develop a tool for creating real applications.


                  A statechart for a search bar that has two high level states inactive and active.
                  The active state has three sub states empty, text entry and result
statechart of a search bar in sketch.systems

One thing that would be interesting to explore further is to allow nested states by introducing the concept of components that have an internal state.

statechart of todo item
example how the internal state of a todo item in a todo app could look like

Demo

A live coding environment for data transformations that always shows you the data on which you're operating. The results of list operations are displayed in an abbreviated form where one example item represents the whole list. This compact format gives the environment enough space to show intermediate results so you can trace back unexpected results step by step.

vanGoghArtworksByYear: @artworks where { @artist name = 'Van Gogh' } groupBy: { @yearOfInception }
data transformation that selects Van Gogh's artworks and groups them by year

The dots indicate the number of items in a list and can also be used to navigate between them. The appearance of the dots can also provide additional information. For example, we can see that the where operation filters the second and third items out. You can also collapse the graphical annotations, so it takes the same space as a purely textual representation.

data transform with collapsed visualization

Demo

Programming with natural language

What if we could use natural language to program?

"Do-it-yourself information software" is a concept I developed as part of my bachelor thesis with the hypothetical programming language Garlang. The programming language was based on the idea that you can define concepts in a naturalistic language in the form of a dictionary. Internally the statements are translated to Datalog predicates. For example, the definition of an employee could look like this:

Employee <plural employees>

An employee has a name that is a text.

An employee works a duration per week.

An employee is assigned to many shifts.

We can use existing definitions to define new concepts. For example, we can define the monthly work duration based on the work duration per week like this:

The worked hours of an employee per month is a duration (monthly work time):

  The employee works a duration (weekly work time) per week.

  The monthly work time: the weekly work time * 4

We can then use these definitions to ask questions about our knowledge model with searches.

search for 'all employees', result is a list of employee names

The result of a search can be visualized in different ways, depending on the type of the result. If we search for shifts, we can either display them as a list or a calendar.


                Two searches for 'all shifts'. The left search is visualized as a list of shifts.
                The right is visualized as a weekly calendar where the shifts are displayed as blocks in the calendar.

Multiple searches can be chained together to create ad-hoc tools to solve problems as they come up. Here we chain together a search for shifts that haven't been fully assigned yet with a search for employees that are available this week to figure out which employees we need to assign.


                The first search for 'all shifts that are not fully assigned' is displayed as a calendar.
                The second search for 'all employees that are available during the visible timeframe of the
                shift search' is displayed as a list

Programming in the physical world

At the beginning of 2018, I became obsessed with Bret Victor's latest project Dynamicland. From the outside, you could only get glimpses into the project through their posts on Twitter. It seemed like a magical place pointing in a radically different direction what computing could be.

Since the whole point of Dynamicland is that it's a physical place, you can't just download a piece of software and play with it. Therefore I decided to build my own version of Dynamicland at my university to play with these ideas. There was already an existing project called Paper Programs that implemented some aspects of Dynamicland. I've used that as a basis to build on top.

In September 2018, I also got the chance to visit Dynamicland for myself. There I created the query builder.

Sept 20, 2018

Last week, I stayed at Dynamicland, a unique experience unlike any other programming environment I've used before. Dynamicland is a community space in Oakland where the computer is the building. Programs are pieces of paper that have their source code printed on them together with a tracking code. Throughout the building are cameras that can track the pages. If a page is detected, its source code is executed. The pages can react to other pages, draw on themselves or the environment through projectors. If you want to change a program, you can grab a physical keyboard, point it at a piece of paper, and change its source code.

During my time there, I built a physical query system that allows you to ask questions like: "What things are part of our solar system?".

To demonstrate how the system works, I want to show you how to construct the question: What things are named after Obama? First, we use the search paper to find the object "Obama" by pointing a keyboard at it and typing "Obama". On the search paper, we can select an option with a query block paper. The query block now represents Obama.

You can flip the page of the search paper to search for relationships instead of objects. Here I'm searching for the relationship "named after". Once the query block is turned into a relationship it gets two whiskers added on both sides and can now be used to connect two objects.

With the two query blocks "Obama" and "named after" and a third empty query block we can now create the query: ??? named after Obama. The empty query block is used as a placeholder and can represent any object. This is useful when we run the query later.

If we want to get answers to a query we can use the finder paper to point it at an empty query block and it will show us a list of matching objects. In this case, we get all objects in the database which are named after Obama.

You can also create more complex queries by adding more relationships. Here I've added also the relationship "instance of" connected to the object "Mountain". In the result, we now get only objects which are mountains and named after Barack Obama.

If there are multiple empty query blocks in a query you can place a tracker marker on one of the results in the finder and it will overlay the corresponding value for the other empty block on the query itself.

Demo

An interactive book that let's you explore how sound works

Demo

Demo

Older projects

Demo

Robotopia is a browser-based programming environment for teaching introductory programming concepts with virtual robots.

Website | Source code

Blob demo

Golf demo

I developed Swip.js together with my friend Tim as part of a hackathon. It's a library to create multi-device applications where smartphones can be physically connected and know their position relative to one another. You can connect two devices with a pinch gesture over the edge of two devices. The touch gesture triggers a pinch event from both devices to the server. If two pinch events from different devices coincide, the server assumes that these two devices are connected at that position.

Source code

Logolino is an educational game teaches children the usage of articles in German. I've developed it for my mom who is a speech therapist.

Website | Source code

Qwery.me is a graphical editor for querying Wikidata.

Website | Source code

Imprint