During the recent week of NFT NYC, I had the pleasure of attending the TokenArt unconference - where ~150 artists, builders, collectors, and thought leaders came together to have focused discussions about generative art. It was a wonderful time, and I really enjoyed the conversations and getting to meet new people in the community.
I moderated a couple of these discussions, and did my best to take some notes on the highlights. With eight discussion groups for both AM and PM, it wasn't possible to attend each session - so I'm hopeful some of these notes will be of interest to others (and I'd love to see notes on the other sessions, if they're out there!). This blog post will focus on the discussion about Design Patterns in Generative Software - in this case meaning software containing an autonomous system used to generate art.
This post is not intended to be a comprehensive guide to all aspects of building a generative art program -- but rather a high-level overview of what the group discussed. This is an expansive topic that is ripe for ongoing discussion, and which I hope we can continue as a community in some form!
Topic and Motivation
Topic: Design Patterns in Generative Software
Prompt: How are generative programs designed? What patterns and tools can we use to effectively create generative art? (e.g. organizing Features, generating test outputs, measuring and optimizing performance, memoization, color palettes, etc.). What is the life cycle of developing a generative script? It seems there are many different techniques being experimented with by artists with diverse backgrounds -- let's discuss some of them! Maybe we can talk about some commonly-used algorithms (e.g. vector fields, automata, pixel sorting, etc.)
At a high level, the goal for this discussion was to talk about how generative art is made.
A Note on Attendees
The background of generative artists seems to run the gamut from technical to artistic, and this discussion group was no exception. I was pleased to see a number of experienced generative artists -- some with art backgrounds, and others with engineering backgrounds. Other attendees had created some digital art, and were looking for tips to make the leap into long-form generative art. Others were collectors, looking for a deeper understanding of what sorts of trials and techniques accompany the creation of the work they're collecting. This diverse background resulted in a bit of a higher-level, more abstract discussion that avoided getting into the weeds of specific code or algorithms.
Getting Started
A common question was how to make the leap from creating a single output to making a generative set. A few considerations we discussed are mentioned here.
It can be helpful to have at least one core technical concept to start -- for example, knowing "I want to achieve this visual effect" or "I want to use this type of an algorithm." Often, these go hand in hand.
One group member mentioned they like to start new projects by drawing/sketching to get the concept on paper before attempting to code it.
That said, it's not uncommon, at the start of a project, to not know exactly where you will end up. Many people in the group mentioned that development tends to be iterative -- deciding at each step how to proceed and what looks/works best. Development often consists of adding features to expand the solution space, then refinement to narrow it.
When developing a generative project, the best time to experiment is early in the development process -- with a variety of different techniques, new algorithms, variations on existing algorithms, new combinations of techniques, new languages or frameworks. It can be more difficult to experiment later during the development process. This can also be helpful to avoid "painting yourself into a corner" with one particular solution when there are a variety of interesting ones to explore.
The barrier to setup / launch seems fairly low on fxhash -- this is a great way to try out the full process of launching a generative art project. There are many aspects of the lifecycle that are hard to fully understand until fully experiencing them, so one of the best ways to learn is by doing! https://www.fxhash.xyz/doc/artist/guide-publish-generative-token
Anatomy of a Generative Program
Broadly speaking, it seems generative programs can be divided into a few main sections:
1 - Features/Data/Setup
This part of the program is responsible for taking the input data (typically a hash) and "exploding" it into a wide variety variables via a deterministic random number generator seeded from the input -- creating parameters that will determine the traits or features of the resulting output. This code can also be used to generate the features displayed for an output on sites like fxhash or Art Blocks.
This part of the program usually runs just once, is independent from later parts of the application, and is similar in structure for both still and animated works.
2 - Core Application Loop
In this stage, the application takes the features/parameters generated in step 1 as input, performs some calculations to turn them into some state representing the artwork (generation/state management), and then translates that state into an output in the target medium (rendering).
For still works, these steps may be run once, or some fixed number of times to layer the output. For animated works, they will run repeatedly in a "draw loop."
2a - Autonomous System State Management/Logic
This section of the application takes the input parameters (including the previous state, if it exists), and turns it into a state representing the art. For example, if the input is "two rectangles" this part of the code might say "there will be a FOO rectangle in the upper left and a BAR rectangle in the lower right."
For animations like fluid simulations, cellular automata, or other systems that change over time, this part of the application is typically what determines how the system will change, and makes those changes.
2b - Rendering
This part of the application translates the state of the system/art into the target medium. For most projects, this means rendering it to the canvas. Following the previous example, this part might be responsible for knowing how to translate "FOO rectangle in the upper left" into HTML5 fillStyle and fillRect commands that draw a blue rectangle at specific coordinates on the canvas.
For animations like fluid simulations or cellular automata, this part of the code doesn't know how the underlying systems work. Its only concern is how to draw fluid, cells, or other aspects of the system.
While in practice sections 2a/2b are often not as independent as 1/2, they certainly can be! You can imagine swapping in a different block of rendering code that outputs the same application state with a different appearance, or in a different medium (e.g., to an SVG file for a plotter).
Making Development Easier
We talked about some approaches that can help streamline the process of creating generative art.
Developing Tooling
While creating a generative art project can be accomplished in days, more likely it's a process that takes weeks or months. Developing tools and code intended for reuse can be extremely helpful for saving time and effort.
As projects are completed, you'll develop code that can be referenced and reused in future projects
Generic code/frameworks for managing things such as interactivity, frame rates, aspect ratios, DPI settings, full-screen viewing, etc. can be highly reusable
Having a web server / webpack setup that enables hot reloading when code changes can speed up development. fxhash provides a nice webpack boilerplate setup here that's a great place to start: https://github.com/fxhash/fxhash-webpack-boilerplate
Beyond tooling, it seems important to mention that developing familiarity with the platforms and services you'll be launching on will streamline the process of creating projects (e.g. smart contracts/Manifold, fxhash, Art Blocks). Developing relationships with other artists in the community and members of the platform team can be really helpful when troubleshooting issues during the launch process.
Scaffolding
Another concept we discussed is writing "scaffolding" -- code used to aid development, but discarded before launch. Sometimes there are algorithms or equations that are the output of other code, or code is needed to help discover an ideal parameter space. Some examples of scaffolding:
Code used to generate many test outputs for checking set diversity / behavior
Interactive code used by the dev to give parameter combinations a "thumbs up / thumbs down". The results of this can be stored in local storage and read back into the program on future iterations to help temporarily narrow the parameter space. When finished, the data can be taken from local storage and either used in the program, or examined to find ways to further constrain the parameter space in code.
Code used to build/train neural networks
Debugging/logging output. It's very helpful to have a log of input hashes to be able to reproduce great or problematic outputs and study them to see how/why they're working that way.
Optimization
We had a good discussion on optimization in generative art -- when to do it, and how to do it. Some takeaways:
It's harder to make detailed optimizations and assumptions about the final product earlier in the course of development. Optimization is an iterative process, but it becomes increasingly important to fine-tune towards the end of a project.
That said, it can be very helpful to make optimizations to core techniques early on (such as iterating state for a cellular automaton). Optimizations that reduce render time or increase animation speed can help speed up development time over the course of weeks or months, e.g. when generating 100 test outputs to compare. Important!
For on-chain generative art (such as on Art Blocks), space optimization is also an important consideration -- again, this is best done at the end of a project, with deduplication, removing code that isn't used in the final product, and of course, minification.
In general, considerations for optimization seem to be more significant for animated work than for still work. Of note, one exception is plotter art, where the number and arrangement of lines can play a significant role in how long the art takes to plot. There's a big difference between a plot that finishes in two hours, and one that finishes in two days! Optimization may still be a consideration for still art when, for example, generating 10000 sample outputs to check diversity/algorithm strength!
Which performance optimizations will be most helpful is largely dependent on the algorithm you're using. We discussed how this can be a bit of a "black magic" that involves experimentation and a good bit of digging. A great candidate for optimization is anything that happens inside of nested loops. Try to avoid using expensive functions like sin/cos/sqrt more times than needed. Make use of memoization where possible.
The strength of your computer can influence the type of art you're making. It can be helpful to work on a machine that's not on the cutting edge, to ensure a broad audience can still easily run your work -- or at least be sure you're taking this into account.
Including interactive features to change the level of detail ensures art is performant for users running on a variety of different devices -- and allows scaling the detail up years in the future, when hardware will be more powerful.
Recurring Themes
A number of concepts that seem important to generative art (or certain projects) came up during discussion.
Technique is a Value-Add, Good Technique is Not Enough
There was agreement between collectors and artists that, while impressive technical prowess in a project has value, ultimately strong technique alone is not enough to make great art. The primary consideration for most collectors/artists is in the beauty or meaning of a work; from there, impressive techniques "under the hood" can add to the impressiveness of a project, taking it to the next level.
The Medium matters to the Art/Process
There's a large difference between art that is viewed on a screen, projected with light, printed, plotted, sculpted, etc. The intended medium plays a role in determining what colors, shapes, and levels of detail are appropriate.
As an example, a thought that came up multiple times is that "making plotter art is all about thinking in terms of lines -- everything is lines, even filled shapes."
Art that Changes over Time
Art that changes over long periods of time isn't a new concept, but the permanent storage of on-chain art affords some interesting opportunities here. How can we make art that gradually changes with time?
Examples:
Textiles/Weaving
A number of artists have done work with weaving -- including multiple on Art Blocks. It seems the rules-based algorithms used for weaving are a good fit for generative art, as well!
Examples:
Perpetua: https://www.artblocks.io/project/277
Inspired by Nature
It's no surprise that many of the themes and algorithms in generative art, as in much of art, are inspired by nature. Algorithms like flocking, fluid simulation, fractals, cellular automata, and many others detailed below produce outputs mimicking the patterns of nature.
Conclusion
The group had a great discussion about high-level technical considerations that are important when developing generative art. We talked about the development processes, and some techniques and approaches that can be used for more effective development. We talked about a number of algorithms that are popular, and examples of their use.
It was clear from our discussion that we were only scratching the surface. An ongoing conversation doing deep dives into very specific or technical aspects of gen art coding seems like it would be both fruitful and in demand. There are a lot of really interesting ideas floating around out there that certain people in the community have deep expertise on, and a lot of opportunities for cross-pollination and recombination.
Thanks for reading my post! If you have any questions, suggestions for the post, or would just like to discuss generative art software -- please reach out to me on Twitter @ixnayokay.
Stay tuned for the next post, where I'll be writing about the highlights of another TokenArt discussion on Drop Mechanics -- exploring the aspects of a project release besides the art itself.
Check below for a list of algorithms, techniques, artists, and artworks that came up during our discussion.
Algorithms/Techniques
This section will list some of the algorithms that came up in the course of discussion.
2D Fluid Simulation / 2D Fluid Solver
A technique modeling the flow of fluids.
Examples:
Flocking / Boids
A technique modeling entities "flocking" together -- moving in a common direction while maintaining some distance from each other.
Examples:
Beauty in the Hurting: https://www.artblocks.io/project/178
Noise Displacement
Technique where points in a vector field are displaced from their position using noise.
Examples:
The Opera: https://www.artblocks.io/project/50
Voronoi Diagrams
Given a set of points in a plane, divides the planes into regions closer to each of those points than any others.
Examples:
Domain Warping
Warped vector fields
Examples:
Pigments: https://www.artblocks.io/project/129
Fractals
Self-similar geometric patterns, usually created by iterative processes, whose boundaries are higher in dimension than typical geometric shapes.
Examples:
Inspirals: https://www.artblocks.io/project/29
Cellular Automata
A technique where a space is divided into cells with a given state. Each cell affects the state of its neighbors at each time interval.
Examples:
mono no aware: https://www.artblocks.io/project/175
See also: Multi-neighborhood CA, continuous CA
Flow Fields
A technique where curved lines are drawn through vector fields.
Examples:
Fidenza: https://www.artblocks.io/project/78
Noise / Perlin Noise
A technique used to create pseudo-random noise, adding texture and increasing realism.
Examples:
Markov Chains
A stochastic model describing a sequence of possible events, where the probability of each event depends on the state attained after the previous event. (Cellular automata are related to this)
Examples:
Evolutionary Algorithm
A technique using reproduction, mutation, and survival of the fittest to evolve an algorithm or state over time.
Examples:
Zoologic: https://www.artblocks.io/project/287
Biometric Art / Biofeedback
Art made using biological signals. Displaying biological signals to give the viewer feedback into their own biological processes.
Examples:
https://www.fastcompany.com/90338484/how-rafael-lozano-hemmer-uses-your-biometric-data-to-create-art
Kubelka-Munk Equations
Equations modeling the appearance of paint films on a colored substrate. Takes into account light diffusion, absorption, and emission
Example:
Mark Lovejoy: https://artisticodyssey.com/artist-mark-lovejoy/
Other Mentions
Some other topics or platforms that came up during the course of discussion:
Julia programming language: https://julialang.org/
Processing: https://processing.org/
Raven Kwok (generative artist): https://ravenkwok.com/
"Poop Machine" (artwork): https://yourmileagemayvary.net/2019/02/18/this-museum-has-a-working-poo-machine/