Anselm Hook is a San Francisco Games Developer.

He focuses on immersive multiplayer games, and has strong personal interests in communication,

organizing knowledge and the philosophical underpinnings of knowledge.

This essay is the outcome of the (finally) emerging presence of the Internet,

many of the materials are based upon browsing the net,

and following up on subsequent references in paper literature.

Much of the effort focused on looking for the formalized language

which other people were already using to express like concepts.

It is written in a centered line approach

Because that seems the easiest way to communicate

The ideas which do not always fit together well.

This essay continues to undergo constant revision.

This is the third full revision and is dated April 11 1998.

ONE

I guess this essay is about how games

are going to be changing soon

in terms of their content

and the way they communicate the experience of play

primarily

in that more thematic and emotional cinematic experiences will become

easier to manufacture, due to being more codified – much like film camera theory

yet retain the interactive strength of immersive realities

And

As a side effect

that the games industry

will eat the greater entertainment industry

And

how it will on sooner than one might think basis

games will change us or at least our children

And

the interesting ways our children will change

as a result of the tools we’re inventing today

essentially that,

as barriers to communication using rich media types fade

our children will be able to use to use rich media grammers to communicate

and this will liberate them from

being merely passive receivers of rich media and speakers of only text or voice

ultimately suggesting that

that the pre-eminence of text and voice language will fade

in favor of universal languages

Such languages

could very well be based on

re-purposed media stolen from the sitcoms and movies of the twentieth century

yet probably

the topics will remain the same as time immemorial

and given all the these things

if they are in fact true

some shorter term practical implications and projects

could be injected into the marketplace

which could have significant impact

and benefit

I guess this essay is about how

this nascent clumsy ugly duckling

that we call the games industry

with its 90 percent failure rate

and its astoundingly inexperienced management

and infamous reputation as the overpaid sweatshop of the 90’s

and its litany of other sins I dare not dwell on

is going to surprise us all

and

"be king of the world!"

by

eating Hollywood

(wocka wocka!)

(much to the surprise of Hollywood)

and

(as an after dinner desert)

a lot of other conventional industries besides.

Such as oh,

Television

And all those evil minds

Who broadcast about their ideas

And who dominate

what the the common men and women sees and hears

Will be bounced out on their ears

By this adolescent.

This youthful hero.

This character out of greek legend

I guess this essay is also always about

Children.

How children, our children, will honor us by being better than us

How they will communicate

among themselves

in ways that will be radically different

from present day dialogue

(yet not unlike like present day media monologue).

How they will breach the rift of communication with a new babel fish.

How the same themes and ideas that

Are inextricably bound to our still slowly changing biology

Will be played out on a new stage,

Romance, love lost, love won,

power struggles, tales of paradise gained and lost,

Wisdoms and colloquialisms,

All shared in grammers richer than our imagining.

(All essays should ultimately be about children

since we are not really who we think we are

but ultimately are our children and are a continuum

even if we forget that when we are born)

I guess this essay also says that

game development in some aspects

deserves special recognition

That games development

(as a subset of computer science in this regard)

That it is an applied science

Concerned with building of functional synthetic models of whole systems

That it relies heavily on exploiting the emergent properties of the interactions of simple rules

(In order to get the job done in a reasonable amount of time)

And that the outcome of a well written application

Is one that meets the use cases it was modelled after

Without too many side effects

Or at least negative side effects

(Since we all know that it is the side effects which lead us to the next level of emergence)

but that in particular games require a kind of

total synthesis, and vision

that the systems modelled need to solve or hack through to some degree

deep and thorny problems of AI, philosophy,

and that a sublime apprehension of how to order and manage

the many different heterogenous objects,

their structural, behaviorial relationships to each other,

as well as to engineer for the final emergent effects.

games designers need to always be engineers today

(the art is not separate from the implementation)

one must figure out

how the game experience will be built based on a design

and how it will play on the musical instrument we call the human beast

how the particular psychological machinery which digests this mortal plane

will best resonate with the thousands of small notes played upon them.

What other trade requires the discipline and raw tenacity

Of a games developer? And what other trade is so unrespected?

Writing stories requires insight into the human condition,

But one can just say "mary climbed the fence".

One doesn’t have to discover the rules of gravity,

and compose entities who can climb fences and safely and on their feet

So ultimately writing – although being a space where game programmers would like to get to –

Is not yet the kind of space that game programmers work in,

And the extra labour game programmers go through

Does at least help them understand systems of behavior better.

And while the leverage is less in the sense that one cannot "say anything",

The leverage is greater in the sense that once one has said one thing well,

All things of a similar type are also said implicitly, and the whole

category of things to similar things that one could say are in a sense already said.

The great power of programming is the ability to automate.

One just can’t hide in an ivory tower and gesture at the truth of things,

At the end of the day, the sucker has to actually run.

This essay relies on things which it has to spell out,

It relies on many other many other congruent emerging trends as well,

Such as,

The continued emergence of a computational sea,

Which our children will swim in,

And

The radically improved power of

digital prothesthes,

And the radical reduction in their

cost and inaccessibility,

The essay is also needs to speak on other aspects,

The ideas here live situated in a context of my mind,

And I have to talk about how I think

And which things are most important to me

And which things I see as trends

It also has to say that

I see the concept of grammers as being important

But not all important

That patterns, mnemonic, rule transformations, mntemony, metaphor are useful

But are side-effects of more fundamental issues

Those primarily being relationships, how the mind composes families of related ideas

That prototypes are a stronger way of forming categories than

Abstract classes and instanced variations (or as Plato would say) imperfections thereof.

And how Aristotle was grieviously wrong in ramming class composition theory down our throats

And how Turing should have lived another five years

And had us building fallible but redundant parallel processors rather than von neumann bottlenecks.

This essay also has to say that

We really need a calculus of philosophy

That one can’t throw in John Searle style infinities

And use them as the substance of some kind of proof

Models usually fail if you try to do them at the use case level anyway.

That’s something a software programmer knows

But these pure theoreticians don’t seem to take into account.

Probably this essay will have some campfire story qualities

Where it revisits old stories we already know

In order to establish its pedigree and to establish familiarity

But I will try to keep such digressions to the right places.

Ultimately when speaking about where games are going

One is speaking about human understanding of the world

Because games require simulating the world

And deriving rules helps us build better models

Here we note that side-effects of better understanding

Will allow us more expressive range more easily

And that like the clever monkeys we are,

We’ll pick up these new tools as handily as the old,

But still - ultimately any essay on where our tools are going,

Is ultimately an essay on the problem of problems,

Which is to say being able to capture how it is we apprehend the world,

Being able to understand how we think,

And in so doing improving our communication ability

And our control and power over the world around us.

It is about understanding how the world ties together

And the grammers, ranging from simple to complex that we use to capture understanding,

Be that simple parameters of things - or actually capturing behaviors of things

( and ideally many of their resultant emergent effects )

Ultimately all discussions gravitate towards understanding,

And strong understanding is the same thing as strong AI

An extension of the tools, machines and levers we use to move the world.

TWO

Coin of the Realm

We live in the age of paucity

We who are so used to living in the heart of things

Sunlight was free for us

In our hands we held the bee who bumbled among the tulips

Cusping it for a moment before letting it free

Quartz wings, brushed yellow, whirling feelers

A sprig we could fashion and brush against the earth

And a spring would gush forth

Now our earth we turn to concrete

A concrete which does not reply to our fiercest blow.

Cars inch over the spaces where once we ran.

A sullen vibration from the street, the dust gathers in clumps

A graveyard of buildings, dead spaces, windows like eyeless sockets

The ghost of wolves flicker between each dry shadow

Seagulls drift overhead, baleful eyes over an alien landscape

A plastic bag, chocolate wrapper

Torn sweater, the remains of a chair, a muffler

Blown into mounds of sodden debris by winter rains

What was once freely given

we must now struggle for

Over and over we echo the clinging dream

Make dead the hollow pretty things, these shadows of what once was ours

A child dragging a dead kitten

Not understanding, not knowing what it has done

Slowly we capture and rebuild, remaking that lost memory

Our virtual playspaces, grinning skulls slowly fleshed again

A semblance of life stirs across this Walden pond

The cetacean bulk in its submarine depths

Swells across our cyber landscape

THREE

We know that games exploit emergence to save effort

Programmers create a substrate defining the way artifacts interact

And the game player can peturb the simulation once it is running

Today there are some emerging trends in games development

In part as a result of increased computational power at the programmers disposal

Simulations based on physical newtonian models of reality are becoming more prevalent

The rules of the virtual simulation follow the rules humans are already familiar with

And thus the way the virtual space is manipulated can be fairly intuitive

And also provide a better reading-in or tutorial on doing the same thing in the real world

Games that exploit emergent properties are becoming popular,

SimCity of course was an early game within which the play experience was resident

In manipulating objects to create conditions which created a desired outcome

And it was an interesting game in that the reward was in achieving voluntary goals

In that the simulations rules were constrained enough and rigid enough that

There was some chance of failing to achieve any solution, let alone a desired solution.

Warcraft and Starcraft today also exhibit a similar tendency

And in fact they are similar to the real world in that resource and scarcity management

In particular with relative equivilance between multiple forces contending for limited resources

Leads to similar tensions and conflicts as in the real world.

One can walk away from Starcraft with real lessons learned.

I don’t necessarily agree that games are about learning

Rather I suspect that games are a chance to escape the local minima

Of pathologically applied solutions

In my opinion games are enjoyable because biologically speaking we are rewarded for

not falling into ruts.

Some say that it is the learning, or the creativity that is directly rewarded,

but I say that discoveries made in new terrain – while clearly a reward

are not directly the reasons that we like to play

although discoveries made may be a crucial aspect of success and survival at living

We play because rational thought is a bad idea

And because reason is constrained by the past

Wheras real success requires doing rationally illogical actions.

Reason is reactive, derivative at best, not anticipatory and speculative.

Also, I think Daniel Bunton Berry has an important point when she said

‘players dial their own level of challenge’.

She implies that players seek the edge of their ability,

Trying to stay in a comfort zone between managability and total chaos,

And that players like to be able to dial down the experience when it gets a bit extreme,

Or dial up the experience when it is a bit boring.

In all activity

I suspect that as humans we most like to explore

the edge of frustration and apprehension

exercising the edges of our faculties

(Why we try to enforce black and white laws and rules over the land I have no idea!)

Video games are similar to other play experiences in this ability

The complexity of the experience

and the degree to which it is under the users control

is a key part of the experience.

Surfboarding, spelunking, writing, and playing board games all allow

Varying measures of dialability.

Some games choose to exercise the players at various different levels,

Some games allow the player to enter an alpha state

Where their higher level intellect becomes dormant,

And a not often exercized – at least in modern society - "closer to the surface" nervous system

is allowed to stretch its muscles.

To play on Starcrafts metaphors,

The conservatists would have us play out the mineral fields that we do know,

Or limit ourselves to the successes that we do know,

When in fact that way lies guaranteed failure as thouse resources play out,

And ultimately we are left at the end of the day hungry but without our next meal.

Often conservatists get local success mixed up with global success,

The local success is to mine resources, the global success is to win the game,

And the two aspirations are actually opposed,

A real sythesis is to mine the opponents resources first

And keep ones own resources as an ace in the hole.

and in that way win by depriving them.

Of course like iterations of prisoners dilemma,

All players quickly begin to climb the ladder of strategies,

Very quickly exploring the problem space

Games can provide real learning

(even though I don't consider play to be related to learning – that is just too glib an assertion)

The game we will consider are those which simulate a set of rules

Such games allow players to become fluent with the dynamics of the simulation

Computer simulations and real world issues are both similar at an abstract level

Playing a computer simulation can teach lessons about the way the real world works

Simulations can undergo resource scarcity or have ante up costs

Simulations can show trends that happen in a division of power

Simulations can show results of not paying up front costs of foresight

Simulations can show results of hedging or not hedging major energy expenditures

Simulations can show results of deploying resources prematurely or to late

These analogues can map to real learning even if the topics are not literally the same

Often game players I have watched seem to exhibit similar chronic behaviors in both the real and play domains,

deploying resources prematurely or failing to exercise foresight.

These players are probably using the same thinking agents in both domains

if this is true then it implies that knowledge in each domain is being fed to the same agents

and that later real world decisions made by those agents

may be mediated by new knowledge gained in game space.

Writing computer games can be an interesting exercise

Aside from modelling the problem space itself,

One has to deal with a rich panoply of grammers,

There are surface level property sheets that parametrically describe the properties of objects

This is the simplest way to modify the behavior of the game without actually programming

Then there are lightweight scripting grammers for choregraphing simple sequences of events

Often these lightweight grammers are used by non technical artisans to place objects or rules

That are not accounted for within the parametric properties of objects

And often the lightweight grammers hook into deeper behavioral aspects of the engine.

And finally there are the heavy languages such as C++ which lift the main body of the game up.

Doing things like physical modelling of newtonian physics,

Or quaternion interpolation of spline paths of avatar body motions

What is fairly evident is that there are different grammers at different levels,

And having a rich set of grammers exposes the game at appropriate levels

To people with appropriate skill and requirements.

FOUR

As humans we seem to have had two revolutions,

One where we mastered the mechanical,

And one where we are mastering the informational.

Both of course intertwine,

And each is similar in that we have specialized organs

Which digest, process and excrete the fiber of their nature.

Each of these organs is delivered stuff in sequence,

And our world seems to strongly reinforce the idea of

sequences of stuff being presented over time.

There is the strong underpinning of "sequences over time" in almost all human endeavor.

It does seem that our apprehension of music, of drama, of problems and their solutions,

Is delivered serially, and computed in serial chunks at some granular level.

Of course I don’t think we suffer from some kind of von neumman bottleneck,

But ultimately we can only be at few places at one time,

And it seems like the grand unfolding of events over time is usually sequencable.

Kazinsky (a painter) sought to paint logic onto his canvas -

to paint a language that would be parsed.

Such a story perhaps could be digested in a single gestalt,

Or perhaps the mind might meander over the canvas growing the model,

much as we do with other forms of information exchange.

In a sense everything is music, and there is genuine pleasure

in that which we derive from sequences

that meet certain patterns of resonance within the instrument that is ourselves

our hardwired neurons are happy to fire off little jolts of pleasure when

they are stroked properly,

and raking ones eyes across a wild rocky mountain vista is

enough to deliver the highest sense of pleasure as a lovers hands upon our body.

Ultimately one must be a sensualist perhaps to see how truly this resonates,

That the world at all levels resonates within us,

And that we’re conditioned to appreciate sequences in particular.

Given the nature of the time flow, it seems sensible that humans are adept at parsing sequences,

If so, then formulating and communicating with what we call "grammers" seems sensible,

after all, if most of the information about our world is in the world,

all we need to do to share an idea is to pluck the right strands in the listeners mind,

to evoke the right clues about the shared space that we are so intertwined with.

We don’t really need big memories or synthetic models, we just need enough to

get us into roughly the right configuration and where the real world is our heavy datastore.

AI scientists try so hard to model the world as an abstract thing.

This really scares me.

I don’t believe that we are pure computational engines

Which just happen to compute using organic juices

(instead of light waves or electricity).

I assert that form and function as intricately bound

(In the same way that AI proponents assert otherwise

which is to say without any substantive argument)

That there is no separation of mind and body

That we are not merely minds which happen to rest on biological underpinnings

Yes you could replace the brain with transistors

But the point is that the brain computes what it computes

Because it is a brain

And if it were transistors

It would begin to have other things it was concerned about computing

Which would make it a different thing than a brain

Albeit equally deft or daft at problem solving

There is no dispassionate world view.

The page layout of a poem affects the apprehension of a poem.

Isomorphic transformations are not possible because state is lost.

And translation makes new things, so translation is impossible.

A white horse is not a horse.

Experiential knowledge can be a good way to learn

In general learning cannot be divorced from time

We must all learn ways to deal with certain circumstances at the rate they occur

There are definately right and wrong actions to take in many situations

Assuming the goal of a situational outcome is to obtain a consciously chosen benefit

And that progression without control indicates that all travels and outcomes are acceptable

And that progression with inappropriate control simply indicates a poor understanding

Rote learning is brittle in that it fails to accomodate slight permutations

Rote learning is fast to deploy but there are too many facts to ever acquire them all

Rote learning can be helpful in that it can be a set of use cases to test a model with

But ultimately humans want to build best models of a situation

One way to build a best model is to peturb a live situation

Being in a situation is one way to have ready access to perturbing that situation

Most situations occupy a potential situation outcome space

This potential outcome space is initially unknown

By observing the outcomes of many different permutations we explore that space

By exploring the potential outcome space we can mature a model of the situation

A mature model of a situation lets us predict similar situations more accurately

For humans, first hand experiences can implicate our selves more thoroughly

Being in a situation versus watching may trigger biological survival instincts

It may make us more aware, alert, improves our memory and concentration

How we use our knowledge acquired through experiential or second hand to solve problems,

Humans like to boil issues down to black and white when speaking

But in fact we model real world issues as pure simulations

We speculate on all possible outcomes attempting to consider subtle implications

We have a rich social, oral and written histories to draw upon

We also have first hand experiences to draw upon

Our knowledge gives us insight into the potential outcomes of a situation

All of pop media, culture and history creates a diverse tapestry of different outcomes

We draw upon these time honed situational dramas as a source of possible outcomes

Stories and other second hand knowledge do not allow us to ask 'what if' with accuracy

And having many stories related over time does let us explore variations

I haven’t read evidence that we are genetically predisposed to speech

That we can communicate non verbally and non textually suggests different dispositions.

Certainly it does seem we are a social animal

And we seem to have an ability to map grammers to agents of thought

The social seems the primary sphere of the human condition

Mastering that domain seems a key to success since all other knowledge is so fickle,

One day we may be working in assembly, the next in C++.

We should make sure children are socialized as much as possible therefore.

There seem to be deep chasms in our human veneer,

That the polish we present on the surface hides a kind of uncivilized beast,

There seems to be a need to constantly talk to sooth the beast, to create like mindedness.

In a sense this may be the mythos that some people talk about,

And here there seems to be a connection between speech and feminism and mythology,

Which are studies in re-evaluations of the human condition.

FIVE

We are a social beast,

Our primary tensions and conflicts are with each other

Not with the natural world

To some degree our fangs and teeth are our fluency as communicators

Speech and text of course are primary communication substrates today and work well together,

Human dialogue seems to be a mechanism for manufacturing the same kinds of agents in a listeners brain.

As is noted later, speech is eminently easy to manufacture, and we’ve had millenia to polish it.

Some argue we are predisposed to speech, but I think we’re just predisposed to communication.

Marvin Minsky talks quite a bit about how this might work in his "Society of Mind",

Probably however the best thinking to date is late Wittegenstein,

and especially Elanor Roach on "Prototype Theory".

Sign language is interesting and similar to rich media types

In that it communicates using gestures which map closely to real world actions,

Being such a young language it has not completely dissolved its literal roots.

It is fully expressive, and often swifter than speech.

The human animal probably has genetic predispositions towards gestural languages

and satisfactory gestural languages most likely pre-date spoken languages.

An interesting aspect of sign language is that it seems easy to learn,

And that it lends itself towards a kind of bluntness that is rude

In English – as if honesty is hard to hide so one learns

to be simply have a thicker hide.

Kazinsky as mentioned earlier, is just one of a plenitude of artists

There has been a fairly consistent evolution of art realization over the last two millenia.

The focus seems to be on entering a perfect reflective state,

one which escapes definition itself, and thus judgement and criticicm,

but which describes the world perfectly.

In essence the perfect foil or perfect argument.

Along the journey there have been many great minds and thoughts,

As painters explored various forms of representation,

mirrored the general directions of philosophical inquiry

And generally continue to explore the boundaries

Of our apprehension and understanding of the world.

Most artistic works are concerted efforts to speak,

And they often seek to speak in ways that language fails them.

Although I suspect that one must always include the title and paragraph description

As part of the work itself, and not a separate thing from the work.

We could follow down the path of looking at all the famous artists,

And the evolution of their thinking,

But that is a discussion of its own.

Comics, the underdog industry,

Can become highly evocative and communicative once the barriers are removed.

It seems a playground for exploring the radical, the neuroses which plague us all,

As a form of communication in itself it seems especially rich and fertile ground.

English and the natural languages have many nice features that other mediums do not

(and which new mediums would have to overcome)

It is sublimely accessible

We have the mechanisms innately, such as the ability to speak.

Probably some degree of specialized mental faculties for speech.

(although I would argue that it is mostly a faculty for communication that we in fact have)

It already takes an entire childhood to become competent in just one language,

There is clearly competition for grammerspace in our daily dialogue,

they would have to overcome these,

new media would have to overcome all of this.

Natural languages also are sublimely polished by now,

They have utilities at all levels to specify the right degree of

abstractions of ideas

and the specificity to nail ideas down as needed.

They’ve clearly outgrown their rough roots.

Visual or Pictorial Languages

Would clearly have to travel far,

To develop the many degrees of appropriate scoping of natural languages.

Yet natural languages clearly have deficits as well,

There are significant transcription and ambiguity problems between different languages,

Researchers such Elanor Roach have shown that the idea that "red" exists is biological,

Yet the labels are random

And the differences between grammer rules are sufficiently complex,

And require such a complete human apprehension of both domains,

that it takes something that essentially mimics a human,

to readily translate between these two spheres.

And such an entity might not *want* to spend its time translating.

underlying concepts

instead it might want to make up some of its own

And natural languages are so optimized for low bandwidth transmission

And error redundant transmission,

That they sometimes take quite a while to describe something,

Even if the listener does know the language.

Business meetings occur every day

Where people announce the discovery of Canals on Mars.

Visual languages could help bridge the gap between differing peoples,

Even those within the same natural language groups,

Visually conveying universal themes

that otherwise are lost in the more subjective translations.

(although I suspect we’d have to become less provincial and less threatened by

images which invoke our sense of nausea or prudery)

Visual grammers can in a sense choose to be more objective on simple themes.

(although I think that cultural referents would always dominate)

Film Theory discusses essentially visual grammers

In fact I was surprised to discover that such a thing as Film Theory actually existed at first.

But logically it does make sense given human predispositions to formalize everything they touch.

Essentially Film Theory is a discussion of grammers utilized to convey information visually

It talks about how best one composes a visual grammer to provide compelling experiences.

The kinds of questions they seek answers to are,

(as purloined from a Usenet conversation from Ron Churgin)

How do films create meaning?

And as a primarily visual medium today -

how does film map or fail to map to linguistic metaphors?

(Some would argue for example that film like music doesn’t

best relate to linguistic metaphors – especially non-narrative film

such as avant-garde/experimental/mythopoetic/trance or structural film)

What effect does visual rhythm, the effects of colors, the physical sensation of light,

The use of repetition, the interplay of music and visuals

have on apprehension of film?

One intriguing question is what is the smallest unit of film, what is the

Morpheme or atomic building block that film is composed out of?

Can a film evoke a emotion or physical sensation purely?

Today we live in a world where we have a populace with a rich heritage of visual referents.

The multigenerational presence of television has filled our minds with rich imagry.

And we have become educated in the particular grammer of emotion and situation in film.

We’ve been taught close up emotional shot,

the caged suspense shot with poor visual acuity,

The opening setting shot.

The phrasing of these canned experiences today is built on formal grammers

which are taught in film schools.

The quality and believability are reinforced by rich characterizations

Capturing the prototypical human condition.

Characterizations using our cultural perspectives and biases for better or worse.

The audience is an instrument played to a greater or lesser degree by the skill of the movie maker.

Narrative movies are not just stories, they define an entire situational space, a "what if space",

That happens to be constrained to a certain route because of the limits of the medium.

But still provide a foundation for dialogue about that space, and about exploring that space.

Movies can transcend language barriers which reduce speech to just so much noise,

However today film is too difficult to generate on the fly,

And inappropriately scoped,

And requiring more of the listeners attention than simple english or text.

One would expect that these are some of the issues which would have to be improved

For us to begin to communicate using rich media types.

A visual language will ultimately require having a rich heritage of visual referents.

becoming fluent in the rich media types requires knowing the referents,

the colloqiualisms and slang.

(which are the larger body of most grammers and it seems define the language

much more so than the simple rules which string the grammer together).

Metonomy, where a part stands for the whole

is a means by which grammers can become rich and malleable,

where the general aspect and overtones of an image

can mean more than the specific instance of its invocation.

Being able to metonomically invoke visual situations by providing only cues

Seems to be a key part of how the broadcasters communicate today.

The sources of some of these colloqiualisms are already present,

(as suggested by the fact that many sitcoms today

relies heavily on self-reference for humor for example)

To find possible sources for

invocation of themes and concepts

one need only consider that which

we can see without nothing more than casting about

in our heads for a few minutes.

The endless parade

of bread and circuses

(serials and comedies),

"Three’s Company", "M.A.S.H.", "Fresh Prince of Bel Air", "Melrose Place",

"Babylon 5", "Star Trek", "The X Files", "Friends", "Cheers", "Abott and Costello",

"Lost in Space", "The Twilight Zone"

and movies,

"Titanic", "E.T.", "2001", "Sound of Music", "A better Tomorrow",

"Wizard of Oz", "Grapes of Wrath", "Clockwork Orange", "1984",

"Brave New World", "Live and let die", "The Terminator"

and the famous events

"One small step for a man", The Challenger disaster, "Oh the Humanity",

The sinking of the Titanic, Watergate,

Collapse of the Berlin Wall,

and

Styles, objects,

places and situations,

Fat Freddy and the Freak Brothers,

The Colliseum in Rome, The Eiffel Tower, Michelangeo’s David,

The Scream, Geigeresque art, Franzetta art, Abstract Expressionism,

the little girl on a dirt road in Vietnam running away from the U.S. napalm blast.

Japanese Tea Gardens

and

The infinity of other

signature and metonymic referents

The particular shape of the tibetian columns –

red with a broader peak than base, Corinthian Columns and Roman Columns.

Gaellic fonts, the mono-spaced computer font, Victorian Chairs, Indian Mandouts,

The formalized grammer itself of Southwest Coast Indian Art. Manja.

The constrained cursives of Eskimo Art. The shiva. Cyclops. Nessie.

So far we have spoken about games and their interactive power

and some of the reasons people play games

Grammers and how we compose sequences and the power of simple sequencing tools

And how grammers are used to form relations which bind directly to our needs as humans

And various flavors of grammers,

from programming to film theory, to music, to human language, to gestural languages

And on the human condition, the particular mechanics of humans,

And that each of these grammers seems to be able to create a response to varying degrees.

We also see that while we think we’re fairly simple beings

that we do live in a rich world with a multitude of grammers of varying scales

(of which we have varying degrees of expertise in).

And that probably we’re pretty good at formulating quick and dirty grammers to help digest our world.

That we rely on metonymy, metaphor, and vast families of relationships to let us generalize fairly quickly,

And that there is a very close binding between our grammers our processing thereof

and the real world we’re solving for – the observer is specialized to deal with our particular world)

Later we speak about how tools are becoming more accessible,

How the barriers to entry are going away

And we note that more people do share information using rich media types

We see that at some point there will be universal access to rich and powerful tools

That these tools will be extensions of our being

prothesthetics side effects of the computational sea that we will be living in.

Given such a sea

and given access to rich media types as easily as we access impoverished types today

it seems logical that we can’t help but communicate using these types

there are so many inducements

we’ll broadcast rich signals and receive them too

probably using machines to augment our broadcasting ability

and probably using our native wetware to receive

(insofar as that we remain human animals with human interests)

We’ve talked about feasibility, probability and likelihood,

Looking at problems with present media types

Now we need to talk about implications

Becoming a ‘man of letters’ in this new age

seems to require only reduced costs of access to the visual printing press.

it is fairly evident that we have an incredibly rich visual history

and it seems fairly evident that the mainstream media does heavily broadcast

to us, and by implication, if we had the means that we might broadcast right back.

Is

This the revenge of TV?

This idiot box that sedates the worlds largest homogenous population

Will ultimately become the seeds of a language

That could lead to powerful new means for discourse and dissent.

Is

This why only our children and not ourselves will speak thusly?

repurposing the rich visual heritage

that we of the franchise era have created

but ultimately reject and disrespect

in our crotchety old ways?

Is

this the same evolution as English?

Even English seems a bastard child of many parents,

A waking beast formed out of the scrap heaps of more beautiful things,

Taking references at the wrong scope,

Generalizing over specific references,

And slowly getting polished and smoothed

From its ignoble background

A grammer now growing word mushrooms

Out of places and people

Which are just so much compost

It seems people speak wherever they are best heard,

Using whatever patois, be it the devices of the legal system,

or the lash of a weapon, or the fang of an argument.

And people who speak without regards to the ammendments to freedom of speech

May endanger themselves, and thus therefore such ammendments may limit new speak.

For example the French stifle their language by having language police

(who protect one strange and quirky patois out of all the flavors which really exist)

How will our laws affect new grammers?

We know that images are powerful,

People seek to own them,

To limit the use of their Mickey Mouses,

Does this mean that the only people able to speak in visual tongues

Will be those who are on off-world colonies,

And who are presumably able to escape the lawsuits implicit in

Repurposing the McDonalds Golden Arch and such like "owned" referents,

(Even though those referents seem a legitimate part of the cultural mileu?)

It is clear all these forms can be used to play the human instrument

To evoke and compose particular strands of thought,

Like the beginnings of music, we are building more and more complex melodies

Farscical thoughts can be evoked with a bit of pie in the face humor,

The invasion of the coast of Normandy could suggest victory at great cost,

The romantic in us could woo our lovers with scenes from Casablanca,

Situations and places can be evoked directly by referring to them,

And can do so in a compelling way.

Yet regain generality by being associated with cues.

We see

That huge industries exist today to manufacture visual fictions,

The television industry is itself a goliath,

Theatre, from "Phantom of the Opera" to the productions at the Artud in the Mission District of San Francisco

all provide support and a lifestyle for the people who manufacture stories with rich media.

And we see

That Colleges and Universities

give degrees and provide fundamental skills

in staging theater, from lighting, to set design, to acting, to costume design

to voice to theoretical understanding of theater and entertainment.

We clearly live in a world where such incredible riches

are at our fingertips

And perhaps therefore we are a bit blasé

Thinking that we need be only consumers,

And not creators.

And of course it is a bit hard to be a creator

when the voice of the corporatists is so much louder

(And so very polished)

But that will swing again in our favor.

I have seen friends speak to each other

In private tongues

A patoi

Formed from years of friendship

Strumming well known mental chords

An intimacy of trust grown from being like minded

And of understandable behavior

Complementary, insightful,

mon frere

And

I have seen friends speak for hours purely with reference,

And derive great pleasure from walking each others trellises

Hopping from a band, to a performer, to another band

Whom they played with,

Or were in, Or redid a famous song of.

Or hopping through algorithms,

From one solution, to another,

To something purely theoretical and back again.

We do learn and we do speak

many tongues

and visual learning is just one more type.

We are clearly able and willing

To digest rich media types

And presuming sufficiently simple tools

We could also deliver rich media types.

But for lack of a visual typewriter

Our kingdoms languish,

For merely a mouth that speak video.

Or a ready staff of able servants

swift enough to compose a dramatic setting

for a small vignette

to convey the slightest tickling of our fancy

to expressively communicate

to another sentient being

the exact nuance

of exactly how we feel

about them

trodding on our foot.

Ouch!

Today many people do communicate using rich media types

These types just aren’t real-time yet

Because the tools are too slow

And the volume of imagery to convey is too rich

But that will change.

Centuries hence,

We could reach an age

Where we communicate in simulation,

Much as game programmers do today,

Sharing synthetic worlds,

Whereupon we float on our proverbial Titanic,

Allowing our listeners to peturb the stalwart captain and crew

Within the domains of the simulation

To see if together we could

find a way to avoid that iceberg

And solve the question posed.

Crunch!

A discussion could be as Leibniz dreamed,

Where observable fact was fairly agreed upon,

And the debate raged at the symbolic logic level,

Or as would be said today,

That the behaviors and relations of objects

was the subject of the debate.

And

Talk in such a time would be an applied exploration of rules

And the emergent effects of those rules

As modelled on a real simulation

And on the repercussions of those rules.

And

Centuries hence, a child learning to speak

Could be a very colorful creature indeed.

Their english only a legacy scripting glue

to conjoin graphical sequences if at all.

In the shorter term,

Work which we develop now

Which follows such a trend,

By allowing ordinary people to compose sequences of events,

And share them readily with friends,

And exploits the emerging internetworks,

Would presumably be a success

And

the degree to which

We permit mass audiences to "talk back"

And to sequence and choreograph their own rich media dialogues,

Is the degree to which such efforts would be successful.

Allowing a corporatist

To build a powerpoint demonstration

is a start,

Allowing a teenager

to use SimCity to play with the ordering and layout of a new land

is a start,

Or to capture Kim Stanley Robinsons Mars series in a user accessible simulation

is a start,

To provide Project Maya

Or SoftImage, or Lightwave or other tools,

Which provide sequencing and choreography of rich media types

is a start,

To provide physical simulations so that burdens of choeography are lightened,

And more of the behavior is emergent

is the next step.

Work such as this

Will become more and more accessible to consumers,

And will gain wider audience as more people use it to communicate with each other,

And slowly there will be a shift,

As more resources get poured into extending consumer frameworks,

Rather than doing specialized productions,

And the consumers will start to share rich media among themselves,

Creating their own Star Trek episodes,

And revisualizing the world in a way that

They are frustrated from doing so right now.

These visual stories that people will share

Will be mutable, being rule based, and potentially immersive,

And can start to explore issues that aren’t simply entertainment,

But recreations or studies of real problems,

And exploring the space and peturbing it,

Can have practical results –

Not to say that some dilemma of a young adult,

Rendered as a visual interactive narrative,

Isn’t as equally important and worthy of solving.

Interactive rule based narratives can potentially become

the primary means of serious communication,

An inversion of the way text today is the domain of serious communication,

and television the domain of the idiot.

Today we are like children at a dinner party

Who don’t know how to speak properly

Television speaks to us

But we can’t speak back

It’s very strength is its weakness

Tomorrow our virtual playgrounds and social spaces

Will become the basis

Of grammers which transcend language between people all over the world

And emerging social necessities

to communicate persuasively

to outshout the other people who are also speaking persuasively

will reverse the broadcast direction.

Ultimately those playgrounds will become spaces

For grass roots decision making by shared participation in synthetic simulations,

Not merely by text, art or static formulae,

or the humiliation of the vote,

(having a lifetime of efficiacy rendered down to a binary decision),

Today we call this a video game,

Tomorrow we will just call it language.

SIX

The media field is so rich

One could start anywhere

And ignore %90 of the things out there

And still have a good overview of the implications.

Who use VRML to compose a choreographed sequence of

The tale of George, an unfortunate existentialist Gorilla,

Who exists only in cyberspace.

This fairly low end experience was generated using fairly accessible tools,

Was broadcast over the internet as a whole,

And while most a technology demonstration,

Certainly showed the declining costs of entry to create and broadcast powerful narratives.

A recipient of the dataset could also choose to peturb poor George,

Although not a simulation, George as a puppet could be easily repurposed.

VRML itself, the outcome of an effort to rationalize divergent grammers for

Describing 3D universes,

And ultimately as a grammer for describing the substrate of a cyberspace,

A visual knowledge space that participants could travel through,

And wherein ideas might be mapped to other things,

Such as an old Redwood forest, where the oldest ideas were the biggest trees,

Or to abstract symbols for that matter.

An effort interesting in that it is supported by many divergent interests,

And by a public interest.

With corporate support from entities like SGI

(and from whose ‘Inventor’ grammer it is somewhat based)

and from private support from long term 3D and VR enthusiasts such as

Bernie Roehl and Josh White and Brian Hook

(being the only ones whom I actually know directly)

and many many others (of course including Marc Pesce who kicked it off).

Of course there is "Motion Factory"

who have built hierarchical state engines

Which in their case are tools to let 3d puppets interact flexibly in their environments,

Decoupling low level issues such as walking and collision avoidance,

From high level issues such as avoiding monsters, or going to a requested location.

And who have a complete user interface

For composing and building situations which follow rules,

And which allow people to explore said situations flexibly.

Hierarchical state machines

Or subsumption theory,

Or simulated annealing

are essentially a more correct model for describing how real world organisms work,

a model that not only works in real application, but feels more sensible,

(than models proposed by the Strong AI proponents, who instead

proposed single massive computations to homogenously solve multilayer problems),

The MIT robotic walking research group

and the ALIFE theorists at the Santa Fe group use these new ideas.

And in that group Tom Ray deserves special recognition for

the emergent life and behavior of his Tierra universe,

and which itself is a good example of the kind of work

game programmers ideally seek to do today,

although "Galapogos" is probably the only game out

yet which does so in any measure.

There are many many sequencing companies,

Even Headspace

(run by Thomas Dolby a British Electropop wizard like Brian Eno)

is to some degree an example of emerging trends spoken about here,

They feature the

beatnik audio player

a tool which allows rich music format

preserving not only sequencing information

but audio waveforms and envelopes as well

Many people are concentrating on squishing data over the net.

And as a result they invent grammers to compress in between

(And decompress against known lookup caches on each end).

Even the Digital TV people, working on fractal based compression

Are to some degree specializing against the type of data they compress,

Where different algorithms work better the grass and player movements of football,

versus the concrete pavilion and clothing of a tennis match.

Ultimately one assumes that the best compression

Is in fact a grammer

Which simply describes the scene at a high level

like a radio play by play.

I would cite Newfire except that they were locked into

a sales model that didn’t provide revenue,

They only provided a way of viewing static worlds quickly.

Their composition tools were not free,

This wasn’t particuarily a viable model – they were dependent

On third parties to generate content, or on licensened parties.

They didn’t have or present a viable revenue strategy.

Today they have paid for their lack of vision,

There is Atomic 3D who while trapped in their own little hell

With an ActiveX specialized, customized (not VRML) web based 3D player

Are still quite likely to be successful

In that they provide a lightweight tool to easily allow

The composition of

Product presentations,

Corporate overviews,

Educational and computer training with visual feedback,

Childrens stories,

Entertainment sitcoms and drama,

Animated 3d websites and 3d mascots,

And ultimately a means for those with free time

To fairly easily make up stories and give them to each other

At any level, be that professional or personal.

Atomic 3D deserves special attention

It is the closest to what this essay speaks about,

It is a simple tool, a complete hack and completely proprietary,

It doesn’t have any physics or behaviors,

But it does have powerful sequencing,

Easily allowing sequencing of events over time,

Which is all one needs to create compelling vignettes (albeit not true simulations).

Even the lip syncing effects are just a side-effect of powerful choreography

And there is no high level abstractions, an arm must be moved by manual labor,

Strangely this company doesn’t seem to have good internal vision,

They don’t see how completely they could dominate the consumer market,

By targeting people who want to simply "play" and share stories with friends.

There is of course a tension between moneyed interests and consumer interests here.

We also see digital avatars beginning to transcend their environments.

Laura Croft is a hit among teen aged men,

And Jack, a general purpose digital puppet is used in many environments

From warfare simulations to digital car crash tests.

Even the avatars in products I work on

Seem to have a character and flavor which exceeds their design.

There is a kind of empathy we have for the human form,

And being able to manipulate that form easily,

To setup emotional situations and drama,

Is a tremendous communicative power.

There are many many projects beyond mention on

Virtual Theater and Improvisational Digital Puppets

Good places to explore these are at the MIT web site

And in the numerous Siggraph papers

As well as at the Computer Games Developers Conference

Today these things number beyond mention

Each to one degree or another evokes strands of ideas

And the total show the trend elucidated herein

The Virtual Theater effort bears special note although it sounds complex:

"The Virtual Theater project aims to provide a multi-media environment

in which users can play all of the creative roles associated with producing

and performing plays and stories in an improvisational theater company.

These roles include producer, playwright, casting director, set designer,

Music director, real-time director and actor. Intelligent agents fill

roles not assumed by the user."

And those crazy guys and gals at MIT deserve special note

"Integrating gesture, speech and facial expression

Into autonomous animated agents,

Allowing multi-modal communication using more than just voice.

And also representing users

communicative intentions non-verbally for graphical online societies"

They also discuss computationally augmented real spaces,

Where computers are used as smart whiteboards to communicate rich concepts

And they even talk about authoring environments for children (see RENGA).

In their meandering way, the MIT people clearly are pursuing the "right ideas"

And it is just a question of time before these ideas percolate down to the public level.

It’s also obvious given the energy overall in this area that there is a gap here.

The "Information Spaces" group has a more pragmatic but still intriguing effort.

They seek to leverage the way that humans are good at navigating 3d spaces,

and they seek to provide tools which let us use these same skills when searching out information.

In essence they try to shape the geometrical space so that it maps logically to idea-spaces.

Given that idea-spaces are more complex, this can be a hard problem

but it is one faced every day by museum exhibit designers, librarians, urban planners etc.

SEVEN

A tool of our own could be used for

entertainment,

Such as letting consumers invent and share new episodes of Star Trek with each other

Or let movie directors prototype real movies to reduce risks and get a feel for the drama

Or for communication,

to transcend language boundaries and to express concepts in loud and evocative ways

utilizing strong visual humor and concepts in ways that have more power than text.

Or for non-communication purposes,

such as modelling people flow and locations in a building,

Or for online collaborative environments,

Where people could represent themselves or ideas with puppetry

And could manipulate those objects directly rather than just describing them verbally,

In a sense the ideal form of a rich media communications environment.

Or (in later versions) for exploratory simulations,

Such as a game where one walked the deck of the Titanic

and tried to change fate, or just explore various fates and aspects.

In such immersive games players could come back to each

Other and talk about how their experiences differed or were the same,

And each player could bring back new knowledge and resonances

That would play upon the whole and enrich the story,

Much in the way game players today love to have long game-postmortem discussions

after a fierce session of strategic play in a video game, or in physical games of sport.

The key for success is staged delivery, and simplicity,

Not to try and solve all problems in the first version, but to get something out there,

(because there is nothing there today which does this,

and because if the trends are right,

this is potentially a much desired tool).

Staged deliveries would also allow consumers to help grow the tool,

And the developer would either be the public itself,

The revenue model would be staged delivery based,

Where the organization derived revenue via derivations of the technology.

Internet distribution and revenue models are substantially different from retail models

Because the cost of product is zero, and the ability for market share to be eroded is high.

The focus becomes retaining and growing customer bases while providing genuine

specialized services which legitimately provide revenue returns.

It would be crucial that the company be able to find aspects of the seed technology

which it could use to generate revenue, ideally mass market revenue,

Immediately upon the initial release.

One approach would be to sell add-ons to the core concept.

Such a strategy might also involve releasing the seed source code,

And thus becoming coordinators of the source and revisions thereof.

As cited in the "Cathedral and the Bazaar" and used in the "Mozilla" strategy,

(and also the strategy of many many other organizations.)

The initial revision of the tool has to be exactly targeted to overcome the

"transition barrier" between where something is a bit to hard to use and

where it is "good enough", and becomes adopted by people into their regular habits.

For example, the Internet and the web-browser resonate with each other in this level,

A web browser makes the Internet "easy enough" to bother browsing through,

And the more determined are able to create their own content with a bit more effort.

Many technologies suddenly burst out into wide market support once having that practical edge.

In some senses there is a richer vein of rich media types that is not prevalent today

Because the tools for communicating and viewing them are not quite yet "good enough".

In its fundamental details, this tool would be a visual scratchpad,

A framing tool for hanging visual concepts on,

Avatars, props, lighting and furniture could be placed and moved,

And sequencing and choregraphy support would be implicit.

This would be released free to garner widespread consumer support,

With the initial focus being on communicating stories in rich media types.

Such a tool is a form of "media engine".

A sequencing tool in the vein of the Beatnik player,

Or any of the manyfold sequencing tools such as Macromedia Director.

But it is specialized for 3D graphics, a virtual stage with virtual puppets.

Interaction with the tool would be visual and iconographic,

Designed for easiest composition of sequences of action.

In staged releases the avatar behavior and collision avoidance would be grown,

This would reduce the need to specify the more subtle details of a drama,

(such as the position of arms or the lip syncing of lips to voice playback).

And subsequent revisions would focus on enhancing the physics

and real world models, such as billowing window curtains.

In essence

This is what game developers do every day.

It is a game engine with choreography support.

Its target market isn’t reliant on the fact that it is a

Canned game experience

Or that it has a high score

Or that it puts the player in an alpha state

Or onto the treadmill of an "infernal machine"

– a machine that never stops - as Ron Martinez says,

But that is simply exposes the kinds of mechanisms

That we as game developers simply take for granted

Every day when we make the games we make.

In a sense it is just a subset of a game,

With pretty polished handles on the levers

That make things move.

At a lower technical level there are issues in,

Database design – flexibly managing lots of heterogenous assets,

Compression and standardization of the data formats,

Quick playback (although today that is diminishing),

Concepts in hierarchical state machines,

Providing good controls for the users.

And providing lots of rich content.