introduction

The purpose of this digital paper is to make connections between

• sound + image + medium + narrative + context + evoking meaning

for the purposes of teaching graphic design students.

 

Electronic media affects the ways in which information is patterned, how it is received, and how we communicate. It conditions our brains to receive and process information in the same way we process sound. (Schwartz, 1972) . This has not been proven, to my knowledge, although the concept has interested me for many years.

The most significant change in graphic design is motion. This technological development goes beyond simply augmenting conventional representation of information; it provides us with an alternate means of storytelling. (Heller and Dooley, 2008)

Visual culture isn’t just visual. (Duncum, 2004)

Some works:

• Elizabeth Rondone, graphic design student, Words
• Daniel Burch, graphic design student, Experiment with Motion

 

 

 

 

 

 

 

 

 

 

recall

Recall plays an important role in time-based media. Consider the different processes used in making a simple comparison between two images.

When images are presented side-by-side, parallel in space, located close together on the page, "we are able to canvas, sort, identify, select, contrast, review, -- ways of seeing all quickened and sharpened by the direct spatial adjacency of parallel elements". (Tufte, 1997)

(Tufte, 1997)

 

 

 

 

 

 

 

 

 

 

 

 

Images appearing as parallel in time, meaning first one image of a particular subject is presented, followed in time by another image of the subject, require the viewer to recall the first image, in order to contrast, review and relate it with the image currently visible. Compare the photographs of Degas’ sculpture, Cheval à l’ arrêt and an x-ray revealing the wire structure. (Tufte, 1997)

(Tufte, 1997)

 

 

 

 

 

screen technology

Although I do not understand fully the processes in which our eyes and brain are engaged when putting together digital display images, I think it is important that the image is never visible in a single moment in time. The complete picture must be "assembled."

TFT-LCD stands for thin-film transistor liquid-crystal display.
TFT-LCD technology is the popular choice for flat panel displays to date. Variations of this technology are used in screens for televisions, computers, and mobile communication devices. (AU Optronics Corp, 2010)

 

The basics of TFT-LCD technology. (Tyson, 2010) (Steer, 2008)

• Transistors and capacitors are arranged in a matrix on a glass substrate
• and they control the voltages sent through to liquid crystals.
• The liquid crystals are bent, based on the voltages they receive, allowing light to pass through to a pixel.
• Each pixel is comprised of three subpixels, with red, green, and blue color filters.
• The intensity of each subpixel can range over 256 shades. Combining the subpixels produces a possible palette of 16,777,216 colors (256 red, 256 green, 256 blue = 16,777,216).
• The capacitor is able to hold the charge until the next refresh cycle
• Refresh rate refers to the number of times the image is updated per second..
• Response time is the time taken for the light throughout a pixel to fully react to a change in its electrically programmed brightness.

 

 

 

 

 

 

 

the ear

(eduMedia, 2010)

Our brains process sound in a pattern similar to the way electronic media conditions our brains to receive and process information. (Schwartz, 1972)

The outer, visible part of the ear, collects sound waves and channels them into the auditory canal.

The eardrum is the thin membrane that separates the outer ear from the middle ear. Air pressure fluctuations force the eardrum to move back and forth in accord with the frequency of the sound waves.

In the middle ear, the eardrum is attached to the ossicles (hammer, anvil, stirrup), which are the smallest bones in the human body. They transmit the vibrations to the cochlea.

The cochlea in the inner ear converts mechanical energy into nerve impulses that are sent to the brain. The cochlea conducts sound through a fluid where thousands of hair cells send electrical impulses through the auditory nerve to the brain. (eduMedia, 2010)

The brain stores this vibration, recalls the previous vibration, and anticipates the next vibration, as it processes the spoken word, music, any sounds it receives.

 

 

 

 

 

contexts

Context shapes the meaning of messages.

"To study the visual culture of contemporary everyday life, it is necessary to acknowledge that no matter how important the visual characteristics of contemporary cultural sites are, in various ways and in varying degrees, they all involve other sign systems and appeal to multiple perceptual systems". (Duncum, 2004)

The experiences we have when browsing the Internet using a desktop computer and a mobile device are quite different. Expectations, the messages we construct and receive are not identical.

 

Desktop	Mobile
Large Screen	Small Screen
Fast Internet	Slow Internet
Have Time to Browse	Don’t Have Time
Quiet Environment	Noisy, Distracting Environment
Have Pen and Paper to Take Notes	Hard to Take Notes
Sitting Down	Standing or Walking
Have Separate Phone	Phone and Web in Same Device
Focused on Task	Multi-tasking
Artificially-lit Environment	May be in Strong Sunlight
Full/Larger Keypad	2 or 3 Letters to a Key

(Boehm, 2010)

 

 

 

 

 

resonance

Communication packages that resonate with information already stored within an individual evoke meaning.

Listeners/viewers draw on their stored life experiences and expectations in making meaning from a communication package. Our original experiences are more directly available when they are recalled, “evoked by a stimulus that is coded in the same way as the stored information is coded”. (Schwartz, 1972)

Listen and think of experiences, time, places, moods, movements associated with each sound.

crickets

drums+voices

 

 

 

inbetween + overlapping spaces

Being literate in contemporary culture goes beyond simply having the ability to encode and decode individual media. Literacy cannot be separated from its social contexts. Meaning of a message is shaped through application, technologies, and social preoccupations. Meaning lies within what we make of the space between discrete communicative systems. (Duncum, 2004)

 

A simple example is to use the same video footage with two different sound tracks.
Coffee version one.
Coffee version two.

 

The original video footage and soundtrack NOAH's Friends, PSA.

 

 

 

 

 

 

narrative

Narrative is a universal method for understanding and organizing events in time.

The narrative presents a succession of events that lead up to what we are experiencing at the moment. We take in the bits of information, compare it to the new information we receive, and anticipate how the story will develop. (Abbott, 2008)

(Katz,1991)

 

 

 

 

 

exploring

In my Art 479, Motion Graphics Class, I ask my students to create a simple sound narrative. They use sound effects, environmental sounds, and melodies to structure the events and suggest an environment, rather than relying on the spoken word to tell and describe the story. The students arrange these sound clips in layers along the timeline in After Effects. During the first critique students realize the listeners make connections to personal experiences as he or she tries to interpret the sounds. It is clear the listener plays a significant role in shaping the meaning.

The students’ success in communicating the intended meaning are linked to their ability to evoke meaning by using sounds grounded in common shared experiences.

Jefferson Rockett Sound Only Narrative

Kirk Gunton Sound Only Narrative

Briana Randle Sound Only Narrative

The idea is for the students to “see” images by recalling the experiences associated with sounds and to try to evoke experiences stored within the indivudal listeners; to create a resonance with the listeners.

 

The second component of the assignment involves adding abstract images,and motion to the sound narrative, again using After Effects. The use of abstract images is intended to guide students away from literal illustrations of events associated with the sounds. The students are asked to explore an alternate means of telling the story, and to allow the listener/viewer opportunities to participate in the interpretation by making connections in the in-between spaces of sound and image.

Jefferson Rockett Sound and Image Narrative

Kirk Gunton Sound and Image Narrative

Briana Randle Sound and Image Narrative

 

bibliography

AU Optronics Corp., Technology, 2010. Retrieved from http://auo.com/auoDEV/technology.php??&ls=en on 10/10/2010.

Abbott, H. Porter, The Cambridge Introduction to Narrative, 2008, Cambridge University Press, Cambridge, UK.p.13.

Boehm, Nathanael, The Mobile Experience is Nothing Like Desktop, purecaffeine.com, September 10, 2010. Retrieved from http://www.purecaffeine.com/2010/09/mobile-desktop-experience-design/ on 9/18/2010.

Duncum, Paul, Studies in Art Education, Volume 45, Spring 2004, Visual Culture Isn’t Just Visual: Multiliteracy, Multimodality, and Meaning, National Art Education Association, p. 252-262.­­

The Ear, 2010, eduMedia, France and Canada, Retrieved from http://www.edumedia-sciences.com/en/ on 1010/2010.

Heller, Steven and Michael Dooley, Teaching Motion Design, 2008, Allsworth Press, NY,NY, p.xi.

Katz, Steven, Shot by Shot, 1991, Michael Wiese Productions, Studio City, CA, p.150.

Schwartz, Tony, The Responsive Chord, 1972, Anchor Press/Doubleday, Garden City, NY, p.24-25.

Steer, W.A., LCD Technology and Tests, Last modified: 21 April 2008, Retrieved from http://www.techmind.org/lcd/ on 10/03/2010.

Tufte, Edward, Visual Explanations, 1997, Graphics Press LLC, p.79.

Tyson,Jeff, “How LCDs Work”, 2010, HowStuffWorks, Retrieved from http://www.howstuffworks.com/lcd.htm on 10/19/10.