"3D" sound effect discussion.

[Insectoid]

You guys have all had the virtual haircut (google it if you haven't, it's the first result). We'll all no doubt agree that it was a very engaging experience that you fully enjoyed. I bet you smiled during it.

It bothers me that this technology isn't being developed and researched. This stuff hasn't really changed in quite a few years. How would you guys react if the next Crysis launched with a new NVidia (or some other brand, doesn't really matter) sound card and supported this crazy '3-D' sound? Current graphics technology is 'good enough' for most people. We aren't really impressed by crazy good graphics anymore (until hardware can render Avatar in real time and artists are willing to do that much work there's no reason to continue improving graphics). There is value in sound. Don't quote me on this, but I think, if backed by a coalition of various hardware/software corporations and experts on sound we could see a ridiculous improvement in sound quality in games and movies. If you're playing a FPS and you get shot, your first clue as to where your attacker is, is a visual representation on the screen (often a red arrow/blood splash on the side of the screen you were hit). This shouldn't be the case. The first cue should be sound. With current sound effects, you might know your assailant is 'somewhere to the left'. If someone in your room is talking to you, do you not know exactly where he/she is, without looking? You don't have to look for your friend; your head just spins directly to his position. Even in our language, we refer to having 'seen' things, weather or not it's possible to have seen. I even had to re-write my first sentence in this post, because I'd written "you guys have all seen..." instead of "you guys have all had...", though you can't 'see' the virtual haircut.

Our strongest of the 5 official senses is sight. We rely on it more than anything else to describe our surroundings. That's why graphics receive such attention. We undervalue sound, because it's more of a supplementary sense that notifies us of changes in our environment. I'm not saying sound is not important; quite the contrary. Sound can do things that sight isn't good at. Navigating a gui menu with a mouse is easy, right? It's even easier in my experience if you add mouseover sounds to the buttons (though some might want to investigate this claim; my 'research' is not scientific at all and may be akin to a placebo- I don't know). Sound is 360 degrees. Sight is not. Sound is much better at picking up changes than sight. "Find the differences between these two pictures" is trivial with sound. We notice small changes in them.

With our current technology we can digitally create very realistic human faces and render them in real time. Have we even tried to recreate a human voice? Microsoft Sam (if I remember his name right) is probably a bunch of recorded sounds that the computer puts together in order to 'read' text. 'Sound' art for games in its current form, is just some guy recording a bunch of existing sounds, combining and tweaking them, and then playing them back.

I suppose a major roadblock in the development of really awesome sound is the fact that sound is 'serial' whereas sight is 'parallel'. When you look at a picture, you get a complete image at once. When you hear a sound, it comes in a stream. We can control what we see. By changing our position, we instantly see dramatic changes in our environment. We have thousands and thousands of years of artistic development that really formed the backbone of digital 3D environments. In most way, visual artists (primarily those attempting to capture reality) are the giants whose shoulders we stand on in the field of 3D computer graphics. It's the same principals. We don't have those thousands of years of 'research' in sound. Not much effort has been put into recreating it. Like we skipped painting and invented the photograph. Music creates new sound, but it doesn't capture or interpret existing sounds (hell, it interprets and captures visuals better). Without this backbone, how are we to digitally synthesize '3D' sounds? I think we can, but we need to start from the beginning. Maybe it will take a long time, but I don't think so. We have technology and existing research in similar fields that could dramatically accelerate the process. The problem is having someone actually do it, with the funding they need.

Hell, maybe we don't need to start from scratch. Maybe we just need a new way of recording sound, that allows it to be 'normalized' and then use a mathematical equation (cetera or something, I forget) to 'move' the sound somewhere else. Start with how a gunshot sounds, inside the explosion. Move the sound through the gun 'object' which modifies the sound. Add sound diffusion and whatnot for being in a room. Basically, you'd calculate the echoes, and the changes in the sound when it echoes off an object. Over time we should be able to find shortcuts to improve the overall quality of the sound by reducing the quality of things we don't need.

As for consumer adoption? I think this would be much more popular initially than 3D video. This '3D' sound is not 3D in the manner than a 3D movie is 3D. It uses perspective to change what is sent to your brain, and let's your brain to the work. We aren't very impressed by 'real' 3D video because our brains don't like it. It forces our brains to apply its built-in algorithms to parse information it was never meant to parse. We aren't used to conforming to a defined framerate. In a regular 2D movie, our brains apply the formula in real time at our own pace- we just 'render' the same frame more than once, or skip frames as needed. In 3D films, we get half an image, then another half image, and then our brains combine them. It's slow and unnatural and so we get headaches. Our brains don't like it. This 3D sound feeds different sounds to your different ears at the same time. Your brain adds the sounds together naturally at its own pace to create a 3D sound, just like it does in real life. If 3D video used 2 screens 'on top of each other' to feed data to each eye simultaneously it might be much more natural and popular, but that's pretty hard to do I'm sure you can see. Listen to the barber shop thing again. Do you experience anything unpleasent? Does anything (except the noteable lack of visual cue) detract from the realism? No. Close your eyes and Luigi is right there. It feels real, whereas 3D video feels fabricated. If 3D video actually felt like a window into another world it would be hugely successful. Bonus: Any stereo headphones create the effect. No need to buy expensive, ugly, uncomfortable 3D glasses. The need for headphones is actually a plus; you can get headphones you forget about, noice-cancelling headphones, you could watch with regular speakers and regular 2D sound. You might not even need a 2D 'version'; I dunno how pronounced the difference between 3D sound from regular speakers and 2D sound is when synced with a movie; I don't notice any anomalies in the virtual haircut but it isn't synced with a video so it might actually be hugely annoying/out-of-sync.


This rant has become quite long and I'm losing track of what I've said and what I've not, so I'll leave you guys with a note: I am by no means an expert on sound. I may be completely wrong. Maybe AMD has a team already that's been working on this for a long time. Maybe it wouldn't work out at all in a video. I guess we need a proof-of-concept micro movie (5-10 minutes long) to demonstrate this (virtual barbershop is notable devoid of video). The 3D microphones already available could be used in concert with ordinary or 3D cameras to produce different comparisons (2D sound + 2D video, 2D sound + 3D video, 3D sound + 2D video, and 3D sound + 3D video). One such might already exist and I just haven't seen it.

And now I leave you (for real this time) with something to think over: I just listened to the virtual barber shop like eight times in 3D sound. I don't notice when it ends. I might hear someone jump into a pool and it fits right in. Non-descript noises like squeaks, birdcalls, etc, might as well be a part of the demo. Minutes later I might realize that what I'm hearing, is not the virtual haircut anymore. When I listen to it in 2D, it's instantly separable from outside noises, no matter what they are.

[Insectoid]

I was discussing this with someone, and he thought this would not work for movies, because they are not 'first person'. I disagreed. I believe that all movies and games are from a 1st person PoV, and your 'character' is a god who just watches the story unfold, and you can fly and time-travel and whatnot. Gods like to hear things from their own perspectives too you know. Maybe 2nd person a more accurate description.

Mostly, I just want to discuss the subject, and theorize about how one might accomplish this. How do you capture a 'base sound'? What would a 3D sound 'object' look like? Would these computations be demanding enough to merit dedicated 3D sound cards? Is this a logical next step in game/movie technology or am I just a complete nut?

Have a listen at this video (it's old so you might have already seen it):
http://www.youtube.com/watch?v=wT1XuB95qMk
It isn't exactly exciting. It's just a bunch of really basic sounds. Matches, tape, a blow-dryer. And yet, people love it. They're amazed by it. They'll listen over and over again. I imagine this is how people felt when 'moving pictures' first came out. Did people feel this way when 3D was new? I've listened to both this and the haircut hundreds of times, and they never, ever lose their charm, as I expect only someone who grew up when video was new could have felt.

I said "I leave you with that" and I meant that as an end to the summary of what I'd already theorized. This post is here to ask questions and begin the discussion, so I didn't lie

EDIT: Here's a different haircut, with video! http://www.youtube.com/watch?v=p6G7JR2Kfnc&feature=related. It also demonstrates a 'voice in your head' which is actually really cool, and could be very well utilized in video games. The video itself doesn't seem perfectly synced to the sound but this doesn't look like the most expensive production either . It's still really, really good.

[DemonWasp]

I don't have enough background (nor good enough ears) to actually have much of a say in this, but my experience has been that a lot of games do have sound down pretty well. In CoD4, for example, I can hear where footsteps, gunshots and bullet impacts are, who's yelling what, and where grenades are. It sounds surprisingly similar to what you hear in the video you linked.

The game has the advantage, over the video, of being able to use all the speakers in my 5.1 headset, though.

[andrew.]

A while ago, I remember reading about a group of people researching how to simulate sound. They put a proof-of-concept video somewhere on YouTube and basically it was just a video of running water. What made it interesting though was that it was completely simulated, both the visual and audio aspects of it. From what I understand, they simulated the water and air particles to make the sound.

I just did a quick Google search and found this: http://www.cs.cornell.edu/projects/Sound/

I'm not sure if it's the same team or even if they're pursuing the same goal, but the video on their page is interesting nonetheless.

[Insectoid]

I've always found surround sound to be low quality (the sound itself is high quality but the 'surround' element is not). I've never used a surround headset though. When I play Cod4 (which is pretty often) I can't hear jack squat. I mean, if someone throws a grenade at me, I know it's near me, and maybe the general direction, but unless I actually get the 'throw back' icon, I don't know where to run (most people cook their 'nades so long though that it doesn't usually matter XD). Damn plain stereo headphones!

Note that the video is poorly synced with the sound and is overall low quality (the owner of the video told me Youtube broke it and the original is much better), so you can't expect the full effect. Also, their sound perspective is off. The idea though is that once the software exists, all you really need is a headset (and maybe a sound card for computers, I dunno). I got my headset for ten bucks. How much did you pay for that 5.1 headset? I'm genuinely curious about this.

How big is your headset? Is it noticeable? Is it something I'd like to stick on my head in a theatre? Or is it small and light? Would you forget about it if you had it on long enough? Again, I just don't know.

The most important question is though, if you closed your eyes, would you hear Call of Duty, or would you hear Iraq? More than just 3D sound, but the details, the things you ignore that make it real. I want a complete overhaul of the way sound is managed in movies and games. If you have 20 artists drawing textures, you should have 20 people developing sounds.

ID tech 5, aka the Doom 4 engine (we can't really call it the RAGE engine can we?) supports 128,000x128,000 pixel textures. Terabytes and terabytes of image data. You can zoom in to the tiniest detail that the artist can create. The game engine isn't the limit anymore. It's what the artist comes up with.

EDIT: This was in response to Demonwasp; Andrew stealth-posted.

So should it be with sound. We have developed light occlusion. We have unique textures for every single thing. Why not unique sounds for everything? The tiniest sounds could exist. The nearly silent buzz from a light bulb, a breeze coming in through a window, the cloth of your pants rubbing against your legs. Most games half-ass it. They go overboard on graphics, but the sound is minimal. It doesn't sound real.

Of course, I was pretty much just discussing 3D sound in my original post, but it just doesn't work without all the tiny little bits. If you listen closely you can hear the wind off the guy's arm as he moves around your head (probably because, there was an actual arm moving around just a couple inches from the mic). These things convince your brain it's hearing the real world. Of course, this is all my own theory and I may again be way off course.

I think as much effort as is put into graphics should be put into sound.

[Insectoid]

@Andrew- that video shows excellent synthesized sound. I'm actually really impressed! The perspective is still off though; I'm hearing the sounds in my head rather than on the screen. It may have been designed for bigger or smaller screens though, or a TV and living room instead of laptop. Or, maybe they just forgot about perspective. Or, maybe it's really hard to pull off. This is how I imagine the first few first-person games to utilize this technology, will utilize this technology. The character is at the screen, and the sounds are happening all around him, so you should hear what he hears right?

Wrong. You should hear what's happening a game-foot or three behind him, because that is where you are. You aren't seeing from the character's perspective; you're seeing through the back of his head. what? Yeah. In a first person game, the guy's eyes are at the screen. His nose extends into the screen, and his head extends out of the screen towards you. He 'exists' between you and the screen. To do this would require either moving the sound a couple feet back, to moving the character a couple feet back. The former would be easy, but it would be more like an alternate first-person view, because you are considered a second 'character' that follows the main character around silently (you might as well be a ghost) instead of being the character himself. You don't see what he sees; you see a projection of what he sees. The later might be more complicated because you change the visual perspective to actually be you. It would be like looking through a hole in a wall. The character is sitting where you are sitting, so you lose a couple degrees of vision (by moving the character further from the screen) but gain much more immersion. If your character is thinking, you can create a 'voice in your head' which is actually ridiculously awesome. your character talking is muffled by your own head (well, a virtual skull that is where your skull is). It sounds like you are talking (The video I linked in my 2nd post does this well). You hear what he hears. You see what he sees, not a projection (I think this would make great horror and war games).

If games incorporated the 'wiimote' method of rendering 3D, that would be much more immersive and allow much more accurate 3D sound. In fact, that sort of 3D video brings out the same feeling of awe that this 3D sound does, so why even bother with current 3D video? We're already sticking stuff on our head for it; a small IR sensor is no worse (though this breaks in theatres or whenever multiple people are viewing...but 3D sound pretty much breaks then too anyway). I'd personally rather my on-body TV/game accessories be as minimal as possible.

I feel with this post that I'm arguing with myself, which is good, but I'd like to see other people's input before I think too much.

The biggest thing that I want to stress here, is perspective. My last post stressed details, which are also important. This post is stressing perspective equally as much.

Perspective and detail seem (in my mind) to be the important things here.

@Andrew, if a school is developing things like that, imagine what a high-budget studio could do with resources from Microsoft or NVidia or Intel.

[DemonWasp]

The headset is pretty big, but not cumbersome for an hour or two of listening; I've also gotten used to it, so I often don't really notice it. It only ever becomes an issue when it's 45C and I'm sweating like a dog. The headset in question is this: http://www.medusa-canada.com/medusa-nx-51-gaming-headset-p-922.html ; I used to recommend them whole-heartedly but I've found that they're disappointingly fragile -- I've had 3 of them fail on me in as many years, and I'm never rough with them.

I play CoD4 on hardcore, which means no grenade-throwback icon. Regardless, I can usually tell exactly which direction I should run to avoid the grenade, where bullets are coming from, etc. The only thing I can't hear is someone with Dead Silence sneaking up on me (because they actually produce almost no sound at all).

The biggest difference between what I hear in CoD4 and what you would hear in the real world is that there's only a few hundred sound effects in CoD4 at most. Each gun has at most a few distinct shot sounds (and I expect that they actually have only one each), there are only a few environmental sounds, etc. You don't hear the wind, or the crunch of frozen grass beneath your feet, or your clothes settling.

The second-biggest difference is that game sound is modulated to be in a comfortable range. In reality, footsteps are only ever around 80DB, but a gunshot can easily hit a deafening 140DB (100,000 times as much sonic energy, roughly 64 times louder, subjectively...I think).

[mirhagk]

DemonWasp has a really good point about the comfortable range. Personally I don't want to listen to realistic level gun shots (I dunno if you guys have ever gone to a shooting range, but there are protective ear gear for a reason). So you have to sacrifice that, which then leaves you with 2 choices:

choice 1: just make everything quitier, obviously this is not desirable

choice 2: make the gun shots and other loud things quieter, sacrificing realism for comfort.


I would also like to note that in a war game, with perfectly realistic sounds, you would not hear 99% of the sounds (do you really think wind could ever be heard in a war zone?) so making them is completely a waste of time. I'd rather they spent that money and resources on stuff I'm going to notice all the time, like gameplay, or even visuals you notice most of the time.

[Ultrahex]

Graphics (not just 3D) is a longtime process that has been developed based on the combination of many different fields, in fact most of the methods today we use in graphics do not originate from the fields of graphics directly but are derivations from other fields.

Sound is also very similar in this way, most sound technology comes from physics, chemistry, and biology.

So your wondering why 3D sound is not a standard, well it is in a way but it is a lot harder to understand.

They commonly make a bunch of crazy claims in sound technology as they do in graphics technology (at least from how I perceive it).
Like think about this, we (I hope we are all humans) only have two main sound receptors in our ears so a 5.1 surround sound system has 4 excess emitters to overcome lack of audio quality and clarity.

We also all remember from physics (if we took it), that even two speakers actually creates a DEAD-ZONE in the room (it may only for matching wavelength, but I could be wrong), resulting in silent even though it could be extremely loud in the room.

For 3D sound to take off we would require a need for it, and the fact of the matter is we don't really have one yet. On-top of this, it would require a lot more computational complexity to make realistic noise based on 3D environments, hence why all sounds in games are recorded/mixed together before being played and take into account distance for how loud something should be.

There is some companies doing research into the area, and 3D sound (to some degree) is already in most video games just not at the complexity you are indicating.

However, there is obvious improvement to be made in visual graphics in video games still today; and the benefits of graphics is a lot more appealing for research than sound (one of the major arguments I could see here).

Next think about how many people actually care about the quality of sound they listen to; sadly it is not as many as you would think considering most are willing to go to concerts in buildings that are not properly designed for the purpose of loud audio.

[Insectoid]

I dunno. If a there was a demo game or movie with perfect 3D sound, and people were exposed to it, then we would see if this is something they'd want. I think it would be, based purely on how people react to hearing the sounds alone.

[Ultrahex]

"perfect" 3D sound is something that is probably not possible with todays hardware, but it totally depends on your definition of what "perfect" is.

[mirhagk]

Actually I would like to point out that we care more about sound then we do graphics in a lot of ways. If the frame rate of a video drops, or the video freezes for half a second, you may not even notice, but if the sound freezes, or cuts out, you will notice immediately.

On the other hand however if I play a song, then the next day play the song in a completely different key, very few people would notice. But if we had a character in a game, and then the character with a different tint, you would notice immediately.

The applications of getting better visuals definetly helps out other areas alot more than sound does. A simple pair of headphones with any old mp3 player is capable of delivering "perfect" sound (it just needs to be created as perfect, delivering it is easy). However we need VERY complex devices to deliver even somewhat realistic looking visuals.

An argument could definetly be made that video games are the reason why we have such amazing computers nowadays, and without them, there never would really have been a need to improve the computer very much.

Sound on the other hand does not provide the same drive, and mastering the output of sound requires very little hardware, yet can be screwed up so easily on the user's end.