How Smartphone Cameras Work.

Smartphones have mostly replaced the point and shoot camera, mobile companies are scrambling to compete where the old imaging giants reigned supreme. In fact, smartphones have completely dethroned the most popular camera companies in photo communities at large like Flickr: which is a big deal.
But how do you know which cameras are good? How do these tiny cameras work, and how do they seemingly squeeze blood from a stone to get good images? The answer is a lot of seriously impressive engineering, and managing the shortcomings of tiny camera sensor sizes.

How does a camera work?

 

With that in mind, let’s explore how a camera works. The process is the same for both DSLRs and smartphone cameras, so let’s dig in:

1. The user (or smartphone) focuses the lens
2. Light enters the lens
3. The aperture determines the amount of light that reaches the sensor
4. The shutter determines how long the sensor is exposed to light
5. The sensor captures the image
6. The camera’s hardware processes and records the image

Most of the items on this list are handled by relatively simple machines, so their performance is dictated by the laws of physics. That means that there are some observable phenomena that will affect your photos in fairly predictable ways.
For smartphones, most of the problems will arise in steps two through four because the lens, aperture, and sensor are very small—and therefore less able to get the light they need to get the photo you want. There are often tradeoffs that have to be made in order to get usable shots.

How does a camera focus?

 

Though the depth of field in a smartphone camera’s shot is typically very deep (making it very easy to keep things in focus), the very first thing you need the lens to do is move its focusing element to the correct position to get the shot you want. Unless you’re using a phone like the first Moto E, your phone has an autofocus unit. For the sake of brevity, we’ll rank the three main technologies by performance here.

1. Dual-pixel
   Dual-pixel autofocus is a form of phase detect focus that uses a far greater number of focus points across the entire sensor than traditional phase-detect autofocus. Instead of having dedicated pixels to focusing, each pixel is comprised of two photodiodes that can compare subtle phase differences (mismatches in how much light is reaching opposite sides of the sensor) in order to calculate where to move the lens to bring an image into focus. Because the sample size is much higher, so too is the camera’s ability to bring the image into focus quicker. This is by far the most effective autofocus tech on the market.
2. Phase-detect
   Like dual-pixel AF, phase detect works by using photodiodes across the sensor to measure differences in phase across the sensor and then moves the focusing element in the lens to bring the image into focus. However, it uses dedicated photodiodes instead of using a large number of pixels—meaning that it’s potentially less accurate and definitely less fast. You won’t notice much of a difference, but sometimes a fraction of a second is all it takes to miss a perfect shot.
3. Contrast detect
   The oldest tech of the three, contrast detection samples areas of the sensor and racks the focus motor until a certain level of contrast from pixel to pixel is reached. The theory behind this is: hard, in-focus edges will be measured as having high-contrast, so it’s not a bad way for a computer to interpret an image as “in focus.” But moving the focus element until the maximum contrast is achieved is slow.

 

What’s in a lens?

 Unpacking the numbers on a spec sheet can be daunting, but thankfully these concepts aren’t as complicated as they may seem. The main focus (rimshot) of these numbers typically encompass focal length, aperture, and shutter speeds. Because smartphones eschew the mechanical shutter for an electronic one, let’s start with the first two items on that list.

 While the actual explanation of focal length is more complicated, in photography it refers to the equivalent angle of view to the 35mm full-frame standard. While a camera with a small sensor may not actually have a 28mm focal length, if you see that listed on a spec sheet, it means that the image you get on that camera will have roughly the same magnification as a full frame camera would with a 28mm lens. The longer the focal length, the more “zoomed in” your shot is going to be; and the shorter it is, the more “wide” or “zoomed out” it is. Most human eyes have a focal length of roughly 50mm, so if you were to use a 50mm lens, any snapshot you took would be roughly the same magnification as what you see normally. Anything with a shorter focal length will appear more zoomed out, anything higher will be zoomed in.

 What is camera sensitivity?

 

When you adjust camera sensitivity (ISO), you’re telling your camera just how much it needs to amplify the signal it records in order to make the resulting picture bright enough. However, the direct consequence of this is increased shot noise.
Ever look at a photo you took, but it has a ton of multicolored dots or grainy-looking errors all over the place? That’s the expression of Poisson Noise. Essentially, what we perceive to be brightness in a photo is a relative level of photons hitting the subject, and getting recorded by the sensor. The lower the amount of actual light hitting the subject, the more the sensor has to apply gain to create a “bright” enough image. When this happens, tiny variations in pixel readings will be made much more extreme—making noise more visible.

 After the shot is taken

 

Once your camera takes the shot, the smartphone has to make sense of everything it just captured. Essentially, the processor now has to piece together all the information the sensor’s pixels recorded into a mosaic that most people just call “a picture.” While that doesn’t sound terribly exciting, the job is a little more complicated than simply recording the light intensity values for every pixel and dumping that into a file.
The first step is called “mosaicing,” or piecing the whole thing together. You may not realize it, but the image the sensor sees is backwards, upside-down, and chopped up into different areas of red, green, and blue. So when the camera’s processor tries to place each pixel’s readings in the correct spot, it needs to place it in a specific order that’s intelligible to us. With a Bayer color filter it’s easy: pixels have a tessellating pattern of specific wavelengths of light they’re responsible for, making it a simple task to interpolate the missing values between like pixels. For any missing information, the camera will dither the color values based on the surrounding pixel readings to fill in gaps.
But camera sensors aren’t human eyes, and it can be tough for them to re-create the scene as we remember it when we snapped the photo. Images taken right from the camera are actually pretty dull. The colors will look a bit muted, edges won’t be as sharp as you might remember them to be, and the filesize will be massive (what’s called a RAW file). Obviously, this isn’t what you want to share with your friends, so most cameras will add in things like extra color saturation, increase contrast around edges so the shot will look sharper, and finally compress the result so the file is easy to store and share.