Number of dimensions? Color image vs gray scale image? Colour video vs gray scale video? Especially in the context of MATLAB

I am very confused as I had a debate with my buddy regarding color (RGB) images. He insisted that color images are two-dimensional, but when I personally try myself to read a color image into MATLAB, I get a m x n x 3 size matrix and when I convert that image into gray-scale, I get m x n size matrix.

My buddy argues that in a two-dimensional image, distance between two objects can not determined properly.

For video, how many dimensions are there in case of a gray-scale video without sound/audio? And how many dimensions are there in a gray-scale video with audio/sound?

Similarly, how many dimensions are there in a colour video without sound/audio? And how many dimensions are there in a colour video with audio/sound?

My buddy says every video, whether gray-scale or colour is two-dimensional, and accompanying sound/audio can be considered a third dimension in case of video.

• @Royi I don't think your edited title accurately reflects the question. Commented Mar 6, 2022 at 6:59
• @Royi sorry i rolled back your edit. Actually i am very confused regarding the subject especially in context of MATLAB Commented Mar 6, 2022 at 7:32
• MATLAB uses memory to arrange the data. The data is the values of the function. The way we access it is the definition of the domain of the function. The layout of the image is 2D hence we have 2D access pattern to it in MATLAB. Gray or Color just define the number of outputs per data point of the image.
– Royi
Commented Mar 6, 2022 at 8:05

Color images are usually modeled as a vector valued function of 2D:

$$I : \mathbb{R}^{2} \to \mathbb{R}^{3}$$

Namely for 2D coordinates input it outputs 3 values (RGB).
Hence images are 2D functions.

In MATLAB, you get the input 2D (Matrix Coordinates) and output as a 3D vector. You see it as a 3D array due to way we want to efficiently layout data in memory. So if the image is given by mI, then its input is row and column indices and its output is 1 x 1 x 3 vector:

mI(5, 2, :)


Easiest to see with the displaying of the data:

You can see input to the function (Domain) is X, Y and the output [R, G, B]. Functions dimensions is set by the number of inputs. Hence 2D.

For videos the input are [X, Y , T] hence they are 3D functions with 3D output (For RGB colored videos).

Holograms, if considered images, have depth, so you have [X, Y, Z] as input and [R, G, B] as output.

Regarding Video + Sound, the function is still 3D (Assuming the temporal sampling grid of the sound and the video match). We just have more output per point. So the input is [X, Y, T] and the output [R, G, B, CH1, Ch2, ..., ChN] where there are N audio channels.

Again the dimensions of the function is determined by its domain, the memory layout in MATLAB is, usually, the number of input dimensions + 1. Where the number of elements of the dim + 1 is the number of outputs. So the data for Video with 2 channels of Audio would be [numRows, numCols, numFrames, 5] where 5 = 3 + 2: 3 color values + 2 audio values.

• You said images are 2D,. But why MATLAB shows a 3rd dimension besides rows any columns ,when we use" whos' command Commented Mar 6, 2022 at 7:32
• Because the functions is sampled. So the grid it is sampled on is the domain of the function (Matrix: 2D, Row and Column) and the data is the number of dimensions of the output (3). So you get 3 outputs for 2 inputs. This is the definition of a multi valued function of 2D space.
– Royi
Commented Mar 6, 2022 at 7:44
• So even in MATLAB, it is a 2D function with 3 output.
– Royi
Commented Mar 6, 2022 at 7:45
– Royi
Commented Mar 6, 2022 at 8:12
• Thanks for your update, what is being referred by [x,y] [379,48], [R,G,B] [222 125 110], Is it referring to the black dot/spot above hat of lena?? Where [x,y] are the spatial coordinates of that black spot and [R,G,B] matrix gives the intensity of red,green and blue at that specific spot?? Commented Mar 6, 2022 at 10:11

My answer became quite long. So normally a digital signal is 1D, a digital image is 2D, a video is 3D. But it can get complicated.

Long introduction on mathematics for the start. In mathematics, the term dimension may denote several concepts, that are sometimes loosely used in engineering (especially with discretization, as we will see later). However, dimension is generally used to characterize geometric objects: a surface, a space, a curve, but less likely a function. This may seem subtle: I would not talk about the dimension of the (simple) function $$f:x\mapsto f(x)$$, but I may talk about the curve object defined by the set $$F$$ of all points with coordinates $$(x,f(x))$$, see The Difference between a Graph and a Function?. Complements can be found in Number of variables and dimension of a function.

An first intuitive interpretation of the dimension of some object is the number of independent variables or parameters needed to defined it. Therefore, while the $$(x,f(x))$$ graph is drawn in the 2-D space or plane (spaces or planes are geometrical objects), it can be considered 1-D: when $$x$$ is known, then $$f(x)$$ is as well. A surface would be the graph of a function $$f:(x,y)\mapsto f(x,y)$$ etc. A second vision is more connected. A point of dimension 0-D removed from a 1-D curve separate it into two disconnected parts. Similarly a 1-D curve may cut through a surface like scissors or zippers on a piece of clothes to split it into two parts. Unfortunately, simple interpretations don't hold too long for more complicated objects in mathematics: there are pathological space-filling curves that seem 1-D but behave more like a 2-D object, different parametrization of independent variables may exist, etc. So mathematicians have designed other notions of dimensions, like fractal metrics (Hausdorff dimension, Hurst exponent). And unluckily, some folks talk about "2-D curves" for circles or parabolas, which is wrong to me.

For a simple answer to your question, in data processing, we tend to avoid pathological behavior. Indeed we often deal with variables that are ordinal and relatively independent "naturally": one dimension of time or space directions are well-ordered. Traditional LTI systems often act as smoothers with mild conditions. And we often assimilate a deterministic signal with its graphical representation, taking its ordinal variables as "the geometric space that define the dimension". A (simple, single-valued) signal or time-series depending only on one time variable $$t$$ is therefore called mono-dimensional or 1-D. A multi-dimensional signal depending on $$D$$ variates would by $$D$$-dimensional. An image with a horizontal $$x$$ and a vertical $$y$$ direction would be 2-D, a video on $$(x,y,t)$$ 3-D etc.

Then a first confusion may arise because ordinal variates are discrete and of finite length or size: a sound signal of 44100 samples, a 10-megapixel camera could yield a $$2592 \times 3872$$ image. We sometimes loosely call $$44100$$, $$2592 \times 3872$$ or $$10036224$$ the "dimension" of the data. It means that "the space where all such data live" possesses that dimension. It is the space of "all signals" or "all images" of the given finite size, were the value at each sample or pixel can take arbitrary values (say in \mathbb{R}). In this, we move to the stochastic (non-deterministic) vision of signal processing, and also to the machine learning mindset, dealing with high-dimensionality. Therefore, a nicely deterministic signal of one dimension as a curve can also be thought of as a point in a 44100-D space of all the one-second sounds. No contradiction for me, only two facets on how to deal wit it.

Now, what happens with data $$d$$ which is multivalued? Let's take some (normally) 3-D data. At each point on the natural ordinal variables $$(m,n,p)$$ (respectively of length $$M$$, $$N$$, $$P$$), we get a set of $$K$$ values $$d(m,n,p) = \{v_1,v_2\ldots,v_K\}$$. The $$v$$s could be the left and right channels for audio ($$K=2$$), the RGB triplet ($$K=3$$) for color images. Generally, the size $$K$$ is either small with respect to $$(M,N,P)$$, or without specific ordering: the two left/right channels or the three blue-green-red colors are not interpreted as genuine ordinal variables. They are digitally represented by array indices $$[1,2]$$ or $$[1,2,3]$$ by shared convention (left is 1, right is 2) for their storage.

Meanwhile, in sensor array processing, as in imaging, it may happen that $$K$$ becomes larger. A seismic antenna may comprise fifty aligned sensors, generally close, providing a 50-valued vector $$\{v_1,v_2\ldots,v_{50}\}$$ at each location $$(x,y)$$. A multispectral or hyperspectral image may have about $$K=256$$ values in a spectral band, ordered in wave-number or frequency at each pixel. And the contiguity/continuity of the range, compared to the initial 3D grid, can be small enough that dimension blur. A single pixel of a 2-D image of a size $$M\times N$$ hyperspectral survey, when recorded a $$P=256$$ close consecutive time samples (time is the third natural dimension), may be looked at like a $$256\times 256$$ image of size $$P\times K$$.

To wrap it up, such situations, the original 3-D image can be considered, and processed as a novel 4-D signal, including the spectral axis as a new ordinal dimension.

Finally, you can meet 1.5-D or 2.5-D. It is a lazy indication that the data should be 1-D or 2-D, but the value space is multivalued with a non-negligible size, or the data can be treated with processing intermediate between signal and image one, or image and volume ones. Yet sometimes, the different dimensions are sufficiently different to not process them as a whole. In video, 2-D space and 1-D time are often treated a bit separately. Attempts to treat them as 3-D volumes for scalable compression for instance was not fully successful so far. For the record, we have been compressing so-called 3-D hexahedral meshes filled with multivariate properties, and due to the heterogeneity of the different dimensions, we have been driven to use 1-D, 2-D and 3-D methods (HexaShrink, an exact scalable framework for hexahedral meshes with attributes and discontinuities, 2019).

Depends what "dimension" means, but I'll say 2D, and interpret in context of convolutions.

A convolution operates on

• 1D: (channels, time)
• 2D: (channels, height, width),
• 3D: (channels, height, width, depth)

What they all have in common, is that the non-channels dimensions are spatial - so in general, (channels, *spatial). That is, adjacent values are

• (A) ordinal: where pixels are positioned relative to one another matters, unlike in channels (RGB = GRB)
• (B) uniformly spaced: pixels 1 and 2 are spaced by 5mm, and so are 2 and 3, and pixel 2 is "between" 1 and 3. There's no such thing for RGB: green is not the midpoint of red and blue, any more than grapefruit is between apple and orange.

Why convolutions? Because they're designed specifically to exploit spatial dependencies: a filter is shaped a certain way because it assumes that weighting and combining pixels a certain way, for all shifts over the image, will produce meaningful results. Different filters (weightings) play role similar to RGB: they are independent descriptors of the input.

distance between two objects can not determined properly

"Directly" is more apt over "properly", as in directly subtracting pixel values between two images won't yield physical distance, since an image is encoded in terms of color intensities irrespective of depth, but neural nets can transform them to latent spaces that provide distance measures.

It's also correct reasoning against images being spatially 3D, since if they were, then per (B), we'd just take Euclidean distance.

• Why take it to the deep learning domain?
– Mark
Commented Mar 6, 2022 at 7:25
• @Mark Just to convey that images contain the necessary information to measure distance, despite not being 3D. Commented Mar 7, 2022 at 16:52
• I actually like this analogy to explain the dimensions. Commented Mar 8, 2022 at 5:06

Color images have 2 (out of 3) spatial dimensions, thus they cannot represent a general scene spatially, only a projection onto 2d.

A color image does have a third dimension in its color representation that needs to be stored somehow, eg as a 3d array. If it was video, it would typically be stored as a 4d array, and one might argue that time was a fourth dimension. It would still be limited to 2 spatial dimensions, though.

Depth cameras are often described as «2.5d». Presumably because they contain some info about the third spatial dimension but not occluded areas.

• I think you mix between the digital representation (as in computer) to the math definition.
– Mark
Commented Mar 6, 2022 at 7:24
• It is not clear from the question which of the two the OT is interested in. I think that my post is quite clear in exploring the ways in which a color image may be said to be «2d» or «3d» with explicit mention of «spatial dimensions» as something different from digital storage? Commented Mar 6, 2022 at 7:41

(New reply to reflect changes in question) Digital computers generally does not «know» about dimensionality. When you store an array in Matlab, it is physically stored in memory and on disk as a serial, «1-dimensional» stream of numbers. Via side information, that stream of numbers can be interpreted as a 1d, 2d or n-d array (for any n). For certain operations to make sense (eg matrix multiply), the object needs to be interpretable in a specific way.

A 2 Megabyte object in memory may represent a 1920x1080 8 bit-per-pixel image. Or a snippet of monaural float audio. If you like, you can reinterpret an image as audio and listen to it (usually won’t sound pleasing).

A physical scene or object is whatever it is regardless of digital computers. My crude understanding is that the world is spatially 3-d, that time could be seen as a fourth (non-spatial) dimension, so too with color, polarization or any number of «extra» characteristics that could be useful. Perhaps more physics oriented people have a less «pragmatic» view?

It might be useful to know where you are going with this discussion. Is it about winning an argument with a friend? If so, I would say that «dimensionality» have different meanings in different contexts so you could probably both be right.

Images

Actually, images have 1 (one) dimension. At least in 99.5% of the cases that you encounter in the wild because most images are ultimately stored in a contiguous block of memory (called a buffer in C) that is read from using a pointer to the first element and an offset from this pointer.

All other concepts like its a vector-valued function in 2D, so 2D or, the MATLAB array has 3 dimensions so its 3D are abstractions. Sometimes they make it easy to solve a problem, so they are very useful and it is worth it to know as many of these as possible. However, if you dig into their gory internals, you will find that they all just convert the 2D or 3D index into a 1D offset that is added to a pointer.

The most consistent abstraction (I think) is to split the dimensions by kind and say: RGB images have 2 spatial dimensions and 1 color dimension or Grayscale images have 2 spatial dimensions (and 0 color dimensions). However, this will again break down when you consider esoteric image formats like raw/bayer (not sure if MATLAB natively supports these). Here, color is stored in a so called bayer filter mosaic. I'll let you decide how many dimensions this should have.

Video

Video follows the same line of thinking; however, we typically don't store the entire video as a contiguous block of memory (because we don't have enough RAM). Instead, most application (and especially MATLAB) consumes them frame-by-frame. So even if we could use the abstraction of saying a video has 3 (or 4) dimensions, it wouldn't be a very useful one.

In fact, treating time separately is particularly useful for video because you can't easily access different time points randomly. Again, some languages build abstractions to make you believe this is possible, but if you look into the gory guts you will find that such indexing involves resetting the video to some specific point in time (e.g. the beginning) and then reading frame-by-frame until we arrive at the desired index.

I guess the closest we can get is to say RGB videos have 2 spatial dimensions, 1 color dimension, and 1 time dimension. However, you can build almost arbitrary abstractions on top of this and your answer will change depending on the abstraction you choose.

(Edit: Also note that the vast majority of videos don't actually use RGB. Instead, they use a colorspace called YUV, because it is more efficient to compress and creates smaller files.)