Eyes; what are they good for?
A summary of 5 different task categories in which we rely fundamentally on vision and some observations on how I manage without any sight. There’s a brilliant but simple app called Be My Eyes described in this post, which despite its limitations, still surpasses what AI can do
So far this blog has not said much about Artificial Intelligence. Honestly that will change very soon. But it seems worthwhile spending a bit more time on the nature of sight and the challenges of not having any - i.e. the problem space where I'm hoping Artificial Intelligence will change my life and the life of hundreds of thousands of other blind people.
We can split the ways we use our eyes into 5 main categories:
- Tool usage (digital and physical)
- Object recognition
- Navigation
- Interaction
- Content assimilation
This categorisation is of my own making, and is not rigorous or exhaustive, but it is easily remembered with the acronym 'tonic'.
Amongst all our senses, the range at which sight can operate is extraordinary. Can you smell a 5 caret diamond at one metre, feel the breathtaking beauty of the Mona Lisa at 100 metres, taste the wonder of The Great Pyramids at 1,000 metres or hear the intensity of the sun at 93 million miles?
Not only does sight operate at an amazing range from near to far, but object recognition is completely integrated into our vision. We have names for almost all the objects we can see, but in contrast the names we use for smells, tastes, textures and sounds are often just generic rather than synonymous with the actual source of the sound, smell, taste or texture.
Here’s a summary of my own approach to object recognition without the benefit of sight.
- I live without it if it's not essential. I've largely given up asking people to describe works of art, or majestic scenery. And although I know my wife gets immense pleasure from our garden, my experience is largely limited to the intoxicating fragrance of the wisteria or the occasional cranial impact of the low hanging mulberry tree. The latter consistently delivers more sensory overload than the wisteria, albeit the sort of overload I could also happily live without
- In limited circumstances I use other senses for recognition. Smelling the contents of a jar mostly avoids dolloping strawberry jam on my prawn curry. Similarly feeling the shape of a container can often be sufficient. Having said that, ever since I confused depilatory cream with toothpaste many years ago, I have never been troubled by hirsute teeth.
- Very occasionally, I ask AI for a description, but much more frequently I ask a human for a description.
A subsequent post discusses why AI descriptions are still mostly unhelpful. In the meantime, object recognition is a brilliant use case for the only assistive technology I truly love: Be My Eyes. BME is a mobile app connecting 800,000 blind users with 9 million volunteers around the world (the second largest global volunteer pool, surpassed only by The Red Cross).
A volunteer answering my BME call is presented with the view through my phone's rear facing camera, or even better the view through the head mounted camera on my Meta Ray Bans. The volunteer can then assist me with object recognition but also with most of the other tasks in the tonic list. They can help me find the keys I just mislaid for the 3rd time today (object recognition), or read the microwave cooking instructions on my prawn curry (content assimilation) and actually operate the microwave (physical tool usage), get me through accessibility hell to complete an online order for an entire grocery basket (digital tool usage), or even guide me to the local supermarket to purchase the same basket of groceries the old fashioned way (navigation).
Be My Eyes is so game changing as an assistive technology that it provides much of the template against which I will measure assistive AI over the coming months and years.
One cautionary note; because BME relies on the goodwill and patience of human volunteers, the experience can undermine the agency and independence of the blind user, at worst leading to a sense of infantilisation. And believe me, at the age of 68, a sense of infantilisation due to an impending second childhood is already the stuff of nightmares, even without relying on BME volunteers to perform the simplest of day to day tasks.
But rather than focusing on the downsides of needing to be helped, I'm genuinely grateful to the huge pool of volunteers who give me a rush of joy every time we overcome an obstacle together, no matter how small or stupid it might seem. And if anyone would like to join my own personal Be My Eyes helper group then please do just comment here or DM @TheOpenEyedMan on X.