FSD Using Cameras in Rain??

damnitjim

Well-known member
First Name
Bones
Joined
Oct 21, 2020
Threads
6
Messages
138
Reaction score
230
Location
Massachusetts
Vehicles
Audi Q7 S-Line, Audi Q5 PHEV S-Line, CT3 Pre-Order
Occupation
AV Field Engineer
Country flag
So I’ve been wondering. If Tesla only uses cameras for FSD and not LiDAR, how will it work/react in the rain? My 2021 Audi Q5 rear camera looks like this in the rain:
A3C6EAAD-5ECC-43EF-A484-D99C71AF6A35.jpeg

 
OP
OP
damnitjim

damnitjim

Well-known member
First Name
Bones
Joined
Oct 21, 2020
Threads
6
Messages
138
Reaction score
230
Location
Massachusetts
Vehicles
Audi Q7 S-Line, Audi Q5 PHEV S-Line, CT3 Pre-Order
Occupation
AV Field Engineer
Country flag
Tesla's camera is tucked away to avoid fouling. Mine on my Mazda is right above the license plate (tho I need to put it on the fin) and it doesn't ever get that bad x-x

-Crissa
That's good. I have never driven a Tesla in the rain, only a Model X once for a short time on a sunny day. But water does distort images. I'm just curious how the tech tackles that variable.
 

Crissa

Well-known member
First Name
Crissa
Joined
Jul 8, 2020
Threads
96
Messages
11,271
Reaction score
18,478
Location
Santa Cruz
Vehicles
2014 Zero S, 2013 Mazda 3
Country flag
That's good. I have never driven a Tesla in the rain, only a Model X once for a short time on a sunny day. But water does distort images. I'm just curious how the tech tackles that variable.
It's a good question! And I really wish we would get that question at AI day. Unskewing images is a massive undertaking.

-Crissa
 


Rootinboy

Active member
First Name
Bryce
Joined
Dec 11, 2019
Threads
0
Messages
41
Reaction score
48
Location
Atlanta
Vehicles
Toyota Sequoia
Country flag
So I’ve been wondering. If Tesla only uses cameras for FSD and not LiDAR, how will it work/react in the rain? My 2021 Audi Q5 rear camera looks like this in the rain:
A3C6EAAD-5ECC-43EF-A484-D99C71AF6A35.jpeg
I drove a model y in the rain using fsd. No different than a clear sunny day...
 

Bill906

Well-known member
Joined
Mar 21, 2020
Threads
3
Messages
1,095
Reaction score
2,482
Location
Wisconsin
Vehicles
Jeep
Country flag
So I’ve been wondering. If Tesla only uses cameras for FSD and not LiDAR, how will it work/react in the rain? My 2021 Audi Q5 rear camera looks like this in the rain:
A3C6EAAD-5ECC-43EF-A484-D99C71AF6A35.jpeg
FSD will drive in the rain using cameras just like humans with eyes drive in the rain. The main forward looking cameras are behind the windshield and the wipers clear the rain off the windshield.
 

slomobile

Well-known member
First Name
Dustin
Joined
Apr 7, 2022
Threads
2
Messages
108
Reaction score
104
Location
Memphis
Vehicles
Cybertruck
Occupation
Roboticist
FSD will drive in the rain using cameras just like humans with eyes drive in the rain.
I've had to pull over and seek shelter under an overpass in severe rain with hail due to lack of visibility. Same in snowstorm whiteout conditions. It makes you a sitting duck target for anyone 'brave enough' (dumb enough) to keep doing 70mph unable to see their hood ornament.

I would pay extra for a radar sensor that keeps me moving slowly in those conditions and avoids hitting or being hit by big things. Autonomously moving over to avoid being hit from behind could avoid some of the most costly massive pileups. Insurance companies ought to recognize that. Perhaps Tesla insurance will look into that when Tesla density results in 2 or more covered vehicles per pileup.

If you are in manual drive mode and your Tesla suddenly changes lanes a split second before a semi barrels through where you just were, would that be acceptable if the cost were a rare unexpected lane change for no apparent reason?

Its an interesting problem because the operational design domain exceeds human capabilities. It requires the vehicle to actively override human inputs and substitute sensors and AI judgement over clear human commands in the moment.
It is not unprecedented. Antilock brakes do the same thing. Stabbing the brake pedal to the floor is a clear human intent, and it produces a sickening feeling as the pedal shudders and groans while you roll slowly into an active icy intersection. It is the feeling of your car betraying you. And it is considered completely ok.

Any light camera can be defeated by a kid with a spray can regardless of how many layers of glass it hides behind. It would be nice to have at least one defeat resistant sensor even if it's vote is ignored most of the time for being generally less reliable.
 
Last edited:


Bill906

Well-known member
Joined
Mar 21, 2020
Threads
3
Messages
1,095
Reaction score
2,482
Location
Wisconsin
Vehicles
Jeep
Country flag
I've had to pull over and seek shelter under an overpass in severe rain with hail due to lack of visibility. Same in snowstorm whiteout conditions. It makes you a sitting duck target for anyone 'brave enough' (dumb enough) to keep doing 70mph unable to see their hood ornament.

I would pay extra for a radar sensor that keeps me moving slowly in those conditions and avoids hitting or being hit by big things.

Any light camera can be defeated by a kid with a spray can regardless of how many layers of glass it hides behind. It would be nice to have at least one defeat resistant sensor even if it's vote is ignored most of the time for being generally less reliable.
Agreed. That all makes sense in theory.

Question. Camera doesn't see anything in front of you. Radar sees a large object. Do you trust the camera and drive through it, or trust the radar and hit the brakes?
 
Last edited:

slomobile

Well-known member
First Name
Dustin
Joined
Apr 7, 2022
Threads
2
Messages
108
Reaction score
104
Location
Memphis
Vehicles
Cybertruck
Occupation
Roboticist
Question. Camera doesn't see anything in front of you. Radar sees a larger object. Do you trust the camera and drive through it, or trust the radar and hit the brakes?
Take conservative action to narrowly avoid the worst case scenario. If you've done everything below and still cant decide, apply the brakes.

You don't trust any sensor fully, they are given variable weight based on trends. If a large portion of camera image has been white or black over the last second, minute, hour then add more weight to radar. If camera images are resulting in high rates of object classification, reduce weight to radar. It is a busy well lit environment. On the highway where car thought an overturned truck was a billboard, classification rate would have been low, and image white, so radar weight would have been high, possibly braking to avoid the incorrectly classified object.

You gather more opinions to influence the decision by alerting driver, check for brake pedal trends, look for corresponding spikes in magnetometer, check if all cameras agree, check for patterns of false radar alerts across the network in this location, or at this time of day, or at this temperature, and weigh them all.

Yes, this has caused undesirable behavior. Yes it is more computationally costly. It is an acceptable cost to me. A cost I chose to bear when I enabled radar. I can opt out of the cost by disabling radar.
I've heard the argument that they require independent models to develop. Hogwash. Just develop for radar sensors allowing 0% weight given to disabled or non existent radar.
 

cyberboi

Member
First Name
jeff
Joined
Oct 19, 2020
Threads
0
Messages
13
Reaction score
12
Location
california
Vehicles
turbo
Country flag
Take conservative action to narrowly avoid the worst case scenario. If you've done everything below and still cant decide, apply the brakes.

You don't trust any sensor fully, they are given variable weight based on trends. If a large portion of camera image has been white or black over the last second, minute, hour then add more weight to radar. If camera images are resulting in high rates of object classification, reduce weight to radar. It is a busy well lit environment. On the highway where car thought an overturned truck was a billboard, classification rate would have been low, and image white, so radar weight would have been high, possibly braking to avoid the incorrectly classified object.

You gather more opinions to influence the decision by alerting driver, check for brake pedal trends, look for corresponding spikes in magnetometer, check if all cameras agree, check for patterns of false radar alerts across the network in this location, or at this time of day, or at this temperature, and weigh them all.

Yes, this has caused undesirable behavior. Yes it is more computationally costly. It is an acceptable cost to me. A cost I chose to bear when I enabled radar. I can opt out of the cost by disabling radar.
I've heard the argument that they require independent models to develop. Hogwash. Just develop for radar sensors allowing 0% weight given to disabled or non existent radar.
I agree. I think the main reason Tesla doesn’t want to include radar is cost.
 

slomobile

Well-known member
First Name
Dustin
Joined
Apr 7, 2022
Threads
2
Messages
108
Reaction score
104
Location
Memphis
Vehicles
Cybertruck
Occupation
Roboticist
That's why they eliminated radar.

Because it couldn't tell the difference between a sign and a truck across the road.

So no, you don't want radar to have you creep along. You're still a sitting duck.

-Crissa
In sudden zero visibility it doesn't need to tell the difference. It just needs to go 15mph to reduce closing speed from the idiot behind me and maintain steering ability, stay between the centerline and shoulder best it can according to GPS, and dont hit anything big. Sign or truck, I don't want to hit either of them and I don't care if my car knows what the blip was.

 

 
CYBERBACKPACK
Top