Aspects of the disclosure provide for generating distributions for hypothetical or potentially occluded objects. For instance, a location for which to generate one or more distributions may be identified. Observations of road users by perception systems of a plurality of autonomous vehicles may be accessed. Each of these observations may identify a characteristic of one of the road users. A distribution of the characteristic for the location may be determined based on the observations. The distribution may be provided to one or more autonomous vehicles in order to enable the one or more autonomous vehicles to use the distribution to generate a characteristic for a hypothetical occluded road user and to respond to the hypothetical occluded road user.
The described aspects and implementations enable efficient utilization of multiple machine learning (ML) models with autonomous vehicles (AVs) to quickly and efficiently determine trajectory information of objects in a driving environment. During a time interval, a perception system of an AV generates, using a first ML model, an embedding based on data that characterizes an object's trajectory in the driving environment. At each sub-interval of multiple sub-intervals during the time interval, a second ML model uses the embedding to generate a probability distribution for the trajectory of the object, and a planning system of the AV generates an update to the AV's trajectory based on the probability distribution.
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
The technology employs a holistic approach to passenger pickups and other wayfinding situations. This includes identifying where passengers are relative to the vehicle and/or the pickup location. Information synthesis from different sensors, agent behavior prediction models, and real-time situational awareness are employed to identify the likelihood that the passenger to be picked up is at a given location at a particular point in time, with sufficient confidence. The system can provide adaptive navigation by helping passengers understand their distance and direction to the vehicle, for instance using various cues via an app on the person's device. Rider support tools may be provided, which enable a remote agent to interact with a customer via that person's device, such as using the camera on the device to provide wayfinding support to enable the person to find their vehicle. Ride support may also use sensor information from the vehicle when providing wayfinding support.
This technology relates to a system for cooling sensor components. The cooling system may include a sensor which has a sensor housing, a motor, a main vent, and a side vent. Internal sensor components may be positioned within the sensor housing. The motor may be configured to rotate the sensor housing around an axis. The rotation of the sensor housing may pull air into an interior portion of the sensor housing through the main vent, and the air pulled into the interior portion of the sensor housing may be exhausted out of the interior portion of the sensor housing through the side vent.
An optical system comprises a light emitter, an optical waveguide, and a lens that optically couples the light emitter to the optical waveguide. The lens has a long axis and a plurality of surfaces surrounding the long axis, the plurality of surfaces including a convex surface and a flat surface. The convex surface faces the light emitter such that light emitted by the light emitter enters the lens through the convex surface. In some examples, the lens serves as a fast axis collimation (FAC) lens. In some examples, the flat surface serves as a mounting surface to mount the lens on a substrate such that the convex surface faces the light emitter and an output surface of the lens opposite the convex surface faces the optical waveguide. In some examples, the convex surface includes an anti-reflection coating.
A method is provided that includes receiving user input identifying a travel destination for a first vehicle, determining, by a processor, a first route for the first vehicle to follow, and configuring the first vehicle to follow the first route. The method further includes obtaining a model for a second vehicle that shares a road with the first vehicle and comparing model to a pre-determined template for a vehicle that is known to be a special purpose vehicle in order to determine whether the first template and the second template match. The method further includes determining, by the processor, a second route that leads to the travel destination, when a match is found to exist, and switching the first vehicle from following the first route to following the second route.
A camera module includes a housing with an opening and a portion that surrounds the opening, wherein the portion of the housing is transparent to near infrared (NIR) light. A fisheye lens is disposed within the opening such that a portion of the fisheye lens protrudes through the opening. An image sensor is disposed within the housing and optically coupled to the fisheye lens. The image sensor is sensitive to visible light and NIR light. A plurality of NIR light emitters is disposed within the housing. The NIR light emitters are configured to emit NIR light through the NIR-transparent portion of the housing. The NIR-transparent portion of the housing may include a light-diffusing structure, such as a pattern of microlenses formed on an inner surface of the NIR-transparent portion of the housing, to spread out the NIR light emitted by the NIR light emitters.
H04N 23/54 - Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
10.
Control Window Camera Direction to Avoid Saturation from Strong Background Light and Actively Adjust the Frame Time on the Spinning Camera to Achieve Directional Control
Example embodiments relate to taking images at certain predetermined angles in order to have consistent exposure throughout the images. An example embodiment includes a method. The method includes determining, using a lidar device, light intensity information of a surrounding environment of the lidar device. The light intensity information includes a plurality of angles within a threshold range of light exposure. The method also includes determining rotation times associated with each of the angles within the threshold range of light exposure. Further, the method includes based on the rotation times associated with each of the angles within the threshold range of light exposure, determining a plurality of target image times. In addition, the method includes capturing, by a camera system, a plurality of images at the plurality of target image times.
Example embodiments described herein involve a system for testing a light-emitting module. The light-emitting module may include a mounting platform configured to hold a light-emitting module for a camera. The mounting platform may also be configured to rotate. The system may further include a housing holding a plurality of photodiodes arranged in an array over at least a 90 degree arc of a hemisphere. The system may also include a controller configured to control the photodiodes and the rotation of the mounting platform.
B60K 35/21 - Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
B60K 35/80 - Arrangements for controlling instruments
B60K 35/90 - Calibration of instruments, e.g. setting initial or reference parametersTesting of instruments, e.g. detecting malfunction
B60Q 3/51 - Mounting arrangements for mounting lighting devices onto vehicle interior, e.g. onto ceiling or floor
G01J 1/00 - Photometry, e.g. photographic exposure meter
12.
Methods and Systems for Gradually Adjusting Vehicle Sensor Perspective using Remote Assistance
Example embodiments relate to gradually adjusting a vehicle sensor perspective using remote assistance. A computing device may receive a request for assistance from a vehicle operating in an environment. The request indicates that the vehicle is stopped at a location with a sensor perspective of the environment that is at least partially occluded. Responsive to receiving the request for assistance, the computing device may display a graphical user interface (GUI) that represents a current state of the vehicle and includes a selectable option configured to enable the vehicle to gradually move forward a predefined distance. The computing device may detect a selection of the selectable option and transmit instructions that enable the vehicle to gradually move forward the predefined distance. The vehicle can then gradually move forward the predefined distance while also monitoring for one or more changes in the environment responsive to receiving the instructions.
An optical receiver includes one or more photodetectors and an analog front end (AFE) configured to accept input signals from the one or more photodetectors. The AFE includes a non-linear gain amplifier (NLGA). The NLGA includes a piecewise linear gain stage configured to apply a piecewise linear transfer function to the input signals to form amplified signals. The AFE also includes a DC offset stage configured to apply a DC offset to the amplified signals. A related method of operation and vehicle are also disclosed.
Evaluating a simulation of an autonomous vehicle may be performed by using one or more processors to receive log data collected for a given area, generate environment data for the given area using the log data, run the set of simulations using an autonomous vehicle software, extract one or more metrics from the set of simulations, and evaluate the set of simulations using the one or more metrics. The set of simulations includes one or more of a selection simulation comprising a selection of a location related to the particular maneuver in the given area, a decision process simulation comprising a decision process playthrough for the particular maneuver in the given area, a maneuver simulation comprising a maneuver playthrough of the particular maneuver in the given area, and a replay simulation comprising a replay of the particular maneuver in a run from the log data.
G06F 30/27 - Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
Aspects of the disclosure provide a method of controlling an autonomous vehicle. The method may include receiving sensor data identifying a road user in a lane adjacent to the autonomous vehicle. A minimum magnitude of road user behavior for the road user that is likely to result in a collision with the autonomous vehicle. A collision cost may be determined based on the determined minimum magnitude of road user behavior and a probability density function of previously observed magnitudes of road user behavior. The autonomous vehicle may be controlled based on the collision cost.
B60W 30/095 - Predicting travel path or likelihood of collision
B60W 30/08 - Predicting or avoiding probable or impending collision
B60W 40/02 - Estimation or calculation of driving parameters for road vehicle drive control systems not related to the control of a particular sub-unit related to ambient conditions
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
16.
Displaying updated vehicle behavior in real time visualizations
Aspects of the technology employs snapshots, which are serialized sets of inputs for evaluating a components of a vehicle's autonomous driving system. Snapshots can be performed very quickly with minimal compile latency and runtime, allowing rapid evaluation of individual autonomous vehicle elements in isolation. Parameter variation and changes to initial conditions associated with a given component may be applied via a robust user interface (UI), enabling a user to perform permutations to a component or scenario and receive immediate feedback, e.g., graphically presented in a three-dimensional (3D) scene to understand how the vehicle is operating. The UI may include various control elements to enable real-time adjustment to the parameters or conditions of a snapshot. This can include one or more mutations of a snapshot to evaluate different situations or conditions, which can further aid in understanding how the vehicle would function in an autonomous driving mode.
The disclosed systems and techniques are directed to tracking and predicting live availability of parking resources by a fleet of vehicles. The disclosed techniques include receiving communications vehicles of a fleet with identification of object(s) located at an edge of a driving environment and edge visibility data for the edge from sensing systems of the vehicles, updating a map of live parking space occupancy for the driving environment with the received identification and the edge visibility data, determining, based on the updated map of live parking space occupancy and a historical parking space availability, a likelihood value associated with a parking space in the driving environment remaining unoccupied within a target time, and directing a vehicle of the fleet to the parking space based at least on the likelihood value.
Example embodiments relate to camera arrangements for vehicular object detection and avoidance. An example system includes a vehicle and at least one camera of a first camera type attached to the vehicle. The system also includes a plurality of cameras of a second camera type attached to the vehicle. Further, the system includes a plurality of cameras of a third camera type attached to the vehicle. Moreover, the system includes a computing device communicatively coupled to the at least one camera of the first camera type, plurality of cameras of the second camera type, and plurality of cameras of the third camera type. The computing device is configured to identify objects located within a first range of distances from the vehicle, objects located within a second range of distances from the vehicle, and objects located within a third range of distances from the vehicle.
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
G03B 11/00 - Filters or other obturators specially adapted for photographic purposes
H04N 23/698 - Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
H04N 23/73 - Circuitry for compensating brightness variation in the scene by influencing the exposure time
H04N 23/90 - Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
Example embodiments relate to reducing auto-exposure latency. An example embodiment includes a method of reducing auto-exposure latency. The method includes determining, by a processor, a first setting of an exposure parameter for a first frame to be captured by an image sensor. The first setting of the exposure parameter is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of the exposure parameter is determined during a first frame period associated with capturing the first frame. The method also includes initiating, by the processor, a first frame exposure operation based on the first setting of the exposure parameter. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.
A method for determining runout error includes generating a first magnetic field to interact with a second magnetic field generated by four or more poles of a magnet ring mounted to a first platform. The interaction may cause the first platform to rotate relative to a second platform. The method may further include receiving, from a magnetic field sensor, data including respective boundaries between neighboring poles of the four or more poles relative to a corresponding nominal boundary defined by substantially uniform boundary spacing. The method may include determining a magnetic field pattern from the data and, based on the pattern, determining an angular position of the four or more poles. The method may further include determining an angular difference between the determined angular position and a nominal angular position. The method may also include determining a runout error based on an amplitude of the angular difference.
H02K 11/215 - Magnetic effect devices, e.g. Hall-effect or magneto-resistive elements
G01D 5/14 - Mechanical means for transferring the output of a sensing memberMeans for converting the output of a sensing member to another variable where the form or nature of the sensing member does not constrain the means for convertingTransducers not specially adapted for a specific variable using electric or magnetic means influencing the magnitude of a current or voltage
G01D 18/00 - Testing or calibrating apparatus or arrangements provided for in groups
H02K 11/33 - Drive circuits, e.g. power electronics
The technology relates to determining whether a vehicle operating in an autonomous driving mode is experiencing an anomalous condition, for instance due to a loss of tire pressure, a mechanical failure, or a shift or loss of cargo. The actual current pose of the vehicle is compared to an expected pose of the vehicle, where the expected pose is based on a model of the vehicle. If a pose discrepancy is identified, the anomalous condition is determined from information associated with the pose discrepancy. The vehicle is then able to take corrective action based on the nature of the anomalous condition. The corrective action may include making a real-time driving change, modifying a planned route, alerting a remote operations center, or communicating with one or more other vehicles.
One example system comprises a LIDAR sensor that rotates about an axis to scan an environment of the LIDAR sensor. The system also comprises one or more cameras that detect external light originating from one or more external light sources. The one or more cameras together provide a plurality of rows of sensing elements. The rows of sensing elements are aligned with the axis of rotation of the LIDAR sensor. The system also comprises a controller that operates the one or more cameras to obtain a sequence of image pixel rows. A first image pixel row in the sequence is indicative of external light detected by a first row of sensing elements during a first exposure time period. A second image pixel row in the sequence is indicative of external light detected by a second row of sensing elements during a second exposure time period.
24.
GENERATING CANDIDATE PLANNED TRAJECTORIES AND PREDICTED FUTURE TRAJECTORIES USING A SHARED PREDICTION NEURAL NETWORK
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for planning the future trajectory of an autonomous vehicle in an environment.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
25.
TRAINING NEURAL NETWORKS FOR SENSOR DATA REPRESENTATION LEARNING THROUGH FUTURE FRAME PREDICTION
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining respective sensor data captured by one or more sensors of an autonomous vehicle at each of a sequence of time steps, the sequence of time steps comprising one or more context time steps followed by one or more prediction time steps; generating respective ground truth birds-eye-view (BEV) representations of the respective sensor data for each of the prediction time steps; for each prediction time step, processing the respective sensor data at one or more preceding time steps in the sequence using a future prediction neural network to generate a predicted BEV representation for the prediction time step; and training the future prediction neural network based on, for each prediction time step, an error between the ground truth BEV representation for the prediction time step and the predicted BEV representation for the prediction time step.
G01S 17/89 - Lidar systems, specially adapted for specific applications for mapping or imaging
G06T 7/246 - Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
G06T 7/73 - Determining position or orientation of objects or cameras using feature-based methods
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
G06V 10/774 - Generating sets of training patternsBootstrap methods, e.g. bagging or boosting
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
G06V 20/70 - Labelling scene content, e.g. deriving syntactic or semantic representations
26.
Methods and Systems for Estimating the Sensitivity of Vehicle Sensors
Examples relate to methods and systems for estimating radar sensitivity. A computing system uses radar data to detect objects located in a vehicle's environment. The system may then filter the radar data corresponding to the objects based on a set of predetermined criteria to identify a particular object that corresponds to a first type of object. The system may perform a comparison between a radar parameter determined based on radar data corresponding to the particular object and an expected radar parameter represented by a data model. The data model is generated based on radar data aggregated for a plurality of objects that match the first type of object. The system may then estimate radar sensitivity loss for the radar based on the comparison and adjust operation of the radar or the vehicle's control strategy when the estimated sensitivity loss exceeds a threshold loss.
Example embodiments relate to image sensors with on-sensor and application-specific object identification. An example embodiment includes a method. The method includes capturing, by an image sensor, image data about a surrounding environment. The method also includes receiving, by an analog-to-digital converter (ADC), the captured image data from the image sensor. The method further includes providing, by the ADC, converted image data to an application-specific integrated circuit (ASIC). The method further includes applying, by the ASIC, a trained machine-learning model to the converted image data to identify one or more objects in the surrounding environment within the converted image data. The method further includes outputting, by the ASIC, an image frame, wherein at least one row of the image frame comprises metadata including object classification data and object location data for the one or more identified objects in the surrounding environment.
G06V 10/94 - Hardware or software architectures specially adapted for image or video understanding
G01S 17/08 - Systems determining position data of a target for measuring distance only
G01S 17/89 - Lidar systems, specially adapted for specific applications for mapping or imaging
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
G06V 10/778 - Active pattern-learning, e.g. online learning of image or video features
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
09 - Scientific and electric apparatus and instruments
35 - Advertising and business services
37 - Construction and mining; installation and repair services
39 - Transport, packaging, storage and travel services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for computer-aided diagnostic testing services for vehicles and vehicle fleet management; downloadable and recorded computer software for operating vehicle sensors; downloadable and recorded computer software for detecting and issuing notifications for vehicle maintenance needs; downloadable and recorded computer software for facilitating and assisting with vehicle maintenance remotely; downloadable and recorded computer software for management of navigating, driving, and directing a vehicle to receive fuelling and servicing; recorded software for the autonomous driving of motor vehicles; downloadable software in the nature of vehicle operating system software; downloadable software for autonomous vehicle operation, navigation, steering, calibration, and management; downloadable software for vehicle fleet management, namely, tracking fleet vehicles for commercial purposes; downloadable computer software for use as an application programming interface (API); downloadable software for managing, monitoring, and optimizing vehicle fleet depot operations and quality control; downloadable and recorded software for vehicle fleet management and demand forecasting, vehicle charging, vehicle fuel monitoring, vehicle maintenance, vehicle depot management, vehicle parking management, and remote assistance with vehicles; recorded software for managing artificial intelligence (AI), machine learning, and deep learning; recorded software for artificial intelligence (AI), machine learning, and deep learning for data processing and contextual prediction, personalization, and predictive analytics; downloadable computer programs and downloadable software for artificial intelligence (AI), machine learning, and deep learning for use in connection with operating autonomous vehicles, systems, devices, and machinery; recorded software for use in connection with and for operating autonomous vehicles, systems, devices, and machinery; safety and driving assistant systems comprised of sensors for determining position, velocity, direction, and acceleration of land vehicles Business consultancy; business advice; business management; business planning; business administration; providing business support consulting services; business operations management relating to vehicles; tracking, locating, and monitoring of vehicles for commercial purposes; business data analysis; fleet management services in the nature of tracking, locating, and monitoring of fleet vehicles, namely, parking management, remote assistance management, vehicle fleet demand management in the nature of business tracking of fleet demand, and vehicle fleet depot business management, all for commercial purposes; business operation of vehicle depots; business operation of depots for autonomous vehicles Vehicle maintenance; vehicle maintenance and repair; fleet management services in the nature of maintenance of fleet vehicles; maintenance, servicing, and repair of autonomous vehicles; maintenance, servicing, and repair of vehicles; vehicle battery charging; charging of electric vehicles; charging stations for electric vehicles; Vehicle charging stations for electric autonomous vehicles; reservation of charging stations for electric vehicles Storage of vehicles; parking lot services; parking garages services; depot services for fleets of autonomous vehicles; depot services for fleets of vehicles Software as a service (SAAS) services featuring software for computer-aided diagnostic testing services for vehicles and vehicle fleet management; technical support services, namely, troubleshooting of computer software problems; development and establishment of testing specifications and procedures in the field of autonomous vehicles; development and establishment of testing specifications and procedures in the field of electric vehicles; development and establishment of testing specifications and procedures in the field of vehicle maintenance; providing online non-downloadable software for detecting and issuing notifications regarding vehicle maintenance needs; providing online non-downloadable software for facilitating and assisting with vehicle maintenance remotely; providing online non-downloadable software for navigating, driving, and directing a vehicle car to receive fuelling and servicing; providing online non-downloadable software for tracking, locating, and monitoring vehicles; providing online non-downloadable software for vehicle fleet management and demand forecasting, vehicle charging, vehicle fuel monitoring, vehicle maintenance, vehicle depot management, vehicle parking management, and remote assistance with vehicles; software as a service (SaaS) services featuring computer software for use as an application programming interface (API); providing online non-downloadable software for vehicle coordination, navigation, calibrating, direction, and management of vehicle on-board computers; software as a service (SaaS) services featuring software for vehicle coordination, navigation, calibrating, direction, and management of vehicle on-board computers; software as a service (SaaS) services for managing, monitoring, and optimizing vehicle fleet depot operations and quality control; electronic monitoring and reporting of transportation data using computers or sensors; providing online non-downloadable software for the autonomous driving of motor vehicles; providing online non-downloadable software for autonomous vehicle navigation, steering, calibration, and management; providing online non-downloadable software for visualization, manipulation, and integration of digital graphics and images; providing online non-downloadable software for managing artificial intelligence, machine learning, and deep learning; providing online non-downloadable software used for data analytics in the field of transportation; providing online non-downloadable software used for data analytics in the field of transportation fleet management; providing online non-downloadable open source software for use in data management; installation, updating, and maintenance of software for use with vehicle on-board computers for monitoring and controlling motor vehicle operation; providing virtual computer systems and virtual computer environments through cloud computing for the purpose of training and monitoring self-driving cars, autonomous vehicles and robots; virtual testing of the functions of self-driving cars, autonomous vehicles and robots using computer simulations; creation, development, programming and implementation of simulation software in the field of self-driving cars, autonomous vehicles and robots; creating simulation computer programs for autonomous vehicles
29.
PEDESTRIAN BEHAVIOR PREDICTION WITH 3D HUMAN KEYPOINTS
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent behavior prediction using keypoint data. One of the methods includes obtaining data characterizing a scene in an environment, the data comprising: (i) context data comprising data characterizing historical trajectories of a plurality of agents up to the current time point; and (ii) keypoint data for a target agent; processing the context data using a context data encoder neural network to generate a context embedding for the target agent; processing the keypoint data using a keypoint encoder neural network to generate a keypoint embedding for the target agent; generating a combined embedding for the target agent from the context embedding and the keypoint embedding; and processing the combined embedding using a decoder neural network to generate a behavior prediction output for the target agent that characterizes predicted behavior of the target agent after the current time point.
Example embodiments relate to a hybrid heater for use as a sensor window. An example embodiment includes a window that includes: (i) a base layer; (ii) a first heater that includes a substantially rectangular shaped layer of transparent conductive material disposed on a first region of the base layer such that passage of current through the first heater results in heating of the first region; and (iii) a second heater that includes a length of wire having a space-filling pattern that spans a second region of the base layer such that passage of current through the second heater results in heating of the second region.
H05B 3/84 - Heating arrangements specially adapted for transparent or reflecting areas, e.g. for demisting or de-icing windows, mirrors or vehicle windshields
The technology relates to generating simulations in order to evaluate software used to control vehicles in an autonomous driving mode. In one example, an initial situation involving a certain kind of interaction between a vehicle operating in the autonomous driving mode and an object may be identified. A search of log data may be conducted in order to identify one or more similar situations based on characteristics of the initial situation. A new simulation may be generated using the identified one or more similar situations by inserting the object into the one or more similar situations. The new simulation may be run in order to evaluate the software.
Example embodiments relate to a hybrid heater for use as a sensor window. An example embodiment includes a window that includes: (i) a base layer; (ii) a first heater that includes a substantially rectangular shaped layer of transparent conductive material disposed on a first region of the base layer such that passage of current through the first heater results in heating of the first region; and (iii) a second heater that includes a length of wire having a space-filling pattern that spans a second region of the base layer such that passage of current through the second heater results in heating of the second region.
H05B 3/86 - Heating arrangements specially adapted for transparent or reflecting areas, e.g. for demisting or de-icing windows, mirrors or vehicle windshields the heating conductors being embedded in the transparent or reflecting material
Example embodiments relate to multi-channel dynamic weather estimations. An example embodiment includes a method. The method includes capturing, using a camera, an image of a first field of view of a surrounding environment from a first perspective. The method also includes capturing, using a light detection and ranging (lidar) device, a point cloud of a second field of view of the surrounding environment from a second perspective. Additionally, the method includes aligning, by a computing device, the image with the point cloud. Further, the method includes identifying, by the computing device, one or more high-intensity regions and one or more low-intensity regions of the surrounding environment. In addition, the method includes determining, by the computing device, a first figure of merit that characterizes environmental conditions of the surrounding environment and a second figure of merit that characterizes environmental conditions of the surrounding environment.
Example embodiments relate to techniques for implementing radar waveform diversity. A technique may involve a radar unit transmitting radar signals into an environment of a vehicle based on a code sequence that indicates a ramp direction and a phase shift for each pulse in a pulse used by the radar unit and receiving radar reflections from the environment. In some instances, the radar unit may leverage antennas in a multiple input multiple output (MIMO) arrangement to further add diversity to transmissions according to spatial code in the code sequence. The technique can further involve using a demodulator to map the environment based on the radar reflections and controlling the vehicle based on the mapping. In some instances, the code sequence is received from a system that is wirelessly providing orthogonal code sequences to multiple emitters.
G01S 13/32 - Systems for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
G01S 7/02 - Details of systems according to groups , , of systems according to group
G01S 13/931 - Radar or analogous systems, specially adapted for specific applications for anti-collision purposes of land vehicles
35.
Methods and Systems for Calibrating Sensors Using Road Map Data
Example methods and systems for calibrating sensors using road map data are provided. An autonomous vehicle may use various vehicle sensors to assist in navigation. Within examples, the autonomous vehicle may calibrate vehicle sensors through performing a comparison or analysis between information about the environment received by sensors with similar information provided by map data (e.g., a road map). The autonomous vehicle may compare object locations as provided by the sensors and as shown by map data. Based on the comparison, the autonomous vehicle may adjust various sensors to accurately reflect the information as provided by the road map. In some instances, the autonomous vehicle may adjust the position, height, orientation, direction-of-focus, scaling, or other parameters of a sensor based on the information provided by a road map.
G01S 3/781 - Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using electromagnetic waves other than radio waves Details
G01S 5/16 - Position-fixing by co-ordinating two or more direction or position-line determinationsPosition-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
Aspects of the disclosure provide for depot behaviors for autonomous vehicles. For instance, a signal to control an autonomous vehicle to a depot area may be received from a server computing device. A prioritized list of staging areas within the depot area may be identified. Each staging area of the prioritized list of staging areas enables the vehicle to observe stopping locations at which a need of the vehicle may be addressed. The vehicle may be controlled to a first staging area of the prioritized list. Once the vehicle has reached the first staging area, whether a stopping location that meets one or more needs of the vehicle is available may be determined. When is available, the vehicle may be controlled to the available stopping location. When not available, the vehicle may be controlled to a second staging area of the prioritized list.
The technology relates to partially redundant equipment architectures for vehicles able to operate in an autonomous driving mode. Aspects of the technology employ fallback configurations, such as two or more fallback sensor configurations that provide some minimum amount of field of view (FOV) around the vehicle. For instance, different sensor arrangements are logically associated with different operating domains of the vehicle. Fallback configurations for computing resources and/or power resources are also provided. Each fallback configuration may have different reasons for being triggered, and may result in different types of fallback modes of operation. Triggering conditions may relate, e.g., to a type of failure, fault or other reduction in component capability, the current driving mode, environmental conditions in the vicinity of vehicle or along a planned route, or other factors. Fallback modes may involve altering a previously planned trajectory, altering vehicle speed, and/or altering a destination of the vehicle.
A light detection and ranging (lidar) device may be coupled to a vehicle and configured to scan a surrounding environment to determine ranges to one or more objects in the surrounding environment of the vehicle. The lidar device may generate data that can be used to form a range image, which includes or is based on range data determined for the one or more objects. The lidar device may also generate data that can be used to form a corresponding background image, which includes background light intensity data that the lidar device measures during the scan. The background image or background image data may be used to add range data to the range image, correct range data in the range image, and/or evaluate the quality of the range data in the range image. In this way, the background image or background image data can be used to generate an enhanced range image that includes range data that is more comprehensive and/or more reliable than the range data included in the original range image.
G01S 17/931 - Lidar systems, specially adapted for specific applications for anti-collision purposes of land vehicles
B60W 40/02 - Estimation or calculation of driving parameters for road vehicle drive control systems not related to the control of a particular sub-unit related to ambient conditions
G01S 17/18 - Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein range gates are used
G01S 17/42 - Simultaneous measurement of distance and other coordinates
G01S 17/89 - Lidar systems, specially adapted for specific applications for mapping or imaging
Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.
The disclosed systems and techniques are directed to identifying and responding to presence of vulnerable road users (VRUs) in driving environments that are at risk of loss of control of their driving trajectories. The techniques include collecting, using a sensing system of an autonomous vehicle, sensing data for an environment of the autonomous vehicle and processing the sensing data by one or more machine learning models to identify a plurality of reference points associated with a VRU in the environment. The techniques further include identifying one or more height differentials for the plurality of reference points, determining that the VRU is at risk of loss of control, based at least on a change of the one or more height differentials, and causing a control system of the autonomous vehicle to perform an avoidance action.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sensor data for a vehicle to perform prediction tasks regarding a driving environment of the vehicle. In one aspect, a method comprises: receiving sensor data comprising an observation of a driving environment, processing the observation using an observation embedding neural network to generate an observation embedding comprising respective observation features associated with each of a plurality of spatial locations within the observation, receiving data characterizing a prediction task, receiving a region proposal specifying a spatial region of the observation, and generating output prediction data characterizing an output prediction for the prediction task and for the region proposal by (i) processing the observation and the region proposal to generate region features characterizing the spatial region and (ii) processing the region features and the data characterizing the prediction task to generate the output prediction data.
B60W 50/00 - Details of control systems for road vehicle drive control not related to the control of a particular sub-unit
G05B 13/02 - Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
42.
CONTRASTIVE LEARNING FOR ENCODING SELF-DRIVING SENSOR DATA
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an observation encoding system to generate observation encodings representing observations of sensor data characterizing an environment of a vehicle. In one aspect, a method comprises: receiving sensor data comprising an observation for a first sensor modality for a vehicle; processing the sensor data using an encoder neural network for the first sensor modality to generate an embedding representing the observation, wherein the encoder neural network for the first sensor modality has been trained using a modality alignment loss function that measures an agreement between (i) embeddings representing observations for the first sensor modality and (ii) embeddings representing text descriptions generated by processing observations for a second sensor modality; and processing an input comprising the embedding representing the observation using a prediction neural network for the first sensor modality to generate a prediction for the vehicle.
Aspects of the disclosure provide a method of providing a destination to an autonomous vehicle in order to enable the autonomous vehicle to collect data according to a targeted driving goal. For instance, a current location of an autonomous vehicle may be received. A set of destinations may be selected from a plurality of predetermined destinations. A route may be determined for each destination. A relevance score may be determined for each destination based on the determined routes and the targeted driving goal. Each destination may be assigned to one of a set of two or more buckets based on the relevance scores. A destination of the set may be selected based on a predetermined sampling probability. The selected destination is sent to the autonomous vehicle in order to cause the autonomous vehicle to travel to the selected destination in an autonomous driving mode.
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
G05D 1/246 - Arrangements for determining position or orientation using environment maps, e.g. simultaneous localisation and mapping [SLAM]
G05D 1/43 - Control of position or course in two dimensions
G05D 101/15 - Details of software or hardware architectures used for the control of position using artificial intelligence [AI] techniques using machine learning, e.g. neural networks
G05D 111/50 - Internal signals, i.e. from sensors located in the vehicle, e.g. from compasses or angular sensors
44.
ENHANCING SCENE PREDICTIONS FOR AUTONOMOUS DRIVING WITH MULTIMODAL LANGUAGE MODELS
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a prediction task using sensor data. The method includes obtaining scene data characterizing a scene in an environment at a current time point, wherein the scene comprises an autonomous vehicle and a plurality of agents, wherein the scene data comprises sensor data captured by one or more sensors of the autonomous vehicle and scene context data; generating, from the sensor data using a multimodal language model (MLM) neural network, one or more text outputs that each describe one or more aspects of the scene; generating, from at least the one or more text outputs describing the one or more aspects of the scene and the scene context data, a prediction input to a prediction neural network; and processing the prediction input using the prediction neural network to generate a prediction output for the prediction task.
The technology employs a variable motion control envelope that enables an on-board computing system of a self-driving vehicle to estimate future vehicle driving behavior along an upcoming path, in order to maintain a desired amount of control during autonomous driving. Factors including intrinsic vehicle properties, extrinsic environmental influences and road friction information are evaluated. Such factors can be evaluated to derive an available acceleration model, which defines an envelope of maximum longitudinal and lateral accelerations for the vehicle. This model, which may identify dynamically varying acceleration limits that can be affected by road conditions and road configurations, may be used by the on-board control system (e.g., a planner module of the processing system) to control driving operations of the vehicle in an autonomous driving mode.
The present disclosure relates to light detection and ranging (lidar) systems, lidar-equipped vehicles, and associated methods. An example method includes causing a firing circuit to trigger emission of an initial group of detection pulses from at least one light-emitter device of a lidar system in accordance with an initial set of one or more light-emission parameters. The method also includes causing the firing circuit to trigger emission of one or more test pulses and receiving, from at least one detector, information indicative of one or more return test pulses. The method yet further includes determining, based on the received information, a presence of a retroreflector based on an intensity of the return test pulse. The method additionally includes determining a subsequent set of light-emission parameters and causing the firing circuit to trigger emission of a subsequent group of detection pulses in accordance with the subsequent set of light-emission parameters.
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sensor data characterizing an environment of a vehicle to generate predictions regarding the environment of the vehicle. In one aspect, a method comprises receiving sensor data comprising one or more observations of a driving environment of a vehicle, receiving a query regarding the driving environment of the vehicle, processing the received sensor data and the received query to generate a network input comprising a plurality of input tokens, and processing the network input using a token processing neural network to generate an output token sequence that represents a response to the received query regarding the driving environment.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
The technology relates to using on-board sensor data, off-board information and a deep learning model to classify road wetness and/or to perform a regression analysis on road wetness based on a set of input information. Such information includes on-board and/or off-board signals obtained from one or more sources including on-board perception sensors, other on-board modules, external weather measurement, external weather services, etc. The ground truth includes measurements of water film thickness and/or ice coverage on road surfaces. The ground truth, on-board and off-board signals are used to build the model. The constructed model can be deployed in autonomous vehicles for classifying/regressing the road wetness with on-board and/or off-board signals as the input, without referring to the ground truth. The model can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
G05D 1/247 - Arrangements for determining position or orientation using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
G05D 1/249 - Arrangements for determining position or orientation using signals provided by artificial sources external to the vehicle, e.g. navigation beacons from positioning sensors located off-board the vehicle, e.g. from cameras
The described aspects and implementations enable vehicle light classification in autonomous vehicle (AV) applications. In one implementation, disclosed is a method and a system to perform the method that includes, obtaining, by a processing device, first image data characterizing a driving environment of an autonomous vehicle (AV). The processing device may identify, based on the image data, a vehicle within the driving environment. The processing device may process the image data using one or more trained machine-learning models (MLMs) to determine a state of one or more lights of the vehicle and cause an update to a driving path of the AV based on the determined state of the lights.
G06V 10/22 - Image preprocessing by selection of a specific region containing or referencing a patternLocating or processing of specific regions to guide the detection or recognition
G06V 10/40 - Extraction of image or video features
09 - Scientific and electric apparatus and instruments
35 - Advertising and business services
37 - Construction and mining; installation and repair services
39 - Transport, packaging, storage and travel services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Downloadable software for computer-aided diagnostic testing services for vehicles and vehicle fleet management; downloadable and recorded computer software for operating vehicle sensors; downloadable and recorded computer software for detecting and issuing notifications for vehicle maintenance needs; downloadable and recorded computer software for facilitating and assisting with vehicle maintenance remotely; downloadable and recorded computer software for management of navigating, driving, and directing a vehicle to receive fuelling and servicing; recorded software for the autonomous driving of motor vehicles; downloadable software in the nature of vehicle operating system software; downloadable software for autonomous vehicle operation, navigation, steering, calibration, and management; downloadable software for vehicle fleet management, namely, tracking fleet vehicles for commercial purposes; downloadable computer software for use as an application programming interface (API); downloadable software for managing, monitoring, and optimizing vehicle fleet depot operations and quality control; downloadable and recorded software for vehicle fleet management and demand forecasting, vehicle charging, vehicle fuel monitoring, vehicle maintenance, vehicle depot management, vehicle parking management, and remote assistance with vehicles; recorded software for managing artificial intelligence (AI), machine learning, and deep learning; recorded software for artificial intelligence (AI), machine learning, and deep learning for data processing and contextual prediction, personalization, and predictive analytics; downloadable computer programs and downloadable software for artificial intelligence (AI), machine learning, and deep learning for use in connection with operating autonomous vehicles, systems, devices, and machinery; recorded software for use in connection with and for operating autonomous vehicles, systems, devices, and machinery; safety and driving assistant systems comprised of sensors for determining position, velocity, direction, and acceleration of land vehicles Business consultancy; business advice; business management; business planning; business administration; providing business support consulting services; business operations management relating to vehicles; tracking, locating, and monitoring of vehicles for commercial purposes; business data analysis; fleet management services in the nature of tracking, locating, and monitoring of fleet vehicles, namely, parking management, remote assistance management, vehicle fleet demand management in the nature of business tracking of fleet demand, and vehicle fleet depot business management, all for commercial purposes; business operation of vehicle depots; business operation of depots for autonomous vehicles Vehicle maintenance; vehicle maintenance and repair; fleet management services in the nature of maintenance of fleet vehicles; maintenance, servicing, and repair of autonomous vehicles; maintenance, servicing, and repair of vehicles; vehicle battery charging; charging of electric vehicles; charging stations for electric vehicles; Vehicle charging stations for electric autonomous vehicles; reservation of charging stations for electric vehicles Storage of vehicles; parking lot services; parking garages services; depot services for fleets of autonomous vehicles; depot services for fleets of vehicles Software as a service (SAAS) services featuring software for computer-aided diagnostic testing services for vehicles and vehicle fleet management; technical support services, namely, troubleshooting of computer software problems; development and establishment of testing specifications and procedures in the field of autonomous vehicles; development and establishment of testing specifications and procedures in the field of electric vehicles; development and establishment of testing specifications and procedures in the field of vehicle maintenance; providing online non-downloadable software for detecting and issuing notifications regarding vehicle maintenance needs; providing online non-downloadable software for facilitating and assisting with vehicle maintenance remotely; providing online non-downloadable software for navigating, driving, and directing a vehicle car to receive fuelling and servicing; providing online non-downloadable software for tracking, locating, and monitoring vehicles; providing online non-downloadable software for vehicle fleet management and demand forecasting, vehicle charging, vehicle fuel monitoring, vehicle maintenance, vehicle depot management, vehicle parking management, and remote assistance with vehicles; software as a service (SaaS) services featuring computer software for use as an application programming interface (API); providing online non-downloadable software for vehicle coordination, navigation, calibrating, direction, and management of vehicle on-board computers; software as a service (SaaS) services featuring software for vehicle coordination, navigation, calibrating, direction, and management of vehicle on-board computers; software as a service (SaaS) services for managing, monitoring, and optimizing vehicle fleet depot operations and quality control; electronic monitoring and reporting of transportation data using computers or sensors; providing online non-downloadable software for the autonomous driving of motor vehicles; providing online non-downloadable software for autonomous vehicle navigation, steering, calibration, and management; providing online non-downloadable software for visualization, manipulation, and integration of digital graphics and images; providing online non-downloadable software for managing artificial intelligence, machine learning, and deep learning; providing online non-downloadable software used for data analytics in the field of transportation; providing online non-downloadable software used for data analytics in the field of transportation fleet management; providing online non-downloadable open source software for use in data management; installation, updating, and maintenance of software for use with vehicle on-board computers for monitoring and controlling motor vehicle operation; providing virtual computer systems and virtual computer environments through cloud computing for the purpose of training and monitoring self-driving cars, autonomous vehicles and robots; virtual testing of the functions of self-driving cars, autonomous vehicles and robots using computer simulations; creation, development, programming and implementation of simulation software in the field of self-driving cars, autonomous vehicles and robots; creating simulation computer programs for autonomous vehicles
51.
PROBABILISTIC PREDICTION OF OCCLUDED PEDESTRIANS AND OTHER ANIMATE OBJECTS IN AUTOMOTIVE ENVIRONMENTS
The disclosed systems and techniques are directed to identifying and responding to presence of target objects in occluded areas of driving environments. The techniques include training, using perception data associated with a first driving scene, a first machine learning model (MLM) to determine a location, within the first driving scene, of a target object masked with a masking transformation. The techniques further include training, using an output of the first MLM for a training driving scene, a second MLM to generate a map of probabilities of one or more target objects to be in an occluded region of the training driving scene, the training driving scene comprising at least one of the first driving scene or a second driving scene, and causing the second MLM to be deployed on an autonomous vehicle.
The subject matter of this specification can be implemented in, among other things, a system that includes a first light source to produce a pulsed beam and a second light source to produce a continuous beam, a modulator to impart a modulation to the second beam, and an optical interface subsystem to transmit the pulsed beam and the continuous beam to an outside environment and to detect a plurality of signals reflected from the outside environment. The system further includes one or more circuits configured to identify associations of various reflected pulsed signals, used to detect distance to various objects in the environment, with correct reflected continuous signals, used to detect velocities of the objects. The one or more circuits identify the associations based on the modulation of the detected continuous signals.
The described aspects and implementations enable reduced use in computational resources by an autonomous vehicle (AV) by predicting third-party collisions for an AV. A method includes obtaining object indications for objects in a driving environment of an AV. An object indication may include a shape definition and one or more predicted future locations of a corresponding object. The method includes projecting each shape definition onto each predicted future location of the one or more predicted future locations of the corresponding object. The method includes determining, based on projected shape definitions of the objects, that the projected shape definitions of at least two objects overlap. The method includes, responsive to the overlap between at least one of the projected shape definitions meeting a collision criterion, modifying the operation of the AV to avoid an area of the overlapped projected shape definitions of the least two objects.
The disclosure describes a system that includes a self-driving system for operating a vehicle autonomously, one or more optical transmitters mounted on the vehicle, and one or more computing devices in communication with the self-driving system and the one or more optical transmitters. The one or more computing devices are configured to operate the self-driving system to cause the vehicle to approach a designated location in proximity of a structure on which one or more receivers are mounted and determine that the one or more optical transmitters have an alignment with the one or more receivers. Then, the one or more computing devices are configured to operate the one or more optical transmitters to establish an optical communication link with the one or more receivers and transmit data to the one or more receivers over the optical communication link.
Disclosed are systems and methods that can be used for adjusting the field of view of one or more sensors of an autonomous vehicle. In the systems and methods, each sensor of the one or more sensors is configured to operate in accordance with a field of view volume up to a maximum field of view volume. The systems and methods include determining an operating environment of an autonomous vehicle. The systems and methods also include based on the determined operating environment of the autonomous vehicle, adjusting a field of view volume of at least one sensor of the one or more sensors from a first field of view volume to an adjusted field of view volume different from the first field of view volume. Additionally, the systems and methods include controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume.
G01S 13/86 - Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
G01S 13/931 - Radar or analogous systems, specially adapted for specific applications for anti-collision purposes of land vehicles
G01S 17/86 - Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
G01S 17/89 - Lidar systems, specially adapted for specific applications for mapping or imaging
G01S 17/931 - Lidar systems, specially adapted for specific applications for anti-collision purposes of land vehicles
G01W 1/06 - Instruments for indicating weather conditions by measuring two or more variables, e.g. humidity, pressure, temperature, cloud cover or wind speed giving a combined indication of weather conditions
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
G05D 1/243 - Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals
G05D 1/247 - Arrangements for determining position or orientation using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
G05D 1/81 - Handing over between on-board automatic and on-board manual control
G06V 10/44 - Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersectionsConnectivity analysis, e.g. of connected components
G06V 10/88 - Image or video recognition using optical means, e.g. reference filters, holographic masks, frequency domain filters or spatial domain filters
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
G08G 1/01 - Detecting movement of traffic to be counted or controlled
G08G 1/04 - Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
G08G 1/048 - Detecting movement of traffic to be counted or controlled with provision for compensation of environmental or other condition, e.g. snow, vehicle stopped at detector
H04N 23/69 - Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
57.
Vehicle Sensor Modules with External Audio Receivers
Example embodiments relate to vehicle sensor modules with external audio receivers. An example sensor module may include sensors and can be coupled to a vehicle's roof with a first microphone positioned proximate to the front of the sensor module. The sensor module can also include a second microphone extending into a first side of the sensor module such that the second microphone is configured to detect audio originating from an environment located relative to a first side of the vehicle and a third microphone extending into a second side of the sensor module such that the third microphone is configured to detect audio originating from the environment located relative to a second side of the vehicle, wherein the second side is opposite of the first side.
A system includes a memory device and a processing device, operatively coupled to the memory device, to identify one or more erroneous predictions of behavior for one or more objects in an environment of an autonomous vehicle traveling along a planned trajectory, and initiate, based on the one or more erroneous predictions, one or more operations to adjust the planned trajectory of the autonomous vehicle. Each erroneous prediction of the one or more erroneous predictions is determined based on a comparison between a corresponding observed spatial overlap between the autonomous vehicle and a corresponding object, and a corresponding predicted spatial overlap between the autonomous vehicle and the corresponding object.
The described aspects and implementations include a method for using transformers to generate maps for use by AVs. The method includes generating an input embedding based, at least in part, on sensing data from a sensing system of the AV; selecting one or more transformer decoder queries directing a transformer decoder to a particular portion of the input embedding; generating, using the one or more transformer decoder queries and the input embedding as input to the transformer decoder, one or more driving environment embeddings for a navigation system of the AV, and each driving environment embedding comprises a vector representation of a feature of the driving environment; providing the one or more driving environment embeddings to the navigation system of the AV, wherein the navigation system is configured to navigate the AV in the driving environment based, at least in part, on the one or more driving environment embeddings.
Example embodiments relate to GNSS time synchronization in redundant systems. A redundant system configured with two subsystems may initially synchronize clocks from both subsystems to GNSS time from a GNSS receiver. The synchronization of the first subsystem's clock may involve using a first communication link that enables communication between the first subsystem and the GNSS receiver while the synchronization of the second subsystem's clock may involve using both the first communication link and a second communication link that enables communication between the subsystems. The redundant system may then synchronize the first subsystem's clock to the second subsystem's clock while the second subsystem's clock is still synchronized to GNSS time from the GNSS receiver based on timepulses traversing a pair of wires that connect the subsystems and the GNSS receiver.
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for modeling human driving behavior. One of the methods includes continually computing, at each time step, a respective observation deviation value for a current driving policy of an agent. An accumulated observation deviation value is computed including accumulating observation deviation values computed for each of a plurality of time steps. If an accumulated observation deviation value satisfies a threshold, a different policy is selected for the agent to execute after the accumulated observation deviation value satisfies the threshold.
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
65.
Systems and Methods for a Tone Mapper Solution for an Autonomous Driving System
A system is provided that includes an image sensor coupled to a vehicle, and control circuitry configured to perform operations including receiving, from the image sensor, an input stream comprising high dynamic range (HDR) image data associated with an environment of the vehicle, and processing the input stream at the vehicle by applying a global tone mapping, followed by offline image processing that can include applying a local tone mapping to the globally tone mapped images of the same input stream.
Example embodiments relate to temperature-based dynamic frequency scaling to enable high-performance automotive silicon design. An example embodiment includes a system that includes an integrated circuit, a temperature sensor, and a controller. The controller may be configured to perform operations, including receiving, from the temperature sensor, a communication indicating a temperature of the integrated circuit, determining whether the temperature of the integrated circuit is outside a predefined range of temperatures, and, in response to determining that the temperature of the integrated circuit is outside the predefined range of temperatures, adjusting an operational parameter of the integrated circuit.
A simulation may be used to determine a difference between progress of a manually-driven vehicle and progress of a simulated autonomous vehicle. The method includes retrieving log data collected for the manually-driven vehicle driving along a route, generating a plurality of path segments for a portion of the route. The plurality of path segments corresponds to points in a lane that the manually-driven vehicle traveled through on the portion of the route. The method also includes running, using a software of the autonomous vehicle, a simulation of the autonomous vehicle driving along the plurality of path segments, extracting metrics from the log data and the simulation, and determining the difference between a first progress of the manually-driven vehicle and a second progress of the simulated autonomous vehicle based on the metrics.
G01C 22/00 - Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers or using pedometers
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
68.
Temperature-Based Dynamic Frequency Scaling to Enable High-Performance Automotive Silicon Design
Example embodiments relate to temperature-based dynamic frequency scaling to enable high-performance automotive silicon design. An example embodiment includes a system that includes an integrated circuit, a temperature sensor, and a controller. The controller may be configured to perform operations, including receiving, from the temperature sensor, a communication indicating a temperature of the integrated circuit, determining whether the temperature of the integrated circuit is outside a predefined range of temperatures, and, in response to determining that the temperature of the integrated circuit is outside the predefined range of temperatures, adjusting an operational parameter of the integrated circuit.
One example system includes a light source that emits light. The system also includes a waveguide that guides the emitted light from a first side of the waveguide toward a second side of the waveguide opposite the first side. The waveguide has a third side extending between the first side and the second side. The system also includes a mirror that reflects the guided light toward the third side of the waveguide. At least a portion of the reflected light propagates out of the waveguide toward a scene. The system also includes a light detector, and a lens that focuses light from the scene toward the waveguide and the light detector.
The technology involves operation of a self-driving truck or other cargo vehicle when it is being inspected at a weigh station. This may include determining whether a weigh station is open for inspection. Once at the weigh station, the vehicle may follow instructions of an inspection officer or autonomous inspection system. The vehicle may perform predefined actions or operations so that various vehicle systems and safety issues can be evaluated, such as the brakes, lights, tires, connections between the tractor and trailer, exposed fuel tanks, leaks, etc. A visual inspection may be performed to ensure the load is secured, vehicle and cargo documents meet certain criteria, and the carrier's safety record meets any requirements. In addition, the weigh station itself may be operated in a partly or fully autonomous mode when dealing with autonomous and manually driven vehicles.
G07C 5/08 - Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle, or waiting time
B64U 101/26 - UAVs specially adapted for particular uses or applications for manufacturing or servicing for manufacturing, inspections or repairs
B64U 101/30 - UAVs specially adapted for particular uses or applications for imaging, photography or videography
G01G 19/02 - Weighing apparatus or methods adapted for special purposes not provided for in groups for weighing wheeled or rolling bodies, e.g. vehicles
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
G07C 5/00 - Registering or indicating the working of vehicles
G07C 5/02 - Registering or indicating driving, working, idle, or waiting time only
A method includes obtaining, from an image sensor, one or more images that represent an object, and determining a speed of the object based on the one or more images. The method also includes determining that the speed of the object exceeds a threshold speed and, based on determining that the speed of the object exceeds the threshold speed, determining a region of interest (ROI) of the image sensor expected to represent the object. The method further includes causing the image sensor to generate one or more ROI images using the ROI.
A system includes a memory device, and a processing device, operatively coupled to the memory device, to receive a set of input data including a roadgraph and a distribution of a plurality of scene types associated with a driving context, the roadgraph including an autonomous vehicle driving path, perform stochastic sampling based on the distribution of the plurality of scene types to obtain a set of synthetic scenes for the driving context, and train a machine learning model to identify, for each synthetic scene of the set of synthetic scenes, a respective modified autonomous vehicle driving path.
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing perception tasks on received sensor data. The method includes obtaining one or more query images and a plurality of context images; generating a sequence of discrete tokens representing the context images; generating one or more continuous tokens representing the one or more query images; processing an input comprising the sequence of discrete tokens representing the context images and the one or more continuous tokens representing the one or more query images using a token processing neural network to generate one or more updated continuous tokens representing the one or more query images; and processing the one or more updated continuous tokens to generate a respective output for each of one or more prediction tasks.
G06V 10/82 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
G06V 10/26 - Segmentation of patterns in the image fieldCutting or merging of image elements to establish the pattern region, e.g. clustering-based techniquesDetection of occlusion
G06V 10/40 - Extraction of image or video features
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
74.
Display screen or portion thereof with graphical user interface
Aspects of the disclosure relate to detecting and responding to malfunctioning traffic signals for a vehicle having an autonomous driving mode. For instance, information identifying a detected state of a traffic signal for an intersection. An anomaly for the traffic signal may be detected based on the detected state and prestored information about expected states of the traffic signal. The vehicle may be controlled in the autonomous driving mode based on the detected anomaly.
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
G06V 40/10 - Human or animal bodies, e.g. vehicle occupants or pedestriansBody parts, e.g. hands
Example systems and methods enable an autonomous vehicle to request assistance from a remote operator in certain predetermined situations. One example method includes determining a representation of an environment of an autonomous vehicle based on sensor data of the environment. Based on the representation, the method may also include identifying a situation from a predetermined set of situations for which the autonomous vehicle will request remote assistance. The method may further include sending a request for assistance to a remote assistor, the request including the representation of the environment and the identified situation. The method may additionally include receiving a response from the remote assistor indicating an autonomous operation. The method may also include causing the autonomous vehicle to perform the autonomous operation.
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
B60W 30/00 - Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
77.
Determining the pick-up/drop-off state of a mass transit vehicle
Systems and methods for determining a passenger pick-up/drop-off (PUDO) state of a mass transit vehicle by an autonomous vehicle (AV) are disclosed. A system includes a memory one or more processing devices, coupled to the memory, configured to perform operations that include identifying a mass transit vehicle in an environment of an AV, responsive to determining the mass transit vehicle is in a PUDO state, providing movement instructions to a planning system of the AV, and autonomously modifying operation of the AV based on the movement instructions.
Aspects of the disclosure relate to an autonomous vehicle that may detected other nearby vehicles and identify them as parked or unparked. This identification may be based on visual indicia displayed by the detected vehicles as well as traffic control factors relating to the detected vehicles. Detected vehicles that are in a known parking spot may automatically be identified as parked. In addition, detected vehicles that satisfy conditions that are indications of being parked may also be identified as parked. The autonomous vehicle may then base its control strategy on whether or not a vehicle has been identified as parked or not.
G05D 1/00 - Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
G06V 20/58 - Recognition of moving objects or obstacles, e.g. vehicles or pedestriansRecognition of traffic objects, e.g. traffic signs, traffic lights or roads
G08G 1/015 - Detecting movement of traffic to be counted or controlled with provision for distinguishing between motor cars and cycles
G08G 1/052 - Detecting movement of traffic to be counted or controlled with provision for determining speed or overspeed
G08G 1/0962 - Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
G08G 1/0967 - Systems involving transmission of highway information, e.g. weather, speed limits
An apparatus includes an optical emitter configured to emit a first optical signal along an optical path towards a target object in an outdoor environment, and an optical detector positioned collinearly to the optical emitter with respect to an optical axis corresponding to the optical emitter. The optical detector is configured to detect a second optical signal that is retro-reflected from the target object.
Aspects of the disclosure may enable an autonomous vehicle to perform evasive maneuvers in order to avoid potential collisions. For instance, a vehicle may be controlled in an autonomous driving mode using a first trajectory generated by a first planning system. Information identifying a characteristic of a road user in an environment of the vehicle may be received and used to determine whether to generate a new behavior prediction for the road user or to use a prior determined behavior prediction. While using the first trajectory, a second planning system may be used to generate a second trajectory based on the determination. The second trajectory may be compared to the first trajectory. The vehicle may be controlled in the autonomous driving mode based on a result of the comparing.
Aspects of the disclosure provide for determining whether a vehicle is in a loop for an autonomous vehicle. For instance, a route from a current location of the vehicle to a destination for a trip may be generated. Locations traversed by the vehicle during the trip may be tracked. Locations of the route may be compared to the tracked locations to determine an overlap value. Whether the vehicle is in a loop may be determined based on the overlap value.
Example embodiments relate to a lower power linearization of lidar signals. An example embodiment includes a method that includes receiving a sample value output by an analog-to-digital converter (ADC) in a processing unit of a lidar system. The ADC may be configured to digitize an optical signal that is compressed by a gain amplifier. The compression may be based on a transfer function comprising one or more linear portions. The method also includes comparing the sample value to one or more threshold values. The one or more threshold values may correspond respectively to the one or more linear portions. The method further includes selecting, for the sample value and based on the comparing, an inverse gain and an associated intercept. The method additionally includes linearizing the sample value based on the selected inverse gain and the associated intercept.
Example embodiments described herein involve a system including a high throughput signal transmitter adjacent to a charging port of an autonomous vehicle configured to transmit information to a high throughput signal receiver. The high throughput signal receiver may be positioned on an electrical charging apparatus. Further, the high throughput signal transmitter and the high throughput signal receiver may be in point to point communication. Finally, the high throughput signal transmitter and the high throughput signal receiver may be separated by a distance up to and including 1.5 meters.
B60L 53/66 - Data transfer between charging stations and vehicles
B60L 58/12 - Methods or circuit arrangements for monitoring or controlling batteries or fuel cells, specially adapted for electric vehicles for monitoring or controlling batteries responding to state of charge [SoC]
B60W 60/00 - Drive control systems specially adapted for autonomous road vehicles
H02J 7/00 - Circuit arrangements for charging or depolarising batteries or for supplying loads from batteries
H04B 7/06 - Diversity systemsMulti-antenna systems, i.e. transmission or reception using multiple antennas using two or more spaced independent antennas at the transmitting station
H04L 43/0876 - Network utilisation, e.g. volume of load or congestion level
84.
AMBIENT LIGHTING CONDITIONS FOR AUTONOMOUS VEHICLES
The disclosure relates to using ambient lighting conditions with passenger and goods pickups and drop offs with autonomous vehicles. For instance, a map of ambient lighting conditions for stopping locations may be generated by receiving ambient lighting condition data for predetermined stopping locations and arranging this data into a plurality of buckets based on time and one of the stopping locations. A vehicle may then be controlled in an autonomous driving mode in order to stop for a passenger by both observing ambient lighting conditions for different stopping locations and, in some instances, also using the map.
Examples relate to near-field radar filters that can enhance measurements near a radar unit. An example may involve receiving a first set of radar reflection signals at a radar unit coupled to a vehicle and determining a filter configured to offset near-field effects of radar reflection signals received at the radar unit. In some instances, the filter depends on an azimuth angle and a distance for surfaces in the environment causing the first set of radar reflection signals. The example may also involve receiving, at the radar unit, a second set of radar reflection signals and determining, using the filter, an azimuth angle and a distance for surfaces in the environment causing the second set of radar reflection signals. The vehicle may be controlled based in part on the azimuth angle and the distance for the surfaces causing the second plurality of radar reflection signals.
The disclosed systems and techniques facilitate efficient prediction of crowd behavior and safe and courteous navigation of crowded areas in driving environments. An example disclosed system includes a sensing system and a data processing system of a vehicle. The sensing system obtains sensing data associated with a driving environment of the vehicle. The data processing system detects, based on the sensing data, presence of vulnerable road users (VRUs) in the driving environment. The data processing system applies one or more clustering metrics to form cluster(s) of VRUs, each cluster associated with a geometric shape enclosing one or more VRUs and a velocity associated with collective motion of these VRUs. The data processing system predicts, using the geometric shapes and the associated velocities, one or more VRU-blocked regions and determine a driving path of the vehicle in the driving environment.
Example embodiments relate to techniques for enabling one or more systems of a vehicle (e.g., an autonomous vehicle) to request remote assistance to help the vehicle navigate in an environment. A computing device may be configured to receive a request for assistance from a vehicle to classify an object and to initiate display of a graphical user interface at a display device. The graphical user interface may be configured to visually represent the object and may comprise one or more graphical user interface elements to enable input to be provided for classifying the object. The computing device may also be configured to generate a response that includes a classification of the object based on detecting a selection of at least one of the one or more graphical user interface elements. Further, the computing device may be configured to transmit the response to the vehicle.
G06V 20/56 - Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
B60W 30/09 - Taking automatic action to avoid collision, e.g. braking and steering
G06V 10/764 - Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing sensor data characterizing an environment of a vehicle to generate predictions regarding the environment of the vehicle. In one aspect, a method comprises obtaining training data comprising a plurality of training examples, wherein each training example comprises (i) example sensor data comprising one or more observations of a driving environment of an example vehicle for the training example, (ii) an example query for the training example, and (iii) a target prediction for the training example; processing the example sensor data and the example query for each training example to generate a respective network input comprising a plurality of input tokens for each training example; and training a token processing neural network to optimize a likelihood of the token processing neural network generating the target predictions for the training examples by processing the corresponding network inputs.
The present disclosure relates to systems and methods that involve a rotatable mirror assembly. An example system includes a shaft configured to rotate about a rotational axis. A portion of the shaft can be tapered, either in a stepped profile or a ramped profile. In the surface of the shaft can be one or more circumferentially-indented grooves. The system also includes a mirror body coupled to the tapered shaft via the at least one circumferentially-indented groove. The mirror body is configured to have a polygonal cross-section, and reflective surfaces can be disposed along at least one external surface of the mirror body. Example methods of manufacturing the rotatable mirror assembly are also disclosed.
G02B 26/08 - Optical devices or arrangements for the control of light using movable or deformable optical elements for controlling the direction of light
G01S 7/481 - Constructional features, e.g. arrangements of optical elements
G01S 17/931 - Lidar systems, specially adapted for specific applications for anti-collision purposes of land vehicles
95.
PRIVACY-RESPECTING DETECTION AND LOCALIZATION OF SOUNDS IN AUTONOMOUS DRIVING APPLICATIONS
The described aspects and implementations enable privacy-respecting detection, separation, and localization of sounds in vehicle environments. The techniques include obtaining, using audio detector(s) of a vehicle, a sound recording that includes a plurality of elemental sounds (ESs) in a driving environment of the vehicle, and processing, using a sound separation model, the sound recording to separate individual ESs of the plurality of ESs. The techniques further include identifying a content of individual ESs and causing a driving path of the vehicle to be modified in view of the identified content of the individual ESs. Further techniques include rendering speech imperceptibly by redacting temporal portions of the speech, using sound recognition models to identify and discard recordings of speech, and driving at speeds that exceed threshold speeds at which speech becomes imperceptible from noise masking.
G10L 15/20 - Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise or of stress induced speech
G10L 15/22 - Procedures used during a speech recognition process, e.g. man-machine dialog
G10L 25/51 - Speech or voice analysis techniques not restricted to a single one of groups specially adapted for particular use for comparison or discrimination
12 - Land, air and water vehicles; parts of land vehicles
Goods & Services
Vehicles; apparatus for locomotion by land, air or water; autonomous vehicles; automobiles; automobiles and structural parts therefor; autonomous automobiles; cars; autonomous cars; self-driving cars; self-driving transport vehicles; land vehicles; land vehicles and structural parts therefor; autonomous land vehicles; motor vehicles, namely automobiles, vans, and structural parts therefor; vans; autonomous vans; vehicle parts; parts for land vehicles.
Example embodiments relate to foveated imagers for automotive applications. An example embodiment includes a device. The device includes a rotationally symmetric foveated lens. The rotationally symmetric foveated lens is configured to receive light from an environment. The rotationally symmetric foveated lens is also configured to produce an image at an image plane based on the received light. The device also includes an image sensor having an associated image sensor resolution. The image sensor is positioned at the image plane and configured to capture an image having an associated field of view of the environment. Based on a distortion profile of the rotationally symmetric foveated lens and the image sensor resolution, the captured image exhibits a first angular optical resolution in a central region of the field of view and a second angular optical resolution in a peripheral region of the field of view.
An example lidar system includes a housing defining an interior space. The housing includes at least one optical window. The lidar system also includes a rotatable mirror assembly disposed within the interior space. The rotatable mirror assembly includes a transmit mirror portion and a receive mirror portion. The lidar system additionally includes a transmitter disposed within the interior space. The transmitter is configured to emit emission light into an environment of the lidar system along a transmit path. The lidar system also includes a receiver disposed within the interior space. The receiver is configured to detect return light that is received from the environment along a receive path. The lidar system additionally includes at least one optical baffle configured to minimize stray light in the interior space.
12 - Land, air and water vehicles; parts of land vehicles
Goods & Services
(1) Vehicles; apparatus for locomotion by land, air or water; autonomous vehicles; automobiles; automobiles and structural parts therefor; autonomous automobiles; cars; autonomous cars; self-driving cars; self-driving transport vehicles; land vehicles; land vehicles and structural parts therefor; autonomous land vehicles; motor vehicles, namely automobiles, vans, and structural parts therefor; vans; autonomous vans; vehicle parts; parts for land vehicles.
100.
Method of Temperature Conditioning Compute Module for Cold Start
Example embodiments relate to methods of increasing a temperature of a computer module to start the computer at environmental temperatures below a threshold temperature. An example embodiment includes receiving, at one or more computing components thermally coupled to a main computer via a liquid-cooled plate, a set of program instructions. The method can also include running the set of program instructions on at least one computing component. Running the set of program instructions on the computing component can generate heat that flows to the main computer via the liquid-cooled plate. The method can additionally include detecting, from at least one thermal sensor coupled to the liquid-cooled plate, a temperature reading indicative of a temperature of the main computer. The method can further include determining that the temperature reading has reached a predetermined temperature threshold and based on the temperature reading reaching the predetermined temperature threshold, powering on the main computer.
