The disclosure describes systems, devices, and methods for automating the reintegration of a failed virtual storage block into a file system service upon recovery of the virtual storage block. In an example embodiment, a method performed by a storage controller in a file system service is provided. In performing the method, the storage controller determines that a virtual storage volume of an array of virtual storage volumes has become unavailable, and in response to determining that the virtual storage volume has become unavailable, pauses use of the virtual storage volume and monitors for the virtual storage volume to return to an available state. Upon the virtual storage volume returning to the available state, the storage controller tests multiple portions of the virtual storage volume. After successfully testing the multiple portions of the virtual storage volume, the storage controller commences with the use of the virtual storage volume.
Systems and methods for scaling application and/or storage system functions of a distributed storage system based on a heterogeneous resource pool are provided. According to one embodiment, the distributed storage system has a composable, service-based architecture that provides scalability, resiliency, and load balancing. The distributed storage system includes a cluster of nodes each potentially having differing capabilities in terms of processing, memory, and/or storage. The distributed storage system takes advantage of different types of nodes by selectively instating appropriate services (e.g., file and volume services and/or block and storage management services) on the nodes based on their respective capabilities. Furthermore, disaggregation of these services, facilitated by interposing a frictionless layer (e.g., in the form of one or more globally accessible logical disks) therebetween, enables independent and on-demand scaling of either or both of application and storage system functions within the cluster while making use of the heterogeneous resource pool.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
3.
Persisting Mutex Lock Data To Enhance Failover Handling By Storage Controllers
Improved failover handling by persisting lease lock data to storage. Storage controllers of a data storage system handle client I/O requests for files or portions of files in a storage aggregate of the system. The controllers obtain lease locks on behalf of clients for the I/O requests and record lock data in the storage aggregate. If a first controller of the system fails over to a second controller, a grace period is triggered. The second controller gets the lock data from storage and processes an I/O request received from a client during the grace period. From the data, the second controller determines if the first storage controller held the lock corresponding to the I/O request. If the first storage controller held the lock, the second controller immediately executes the request. Where the first storage controller did not hold the lock, the second controller obtains the lock to execute the request.
G06F 9/52 - Program synchronisationMutual exclusion, e.g. by means of semaphores
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Techniques and systems for enhanced storage system management and failure detection/mitigation are presented. In one example, a method includes, responsive to detection of a trigger event for a first host node monitored by at least a second host node, initiating fencing of the first host node by at least transferring a fencing notification to a controller module for a set of storage drives assigned to the first host node. The method also includes preventing the first host node from communicating with the set of storage drives, and alerting a client node communicating with the first host node that the second host node is handling communication with the set of storage drives.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
5.
Dynamic Balancing Between Stability And Cost Optimization In Compute Clusters
The disclosure describes a node management service that dynamically adjusts resource allocations for workloads in a compute cluster by balancing cost and stability considerations. The node management service implements a disruption budget that allows for a predetermined level of disruptions of applications within the workload. As the disruption level of the workload approaches the disruption budget, the node management service initiates disruption reduction actions that lower the likelihood of further disruptions.
OP logs are utilized for fast recovery of synchronous replication in a storage system. According to one embodiment, a computer-implemented method comprises establishing synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site, initiating a peer health check to determine a health condition of the primary storage site and concurrently initiating a create snapshot process to create a snapshot of one or more storage objects of the CG2 of the secondary storage site in response to detection of missing heartbeat communications from the primary storage site, and performing inflight tracking replay and reconciliation between a first Op log of the primary storage site and a second Op log of the secondary storage site when an out of sync state or failover occurs.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
7.
Managing Parity Operations Using Controller-Specific Parity Bitmaps
The disclosure describes systems, devices, and methods for tracking operations of controllers in a data storage environment on a per-controller basis. In an implementation, a method for re-assigning responsibility of individual controller-owned parity bitmap sections and re-performing an incomplete operation is provided. In the method, a controller detects a failure of a first controller and responsively assign responsibility for a second of a parity bitmap previously assigned to the first controller to the second controller. The controller directs the second controller to read the assigned section of the parity bitmap. The controller further directs the second controller to, for a status indicator in the section indicating incomplete, compute new parity data based on source data associated with the status indicator, store the new parity data in the section, and update the status indicator from incomplete to complete.
In one embodiment, a method comprises providing a first dynamically extensible file system (DEFS) of a first node and a second DEFS of a second node of a storage cluster having a disaggregated storage space within a storage pod, detecting, with the first DEFS of the first node, blocks and associated physical volume block numbers (PVBNs) to be freed, determining whether the first DEFS owns or does not own an allocation area (AA) having the PVBNs to be freed, and transferring non-owned PVBNs to be freed to a remote free log of the first DEFS when the AA having the PVBNs to be freed is not owned or assigned to the first DEFS.
Systems, methods, and software are disclosed herein relating to processing telemetry data of a storage environment for anomaly detection in various implementations. In an implementation, a computing apparatus detects a sequence of missing data in telemetry data associated with a user of a data storage environment and verifies the user was active during a time period of the sequence of missing data. Upon verification, the computing apparatus generates synthetic data to replace the sequence of missing data; the synthetic data is based on activity data of users similar to the given user. The computing apparatus generates augmented data with the telemetry data and the synthetic data and incorporates the augmented data into a historical user activity profile. The computing apparatus detects anomalous behavior in the storage environment based on a comparison of new telemetry data with the historical user activity profile and takes corrective action associated with the anomalous behavior.
The technology disclosed herein enhances data protection in a distributed storage system. In a particular example, a method includes determining a drive in a subject node of the distributed storage system has failed while storing first data of a data set distributed across nodes of the distributed storage system by a data protection mechanism. The method further includes broadcasting failure information indicating the data set from the subject node to other nodes of the distributed storage system. At the other nodes, in response to receiving the failure information, the method includes identifying a subset of the other nodes that also store a portion of the data set. In each identified node of the subset, the method includes identifying second data of the data set stored on a local drive and copying the second data to a different local drive to protect the data set from further drive failure.
With a forever incremental snapshot configuration and a typical caching policy (e.g., least recently used), a storage appliance may evict stable data blocks of an older snapshot, perhaps unchanged data blocks of the snapshot baseline. If stable data blocks have been evicted, restore of a recent snapshot will suffer the time penalty of downloading the stable blocks for restoring the recent snapshot. Creating synthetic baseline snapshots and refreshing eviction data of stable data blocks can avoid eviction of stable data blocks and reduce the risk of violating a recovery time objective.
Systems and methods for sharing a namespace of an ephemeral storage device by multiple consumers are provided. In an example, a storage driver (e.g., a non-volatile memory express (NVMe) driver) of a virtual storage system deployed within a compute instance of a cloud environment facilitates sharing of the namespace by exposing an application programming interface (AP)I through which the multiple consumers access an ephemeral storage device associated with the compute instance. During initialization processing performed by each consumer, for example, during boot processing of the virtual storage system, the consumers may share the namespace by reserving for their own use respective partitions within the namespace via the API and thereafter restrict their usage of the namespace to their respective partitions.
Systems and methods for identifying anomalous activities in a cloud computing environment are provided. According to one embodiment, a customer's infrastructure may be fortified by leveraging deep learning technology (e.g., an encoder-decoder machine-learning (ML) model) to predict events in the cloud environment. During a training phase, the ML model may be trained to make a prediction regarding a next event based on a predetermined or configurable length of a sequence of contextual events. For example, historical events (e.g., cloud application programming interface (API) events logged to a cloud activity trace) observed within the customer's cloud infrastructure over the course of a particular date range may be split into appropriate event/context pairs and fed to the ML model. Subsequently, during a run-time anomaly detection phase, the ML model may be used to predict a next event based on a sequence of immediately preceding events to facilitate identification of anomalous activity.
H04L 41/16 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
Systems and methods for enhancing API security by identifying anomalous activities in a cloud environment are provided. In one example, the lack of awareness of an external API with respect to how calls to the external API may affect a cluster of a container orchestration platform is addressed. For instance, the views of the external and internal APIs may be combined to achieve better API security by correlating external API calls with undesirable behavior or other anomalies arising in the internal API. Responsive to identifying such undesirable behavior, information (e.g., a host, a source IP, a user, a specific payload) associated with the offending external API call may be added to a network security feature (e.g., a deny list, an IPS, or a WAF) utilized by the external API to facilitate performance of enhanced filtering of subsequent external API calls by the external API on behalf of the internal API.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
A method, a computing device, and a non-transitory machine-readable medium for classifying documents. A document collection is sorted into a plurality of categories. A classifier corresponding to a category of the plurality of categories is trained to output a probability that a document associated with the category is of a selected type (e.g., confidential). The training includes determining, by the processor, that a cardinality of a set of negative samples in a train set is not above a pipeline threshold but is at least one and training the classifier via a first pipeline and a second pipeline using a training group that includes a first portion of a group of positive samples in the train set, a second portion of a set of negative samples in the train set, and a third portion of a group of unlabeled samples in the train set
G06F 18/214 - Generating training patternsBootstrap methods, e.g. bagging or boosting
G06F 18/23213 - Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
G06F 18/2415 - Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
Systems and methods are disclosed for implementing a process for graph database storage optimization, applicable to delta-based cloud asset tracking. In certain embodiments, a method may comprise implementing a delta-based graph storage optimization system for asset tracking in a cloud environment, including storing a graph database representing a configuration of a cloud environment, obtaining configuration settings representing a current state of the cloud environment from a cloud platform, and identifying a delta based on changes between the configuration from the graph database and the configuration settings from the cloud platform. The method may further comprise creating an asset property node based on the delta, and adding the asset property node to the graph database without creating a new graph based on the configuration settings.
The disclosure describes a node management service that dynamically adjusts node capacity in orchestration platforms. The service leverages an extended resource definition to reflect the increased resource needs during the startup phase of workloads, then reallocates these resources once the workload transitions to steady-state operation. Workloads are provisioned with both a native resource request and an extended resource request to cover initialization spikes. Once initialization completes, the node management service updates the node's metadata to increase available extended resource capacity, thereby freeing up headroom for future deployments.
Systems and methods for selectively bypassing an external cache (EC) of a storage system are provided. In one example, when the EC backing storage device is saturated, reads bypass the EC and are completed via a redundant array of independent disks (RAID) subsystem of the storage system. One or more performance metrics for the EC backing storage device may be monitored to predict one or more saturation thresholds (e.g., in terms of latency and/or throughput). Based on this monitoring, tuning may be performed to drive utilization of the EC into a “knee region” of a performance (or response) curve of the EC backing storage device. For example, the depth of one or more request queues at the front-end of an EC lookup may be manipulated based on a current measure of saturation relating to the EC backing storage device to limit the number of in-flight reads pending for the EC.
Systems and methods for selectively bypassing an external cache (EC) of a storage system are provided. In one example, when the EC backing storage device is saturated, reads bypass the EC and are completed via a redundant array of independent disks (RAID) subsystem of the storage system. One or more performance metrics for the EC backing storage device may be monitored to predict one or more saturation thresholds (e.g., in terms of latency and/or throughput). Based on this monitoring, tuning may be performed to drive utilization of the EC into a “knee region” of a performance (or response) curve of the EC backing storage device. For example, the depth of one or more request queues at the front-end of an EC lookup may be manipulated based on a current measure of saturation relating to the EC backing storage device to limit the number of in-flight reads pending for the EC.
Systems and methods for performing journal swapping are provided. In one example, the backing storage used for virtual non-volatile random access memory (vNVRAM) is dynamically switched based on the high-availability (HA) state of an HA pair of nodes of a virtual storage system. By default, in-memory logging may be used for write journaling when the HA pair is operating in a normal HA state. When the HA pair is in an HA degraded state, the write journaling may be performed to local ephemeral storage. This allows the file system of the virtual storage system to run in a more performant configuration most of the time (e.g., during which HA is enabled and healthy). At the same time, the file system has the ability to swap to a less performant but more resilient configuration during planned events (e.g., scheduled maintenance and throughput scaling) in which HA is in a degraded state.
Systems and methods for performing journal swapping are provided. In one example, the backing storage used for virtual non-volatile random access memory (vNVRAM) is dynamically switched based on the high-availability (HA) state of an HA pair of nodes of a virtual storage system. By default, in-memory logging may be used for write journaling when the HA pair is operating in a normal HA state. When the HA pair is in an HA degraded state, the write journaling may be performed to local ephemeral storage. This allows the file system of the virtual storage system to run in a more performant configuration most of the time (e.g., during which HA is enabled and healthy). At the same time, the file system has the ability to swap to a less performant but more resilient configuration during planned events (e.g., scheduled maintenance and throughput scaling) in which HA is in a degraded state.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
23.
Methods and Systems for Automated Detection of Personal Information Using Neural Networks
A method, a computing device, and a non-transitory machine-readable medium for detecting personal information. Terms that are of interest are extracted from a corpus of raw text that has been extracted from a collection of documents. For each of the terms, a surrounding sentence is extracted to form a target sentence to thereby form a plurality of target sentences. The surrounding sentence includes at least one reference to a data subject. A matrix of feature information is generated for each of the target sentences to form a plurality of matrices. A neural network model is trained, using the matrices as input, to compute an output that indicates a likelihood of a given sentence containing personal information.
According to an example, a computer-implemented method comprises initiating a first process for atomically setting the primary bias state with a first node of a primary storage cluster of a multi-site distributed storage system due to a temporary loss of connectivity to a mediator or a temporary mediator failure, releasing an atomic lock for the first process on the first node of the primary storage cluster, sending the first process and an associated first generation indicator to a first node of a secondary storage cluster of the multi-site distributed storage system to handle the first process for setting the primary bias state, and initiating a second process for atomically clearing a primary bias state with the first node or any node of the primary storage cluster based on detecting a connection to the mediator or detecting that the mediator is available.
The disclosure describes systems, devices, and methods for replicating metadata in data storage environments. In an example embodiment, a method for operating a controller in a data storage environment to provide cross-shelf data replication is provided. In performing the method, the controller generates metadata for an input/output (I/O) request upon receiving the I/O request. The controller identifies a primary location at which to store the metadata and identifies a secondary location at which to store a replicated version of the metadata, then stores the metadata at the primary location and the replicated version at the secondary location. The primary and secondary locations correspond to storage devices in the data storage environment located on different physical shelves or enclosures, such that the metadata is stored across multiple drive shelves for redundancy and recovery purposes.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
26.
Cross-Grid Replication Within A Distributed Storage System
Various embodiments of the present technology generally relate to systems and methods for providing cross-grid replication within distributed storage systems. In an example, a method includes identifying an object for ingest into a first storage grid containing a first distributed storage system and replicating the object to one or more nodes within the first storage grid. The method may also include determining a cross-grid replication status of the object to a second storage grid containing a second distributed storage system and performing a cross-grid replication of the object to the second storage grid based on the cross-grid replication status of the object.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
Systems and methods for enhancing container security are provided. In one example, exposure of a containerize application to potential security vulnerabilities is reduced by identifying dynamically loaded symbols by the application via performance of static and/or dynamic symbol analysis to identify dynamically loaded symbols that are potentially and/or actually used, respectively, and that correspond to functions contained within shared libraries. Based on a shared library's usage of functions within a standard library and a known mapping between functions of the standard library and system calls, those system calls potentially and actually accessed by the application may be identified and a security policy may be generated and configured for enforcement by a kernel security module to limit system call usage accordingly. Additionally, removal of files or functions of libraries that are deemed unnecessary for proper execution of the applications may be performed to reduce the footprint of the application.
G06F 21/57 - Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
G06F 8/75 - Structural analysis for program understanding
G06F 21/51 - Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
28.
MAINTAINING TIMESTAMP PARITY OF OBJECTS WITH ALTERNATE DATA STREAMS DURING TRANSITION PHASE TO SYNCHRONOUS STATE
Techniques are provided for maintaining timestamp parity during a transition replay phase to a synchronous state. During a transition logging phase where metadata operations executed by a primary node are logged into a metadata log and regions modified by data operations executed by the primary node are tracked within a dirty region log, a close stream operation to close a stream associated with a basefile of the primary node is identified. A determination is made as to whether the dirty region log comprises an entry for the stream indicating that a write data operation previously modified the stream. In an example, in response to the dirty region log comprising the entry, an indicator is set to specify that the stream was deleted by the close stream operation. In another example, a modify timestamp of the basefile is logged into the metadata log for subsequent replication to the secondary node.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
29.
DEFRAGMENTATION FOR LOG STRUCTURED MERGE TREE TO IMPROVE READ AND WRITE AMPLIFICATION
Techniques are provided for implementing a defragmentation process during a merge operation performed by a re-compaction process upon a log structured merge tree. The log structured merge tree is used to store keys of key-value pairs within a key-value store. As the log structured merge tree fills with keys over time, the re-compaction process is performed to merge keys down to lower levels of the log structured merge tree to re-compact the keys. Re-compaction can result in fragmentation because there is a lack of spatial locality of where the re-compaction operations re-writes the keys within storage. Fragmentation increases read and write amplification when accessing the keys stored in different locations within the storage. Accordingly, the defragmentation process is performed during a last merge operation of the re-compaction process in order to store keys together within the storage, thus reducing read and write amplification when accessing the keys.
METHODS AND SYSTEMS FOR A NON-DISRUPTIVE AUTOMATIC UNPLANNED FAILOVER FROM A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for providing an automatic unplanned failover (AUFO) feature to guarantee non-disruptive operations (e.g., operations of business enterprise applications, operations of software application) even in the presence of failures including, but not limited to, network disconnection between multiple data centers and failures of a data center or cluster.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
Various embodiments of the present disclosure relate to compression techniques that can be used for data storage, indexing, and retrieval. In an example embodiment, a controller of or in communication with a data storage system can obtain a request to store a file in the data storage system. The controller identifies time-series data associated with input/output of the file and metrics of the file, generates a feature vector for the file, and compresses the feature vector using a compression technique, resulting in a compressed feature vector having a compression ratio. The controller performs a hash algorithm using the compression ratio to determine a storage device at which to store the compressed feature vector, the compression ratio associated with the file, and an indication of the compression technique. Upon receiving a query for a file, the controller compares compression ratios to identify a storage device storing the requested file.
Techniques are provided for parsing files using node objects within a pool. Conventional techniques for parsing files, such as extensible markup language log files, may not efficiently parse the files, and thus cannot keep up with a rate at which the files are generated and/or populated (e.g., a storage system may generate a significant amount of log data stored in log files over time). The disclosed parsing technique is capable of more efficiently processing the files utilizing less memory and time. In particular, the files are parsed by threads that use node objects within a pool of memory (e.g., a sync pool) to store data being parsed and processed by the threads in parallel. When a thread finishes using a node object, the node object is cleared and returned to the pool for subsequent use by the thread or a different thread, which is memory efficient.
The disclosure describes a system for enforcing role-based access control in a multi-tenant compute cluster with cross-namespace references. A control plane of the compute cluster receives a custom-resource request from a tenant to create or modify a first custom resource in a tenant namespace. The first custom resource references if a second custom resource in an administrative namespace. In response to the request, the control plane transmits a validating admission request to a data-protection controller registered as a webhook endpoint for admission validation. The data-protection controller retrieves access metadata from the referenced second custom resource and generates an admission determination indicating whether the tenant's request satisfies cross-namespace access conditions defined in the metadata. The controller returns the admission determination to the control plane, which admits or denies the custom-resource request accordingly.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for forwarding operations to bypass persistent memory. A modify operation, targeting an object, may be received at a persistent memory tier of a node. If a forwarding policy indicates that forwarding is not enabled for the modify operation and the target object, then the modify operation is executed through a persistent memory file system. If the forwarding policy indicates that forwarding is enabled for the modify operation and the target object, then the modify operation is forwarded to a file system tier as a forwarded operation for execution through a storage file system.
Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.
An object storage system with enhanced processes for managing replicas of data objects. Storage nodes of the object storage system receive replicas of objects to store. In response, a storage node identifies one of multiple virtual object space and its corresponding placement group to associate with the replica of the object. The storage node identifies a storage device associated with the virtual object space, stores the replica on the storage device, and updates system metadata to associate the placement group with the replicated object. In response to a restoration event for a storage device of the system, a storage node identifies associated virtual object spaces and placement groups for the effected storage device. Using the identified virtual object spaces and placement groups, the storage node identifies the objects needing restoration and restores replicas of the objects to a different storage device of the system.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
37.
Migrating data using volume clones in a distibuted storage system
Systems and methods are provided for data migration including cloning a multiple-logical unit number (LUN) volume into a plurality of cloned single-LUN volumes on a first storage node of a computing system; creating a plurality of new volumes on a second storage node of the computing system based at least in part on the plurality of cloned single-LUN volumes; selectively copying snapshot data from the plurality of cloned single-LUN volumes to the plurality of new volumes; establishing snapshot mirror relationships between the plurality of cloned single-LUN volumes and the plurality of new volumes; synchronizing the plurality of cloned single-LUN volumes and the plurality of new volumes; and performing a migration of data logical interface failovers (LIFs) from the plurality of cloned single-LUN volumes to the plurality of new volumes.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Dynamic distribution of compression and decompression job requests to hardware accelerators is disclosed. A set of requests is evaluated to determine a number of compression jobs and a number of decompression jobs in the set of requests. A first set of hardware accelerator engines is allocated to perform compression jobs, and a second set of hardware accelerator engines is allocated to perform decompression jobs. Compression jobs are assigned to the first set of hardware accelerator engines based, at least in part, on a compressibility score of the corresponding job and a workload of the selected hardware accelerator engine. Decompression jobs are assigned to the second set of hardware accelerator engines based, at least in part, on a decompression weight of the corresponding job and a workload of the selected hardware accelerator engine.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
41.
Adaptive Request Grouping Based On Node Pool Impact
The disclosure describes a node management service that groups pods based on an impact of the available instance pool. The node management service identifies a request group associated with a scale-up request to scale up a cluster of compute nodes to host pods in the request group. The node management service iteratively determines to add pods to the request group until an impact of a next pod on a pool of available nodes exceeds a threshold. The node management service sends a request to a distributor to distribute the pods in the request group to one or more nodes obtained from the pool of available nodes.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
42.
MANAGING SHUTDOWN AND RESET OF A NETWORK INTERFACE CARD (NIC)
Managing shutdown and reset of a network interface card (NIC) in response to an error condition is disclosed. An indication to initiate a network interface card (NIC) reset and reconnection sequence is received. A notification of a link down condition is transmitted. Pending connections are disconnected. Queue pairs corresponding to the interconnect channels are destroyed. Links corresponding to the NIC are disconnected. Packets are cleared from queues corresponding to the NIC. Send and receive queues are reset. Queue pairs corresponding to the NIC are recreated. Queue pairs are connected to corresponding links. Data transfer resumes over the links.
The disclosure describes a system for managing a namespace move between nodes of a data storage environment. During a namespace move associated with a data storage system, the system receives requests at a first node (e.g., a source node) in the data storage system to perform input/output (I/O) operations associated with a namespace subject to the namespace move and stores the requests in a queue at the first node until completion of the namespace move. Upon completion of the namespace move, the system forwards the requests from the first node to a second node (e.g., a destination node) in the data storage system and performs the I/O operations at the second node.
Techniques for efficiently and durably implementing erasure coding. Data objects are processed to extract metadata, which is used to identify fragments of the data object and the stripe to which the fragments belong. The techniques described herein evaluate failure domains at the drive-level rather than at the node-level, thereby greatly expanding the number of failure domains for fragment storage. To support object availability and durability, each drive-level failure may only hold one fragment from any given stripe. Further, the storage drives of each storage node are restricted such that the total number of fragments from any given stripe stored in the storage node does not exceed a limit. With this arrangement, not only can erasure coding can be implemented while using fewer computing resources when compared with node-level failure domains, but data objects can also be reconstructed without compromising data integrity under configurable levels of tolerance to unavailable fragments.
The disclosure describes a node management service that determines to deploy a primary workload to a new instance in a compute cluster. The node management service projects an allocatable capacity for the new instance based on historical capacity data. The allocatable capacity is the amount of compute resources available for running the primary workload in the new instance after deployment of a supporting workload to the new instance. The node management service associates the primary workload with the new instance upon determining that the projected allocatable capacity is sufficient for running the primary workload.
Techniques are provided for a get key bot that securely provides access to passphrases. A set key workflow is executed to generate a get key executable binary that is implemented as the get key bot. The get key executable binary is encapsulated with encrypted information that includes a verified user identifier, a passphrase, and/or a bot expiry time. Upon receiving a request for the passphrase from a requestor, the get key executable binary is invoked. The encrypted information is decrypted and compared to a logged-in user identifier and current time for verification. In response to successful verification, the passphrase is provided to the requestor. Otherwise, the requestor is denied access to the passphrase.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup also includes storing in metadata for the backup the LUN ID and whether the LUN ID is based on the first virtual mapping or the physical mapping. Backup includes creating a backup of the application data stored in the storage volume. Application data can subsequently be restored based on the application data that is backed up.
The disclosure describes a node management service that integrates right-sizing and node scaling operations. The node management service modifies a request parameter for a workload deployed in a compute cluster. The node management service determines an updated set of compute nodes for nodes affected by the updated request parameter. The node management service obtains, from a compute provider, the updated set of compute nodes for the compute cluster. The node management service provides the modified request parameter to a control plane after obtaining the updated set of compute nodes.
In one example, a computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a non-disruptive planned failover (PFO) to change a role for the secondary storage site and change a role for the primary storage site, initiating, with the primary storage site and/or secondary storage site, a PFO out of synchronization (OOS) event that is sent to a mediator agent with no indication of a role for serving IO, and starting a configuration independent unplanned failover if a disaster or site failure occurs during the PFO.
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
50.
SYSTEMS AND METHODS FOR NON-DISRUPTIVE PLANNED FAILOVER WITHIN A CROSS-SITE STORAGE SYSTEM HAVING BIDIRECTIONAL SYNCHRONOUS REPLICATION
In one example, the present storage solution provides an order of operations of a computer-implemented method that includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO). The method includes initiating a non-disruptive planned failover to change a role for the secondary storage site and change a role for the primary storage site while maintaining in sync status of the bi-directional synchronous replication between the one or more members of the CG1 of the primary storage site and the one or more members of the CG2 of a secondary storage site, and while maintaining zero data loss protection.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
51.
Replication Engine(s) For Preserving Discontiguous And Fragmented Compressed Data Units
Various embodiments of the present technology generally relate to systems and methods for providing a replication engine for preserving discontiguous and fragmented compressed data extents (CDEs). In an aspect, a replication engine may determine a replication request to replicate one or more data blocks from a source storage system to a destination storage system. Based on the replication request, the replication engine may determine a first CDE containing the one or more data blocks. The replication engine may also determine a transfer map associated with the replication request. Based on the transfer map, the replication engine may determine a replication state associated with the one or more data blocks and initiate replication of the first CDE from the source storage system to the destination storage system based on the replication state associated with the one or more data blocks.
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.
Techniques are provided for incremental backup to an object store. A request may be received from an application to perform a backup from a volume hosted by a node to a backup target within the object store. A set of changed files within the volume since a prior backup of the volume was performed to the backup target is identified, along with metadata associated with the set of changed files. The metadata is utilized to identify changed data blocks comprising data of the set of changed files that was modified since the prior backup. The changed data blocks are backed up to the object store.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to establish a remote direct memory access (RDMA) connection with a client computer system and remotely access a crash dump file stored at the client computer system, including providing a file offset of interest and size of data to be read from the client computer system and translating the file offset of interest and size of data into a plurality of RDMA messages.
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to receive a request from a remote computer system to initiate a remote direct memory access (RDMA) connection, establish the RDMA connection with the remote computer system and provide access to a crash dump file via the RDMA connection.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Systems and methods for implementation and use of remote clone volumes within a distributed storage system are provided. In one of various contemplated examples, all nodes of multiple nodes of a cluster representing a distributed storage system are able to access an entirety of a global physical volume block number (PVBN) space of a storage pod. A remote clone volume of a parent volume of a source node may be created for use by a destination node by creating a dummy volume on the destination node. The dummy volume may then be converted into the remote clone volume by copying metadata associated with a backing snapshot of the parent volume to the dummy volume. After completing creation of the remote clone volume, the backing snapshot may then be locked to protect the backing snapshot from deletion.
Systems, methods, and software are disclosed herein for identifying duplicate blocks of a storage system and deduplicating the storage system. In one example, a method of operating a computing device includes scanning first metadata of blocks of a container file of a virtual volume to generate a first log file including records of virtual volume block numbers (VVBNs) and fingerprints of the blocks; scanning second metadata of blocks of an active file system of the virtual volume to generate a second log file including records of VVBNs and file block numbers (FBNs) of the blocks; generating tuples based on merging the records of the first log file and the second records of the second log file according to the VVBNs; identifying duplications among the blocks based on the tuples; and deduplicating the blocks based on the duplications in the active file system identified based on the tuples.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
A liveness monitoring architecture to deterministically ascertain the functional status of nodes in a high-availability (HA) environment to more efficiently recover from a split-brain condition is disclosed.
A method, system and computer program product, the method comprising: determining properties of a set of containers that are deployed over a computer infrastructure, wherein the computer infrastructure is provisioned via an infrastructure management service; determining properties of one or more headroom containers, wherein the one or more headroom containers are not deployed over the computer infrastructure; simulating the container orchestrator using the properties of the set of container and the properties of the headroom containers, for obtaining an expected deployment of the set of containers together with the one or more head room containers; based on the expected deployment, determining whether the computer infrastructure is sufficient for deploying the set of containers together with the one or more headroom containers; and subject to the computer infrastructure being insufficient, issuing a request to the infrastructure management service to allocate additional computer infrastructure.
Various embodiments of the present technology generally relate to systems and methods for providing a data preparation engine for curating secure and compliant data collections from distributed storage systems. In an aspect, a data preparation engine receives a query from a client device and determines files from one or more distributed sources based on the query. The data preparation engine determines sensitive data within the files and anonymizes the sensitive data while preserving context and integrity of the underlying information. The data preparation engine generates a data collection including the files with anonymized sensitive data. The data collection may then be deployed to downstream applications or workflows, such as used to generate curated data sets for training of artificial intelligence applications. Once deployed, the data preparation engine may continuously monitor the distributed sources for changes to data within the files and automatically update data collections in real-time.
Systems and methods are described for performing an instant recovery of data associated with a locked snapshot. In various examples, the amount of time for performing a recovery of data associated with a locked snapshot is performed by making use of volume cloning functionality instead of making an actual copy of the data to be recovered. In one embodiment, the resulting volume clone representing the recovery volume is cleared of all data protection information (e.g., WORM flags and/or lock metafiles) that was previously used to protect the content from being changed when stored on the data protection volume so as allow the recovery volume to be used in read-write mode.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/176 - Support for shared access to filesFile sharing support
The disclosure describes artificial intelligence (AI) data platform that utilizes snapshots obtained from a storage node to update a vector database. The AI data platform compares snapshots to generate differential snapshots that identify changed data in storage volumes. The AI data platform uses the differential snapshots to update vector embeddings in a vector database for retrieval-augmented generation (RAG) workflows.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
37 - Construction and mining; installation and repair services
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Installation, maintenance, and repair of computer hardware for storing, managing, replicating, transferring, securing, retrieving, and restoring data and files; installation, maintenance, and repair of computer hardware for sharing, updating, partitioning, and accessing files over a computer network; installation, maintenance, and repair of computer hardware for optimizing the accessibility, delivery, backup, security, restoration, and replication of data; installation, maintenance, and repair of computer hardware for managing, monitoring, and securing networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; installation, maintenance, and repair of computer hardware using artificial intelligence for the purpose of managing business data; consulting services related to installation, repair, and maintenance of computer hardware and artificial intelligence (AI) in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Educational services, namely, arranging and conducting training courses, lectures, classes, workshops, seminars and conferences in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; providing educational testing to determine professional skills in the fields of computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Providing online and cloud-based non-downloadable software for monitoring, securing, optimizing, analyzing, and managing electronic and cloud data storage, computer servers, databases, and other storage devices and systems across computer networks; technical support services, namely, monitoring and troubleshooting for problems with computers, computer software and hardware and computer storage networks, and providing back-up computer programs and facilities; software as a Service (SAAS) services featuring software for management, monitoring, modeling, troubleshooting, optimization, reporting and analysis of network storage infrastructure and data; cloud computing featuring software for use in controlling, monitoring, and managing cloud infrastructure, virtual services, and networking configuration; design, deployment and management of software and hardware using artificial intelligence for the purpose of managing business data; technical consulting in the field of artificial intelligence (AI) software customization; technical consulting in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; artificial Intelligence as a Service (AIAAS) services featuring software using artificial intelligence (AI) for use in database management; providing online and cloud-based non-downloadable software for deploying, implementing, monitoring, securing, optimizing, analyzing, storing, managing, and troubleshooting artificial intelligence (AI) platforms and internal processes; providing temporary use of online non-downloadable software for use in connection with data management, data storage, data protection, security, and compliance; providing temporary use of online non-downloadable software for managing, accessing, modifying, updating, organizing, delivering, synchronizing, processing, transmitting, storing, and restoring data
66.
SYSTEM AND METHOD FOR EFFICIENT BLOCK LEVEL GRANULAR REPLICATION
A system and method for efficiently restoring one or more data containers is provided. A common persistent consistency point image (PCPI) is identified between a source and a destination storage systems prior to the destination storage system performing a rollback operation to the commonly identified PCPI. Differential data is then transmitted from the source storage system in a line efficient manner to the destination storage system.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
67.
SERVERLESS TIEBREAKER FOR SHARED-NOTHING ARCHITECTURE
Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and/or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and/or assisting in the failover and failback processes.
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
68.
Methods and Systems for Raid Protection in Zoned Solid-State Drives
Methods and systems for a storage environment are provided. One method includes splitting storage of a plurality of zoned solid-state drives (ZNS SSDs) into a plurality of physical zones (PZones) across a plurality of independent media units of each ZNS SSD, the PZones visible to a first tier RAID (redundant array of independent disks) layer; generating a plurality of RAID zones (RZones), each RZone having a plurality of PZones; presenting one or more RZones to a second tier RAID layer by the first tier RAID layer for processing read and write requests using the plurality of ZNS SSDs; and utilizing, by the first tier RAID layer, a parity PZone at each ZNS SSD for storing parity information corresponding to data written in one or more PZone corresponding to a RZone presented to the second tier RAID layer and storing the parity information in a single parity ZNS SSD.
Systems and methods for providing a file system with object versioning support are provided. Rather than adding object records for each version of an object to a chapter database, in one example, the chapter database may be limited to a single object record for a given object including: (i) a name of the object; (ii) an object file handle containing information regarding a file containing data of a current version of multiple versions of the object; and (iii) a version table file handle containing information regarding a file containing a version table. In this manner, enumeration of objects associated with a given chapter may be performed more efficiently and prior versions of objects may be maintained separately within the version table without causing disproportionate growth of object records and without increasing the search depth with objects that are not referenced by the search at issue.
Techniques are provided for compressing weights of models during training of the models. A model is trained for execution on a target device. As part of training, weights of the model are compressed utilizing palettes to represent weight values using bits. A coding procedure, such as Huffman coding, is used to remove or modify the bit representations of infrequently utilized palettes. The model may be iteratively trained to compress the weights of the model in order to reduce the amount of storage consumed by the model without unduly sacrificing quality of the model. Reducing the size of the model provides the ability to deploy the model on devices that would otherwise lack storage and compute resources for storing and running an uncompressed version of the model.
Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.
The disclosure describes system, devices, and methods for dual-stage vector search. In an example implementation, a method for operating a computer-implemented service is provided. The method includes receiving a context request for content with which to augment a prompt, generating a base vector based on input data in the context request and quantizing the base vector to produce a quantized vector. The method also includes searching a vector database to identify content items based at least on the quantized vector and obtaining the content items and generating base vectors for the content items. The method further includes selecting a subset of the content items based on at least on the base vector generated for the input data and the base vectors for the content items.
The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.
The disclosure describes systems, devices, and methods for managing access to storage devices in a shared-everything data storage environment in which any controller can access each storage device of a storage aggregate. In an implementation, a method for managing the layout of the storage aggregate is provided, which may be performed by a controller. The controller receives a request to add a storage device to the data storage environment, processes the request to identify metadata associated with the storage device, including characteristics of the storage device, processes characteristics of the storage device and characteristics of redundancy groups in the storage environment to select a redundancy group for the drive, and adds the storage device to the redundancy group.
The disclosure describes systems, devices, and methods for tracking operations of controllers in a data storage environment on a per-controller basis. In an implementation, a method for re-performing an incomplete operation is provided. In the method, a controller reads, from a parity drive in the data storage environment, a parity bitmap associated with the controller. The parity bitmap includes sections each corresponding to a different controller in the data storage environment, and each section includes status indicators at specific locations indicative of a status of parity data stored at corresponding locations of a parity region of the parity drive. For each incomplete status indicator, the controller re-computes parity data based on source data associated with the status indicator, stores the parity data at a location of the parity region corresponding to a location of the status indicator in the parity bitmap, and updates the status indicator from incomplete to complete.
The disclosure describes systems, devices, and methods for re-computing lost data in data storage environments. In an example embodiment, a method for rebuilding a failed storage device by multiple controllers in a data storage environment is provided. In the method, each of the controllers determines a failed state of a storage device in the data storage environment. Upon replacement of the failed storage device with a replacement storage device, each controller identifies corresponding storage allocation areas of the storage device, then rebuilds corresponding portions of the failed storage device at portions of the replacement storage device.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
G06F 11/10 - Adding special bits or symbols to the coded information, e.g. parity check, casting out nines or elevens
A data management system can include a disk unit and a set of controllers. The disk unit can contain, at least in part, a set of storage media, a first persistent memory, and a second persistent memory. The set of storage media can be configured to implement a storage space. The set of controllers can be configured to write to the storage space and to implement a set of nodes including a first node and a second node. The first node can be configured to generate and write first node journal data to the first persistent memory. The second node can be configured to obtain a failure indication for the first node, obtain the first node journal data from the second persistent memory, and generate and provide a reply to a backend using the first node journal data.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
Systems, methods, and machine-readable media are disclosed for isolating and reporting a volume placement error for a request to place a volume on a storage platform. A volume placement service requests information from a database using an optimized database query to determine an optimal location to place a new volume. The database returns no results. The volume placement service deconstructs the optimized database query to extract a plurality of queries. The volume placement service iterates over the plurality queries, combining queries in each iteration, to determine a cause for the database to return no results. The volume placement service determines based on the results of each iterative database request a cause the database to return an empty result. The volume placement service provides an indication of the cause for returning an empty result.
Techniques are provided for determining a physical size of a snapshot backed up to an object store. Snapshot data of the snapshot may be backed up into objects that are stored from a node to the object store, such as a cloud computing environment. A tracking object is created to identify which objects within the object store comprise the snapshot data of the snapshot. In order to determine the physical size of the snapshot, the tracking object and/or tracking objects of other snapshots such as a prior snapshot are evaluated to identify a set of objects comprising snapshot data unique to the snapshot and not shared with the prior snapshot. The physical sizes of the set of objects are combined with a metadata size of metadata of the snapshot to determine the physical size of the snapshot.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for implementing a snapshot copy operation between endpoints. One or more snapshots (e.g., snapshots of an on-premise volume) is stored within a source endpoint, such as a source bucket of an object store. A post operation is executed to copy objects comprising snapshot data of a snapshot from the source endpoint to a destination endpoint. A get operation and a tracking object such as a cookie is used to track progress of copying the objects from the source endpoint to the destination endpoint. The tracking object is used to restart the copying of the objects from a point where the copying left off (e.g., in the event there is a failure) without having to restart from the beginning.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
81.
Prevention Of Residual Data Writes After Non-Graceful Node Failure In A Cluster
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.
G06F 11/18 - Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits, e.g. by quadding or by majority decision circuits
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
H04L 65/00 - Network arrangements, protocols or services for supporting real-time applications in data packet communication
H04L 67/00 - Network arrangements or protocols for supporting network services or applications
A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to collect telemetry data of a distributed storage system associated with a client device, monitor a first set of the IOPS values, select a first IOPS value in the first set of the IOPS values as a highest IOPS value, determine whether the first IOPS value is unequal to a current Max-IOPS parameter value and adjust the Max-IOPS parameter value to be equal to the first IOPS value upon a determination that the first IOPS value is unequal to the current Max-IOPS parameter value.
Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.
Improved write allocation in data storage systems is described. A data storage system controller determines a contiguity score for an allocation area on drives of the data storage system. The contiguity score for the allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the allocation area. A contiguity score is then determined for a secondary allocation area within the allocation area. The contiguity score for the secondary allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the smaller allocation area. The physical storage blocks mapped to the secondary allocation area are a subset of the physical storage blocks mapped to the primary allocation area. Where the contiguity score for the secondary allocation area meets or exceeds the contiguity score of the primary allocation area, the secondary allocation area is selected for use.
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer includes a storage device connector configured to couple a dual port interface selected among a first interface protocol and a second interface protocol, and a protocol unit configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue shared among the ports. A single port storage drive coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
G06F 3/06 - Digital input from, or digital output to, record carriers
86.
Coalescing multiple small writes to large files or multiple writes to a number of small files to generate larger compressible chunks for inline compression
Systems and methods for coalescing writes to facilitate generation of larger compression groups for use during inline compression are provided. According to one embodiment, inline compression performed by a storage system is improved by temporarily staging writes to in-memory data structures (e.g., inline storage efficiency (ISE) index nodes (inodes)) and performing coalescing in a deferred manner to generate larger compression groups for use during performance of inline compression. In one example, all files may be treated in the same manner, for example, by staging writes within a staging area and then processing the staged data by an inline compression workflow. In another example, the staging processing for small and large file may be different. For instance, the data blocks associated with small files may be staged separately from data blocks associated with large files and/or data blocks of multiple small files may be staged within the same ISE inode.
G06F 3/06 - Digital input from, or digital output to, record carriers
87.
SYSTEMS AND METHODS TO HANDLE DEPENDENT DATA, CONFLICTING DATA, OR METADATA OPERATIONS ON A DUAL COPY CROSS-SITE STORAGE SYSTEM WITH SIMULATANEOUS READ-WRITE ABILITY ON EACH COPY
The present storage solution provides an order of operations of a computer-implemented method that includes implementing a primary-First principle with a first data Op received by the primary storage site being executed on the primary storage site and then replicated to the secondary storage site and a second data Op received by the secondary storage site being first replicated to the primary storage site. The method further includes acquiring overlap write manager (OWM) lock locally on the primary storage site for the first data Op if there are no conflicting ops that are already inflight working on an overlapping range, sending the first data Op to a file system of the primary storage site to modify the file system as per primary-first principle, and suspending any new Ops from the primary storage site that have an overlapping range that overlaps with a range of the first data Op.
The present storage solution provides an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing persistent failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time to maintain dependent write order consistency for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing secondary side write Op handling to maintain dependent write order consistency for a symmetric distributed storage system; and an order of operations of a computer-implemented method for performing secondary side read Op handling to maintain dependent write order consistency for a symmetric distributed storage system in accordance with some embodiments.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
A computer-implemented method includes receiving, with the primary storage site, a clone request for a copy of data, invoking, based on the clone request an asynchronous drain with hold (DWH) process to drain any inflight operations (ops) on the primary storage site and hold any new ops received on the primary storage site, sending a replication message from the primary storage site to the secondary storage site to invoke an asynchronous DWH process on the secondary storage site to drain any inflight ops on the secondary storage site and hold any new ops received on the secondary storage site, and waiting for a completion notification from both the DWH process of the primary storage site and the DWH process of the secondary storage site.
Techniques are provided for creating file clones of multipart files. Creating clones of files is an integral part of providing backup, restore, and other storage services. However, conventional file cloning techniques are unable to create clones of multipart files that are composed of multiple parts stored across different volumes and/or nodes in a constant time. The disclosed techniques are capable of cloning multipart files by creating a clone parent file into which catalog entries from a source multiple file are moved. A destination multipart file is initially created as an empty clone of the source multipart file. Block sharing of the catalog entries from the clone parent file to the source and destination multipart files is performed, and cloning of the source multipart file is declared complete in a constant time such as within a few seconds or less.
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer (110) includes a storage device connector (112) configured to couple a dual port interface (141, 142) selected among a first interface protocol (141) and a second interface protocol (142), and a protocol unit (120, 725) configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit (120, 726) configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue (121) shared among the ports. A single port storage drive (130) coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
G06F 3/06 - Digital input from, or digital output to, record carriers
92.
SYSTEMS AND METHODS TO REDUCE APPLICATION INPUT/OUTPUT RESUMPTION TIME DUE TO A FAILURE OF A STORAGE SITE OR A NETWORK PARTITION WITHIN A CROSS-SITE STORAGE SYSTEM
A computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access. The method includes detecting a disruption in a data replication session from one or more members of the CG1 to one or more members of the CG2 due to a disaster event, initiating a consensus establishment request to be sent to a mediator agent of the primary storage site, rejecting with the mediator agent the consensus establishment request, and initiating a role flip process for primary and secondary roles in serving I/O Operations in response to the rejection of the consensus establishment request to reduce an application input/output (I/O) resumption time due to the disaster event.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its related functions. In an aspect, a waypoint prediction engine may determine navigation data associated with a client device and a source model. Then a decompression-side of the waypoint prediction engine may generate a predicted waypoint based on the source model and the navigation data for the client device as the client device travels along a navigation route. The decompression-side may receive, from a compression-side of the waypoint prediction engine, a correction factor for the predicted waypoint. Responsive to receiving the correction factor, the decompression-side of the waypoint prediction engine may store the correction factor as associated with the source model, where the correction factor and the source model allow for recreation of the navigation route of the client device.
Various mechanisms and workflows are described that can utilize power and/or carbon footprint-based metrics to manage storage unit usage and/or configuration, which can provide a more efficient and environmentally friendly computing environment. In some example configurations, storage system management mechanisms collect power consumption for storage units (e.g., individual drives, storage shelfs, nodes, clusters) and can utilize the power consumption information with other storage unit characteristics to generate power and carbon footprint metrics.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its functions. For example, a waypoint prediction engine may determine navigation data associated with a client device and a source model. The waypoint prediction engine may determine a current waypoint of the client device as the client device travels along a navigation route and generate, by a compression-side of the waypoint prediction engine, a predicted waypoint based on the source model and the navigation data. The compression-side may determine an accuracy of the predicted waypoint and generate a correction factor based on the accuracy of the predicted waypoint. The compression-side may transmit the correction factor to a decompression-side of the waypoint prediction engine, which may, in turn store the correction factor such that the correction factor and the source model allow for recreation of the navigation route of the client device.
Techniques are provided for upgrading an external distributed storage layer that provides storage services to containerized applications hosted within a container hosting platform. An operator within the container hosting platform is custom configured to orchestrate, from within the container hosting platform, the upgrade for the external distributed storage layer. Because the external distributed storage layer and the container hosting platform are separate computing environment that utilize different namespaces, semantics, operating states, and/or application programming interfaces, a cluster controller within the container hosting platform is custom configured to reformat/translate commands between the external distributed storage layer and the container hosting platform for performing the upgrade. Because the external distributed storage layer upgrade may be part of an overall upgrade that upgrades the containerized applications hosted within the container hosting platform, the operator and cluster controller provide a single upgrade orchestration point for perform both upgrades in an orchestrated manner.
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Systems and methods for reducing the provisioned storage capacity of a storage device or aggregate of storage devices are provided. According to one embodiment, the size of the aggregate may be reduced by shrinking the file system of the storage appliance and removing a selected storage device from the aggregate. When an identified shrink region is less than the entire addressable space of the selected storage device, the file system is shrunk by relocating data from the shrink region of the selected storage device to one or more regions outside of the shrink region, mirroring data of the selected storage device from outside of the shrink region to a smaller storage device added to the aggregate, and then removing the selected storage device after the mirrors are in sync, thereby reducing the provisioned storage capacity by the difference in size between the selected storage device and the smaller storage device.
Techniques are provided for artificial intelligence (AI) based application error detection and resolution. Extensive amounts of time and resources are consumed by service providers when attempting to resolve application errors experienced by customers. Unfortunately, a service provider may spend tedious amounts of manual effort to evaluate and solve an error that is already known or already solved. The techniques provided herein reduce the amount of time and resources involved in detecting and resolving errors associated with applications. In particular, an error mapping is generated for a current troubleshooting case to resolve for an application. The error mapping is compared to error mappings of previously resolved troubleshooting cases. If a match is found, then a troubleshooting action associated with a previously resolved troubleshooting case is suggested or executed. Otherwise, a service ticket is created for solving the current troubleshooting cases.
Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
