Techniques are provided for forwarding operations to bypass persistent memory. A modify operation, targeting an object, may be received at a persistent memory tier of a node. If a forwarding policy indicates that forwarding is not enabled for the modify operation and the target object, then the modify operation is executed through a persistent memory file system. If the forwarding policy indicates that forwarding is enabled for the modify operation and the target object, then the modify operation is forwarded to a file system tier as a forwarded operation for execution through a storage file system.
The disclosure describes a system for enforcing role-based access control in a multi-tenant compute cluster with cross-namespace references. A control plane of the compute cluster receives a custom-resource request from a tenant to create or modify a first custom resource in a tenant namespace. The first custom resource references if a second custom resource in an administrative namespace. In response to the request, the control plane transmits a validating admission request to a data-protection controller registered as a webhook endpoint for admission validation. The data-protection controller retrieves access metadata from the referenced second custom resource and generates an admission determination indicating whether the tenant's request satisfies cross-namespace access conditions defined in the metadata. The controller returns the admission determination to the control plane, which admits or denies the custom-resource request accordingly.
G06F 21/62 - Protecting access to data via a platform, e.g. using keys or access control rules
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for parsing files using node objects within a pool. Conventional techniques for parsing files, such as extensible markup language log files, may not efficiently parse the files, and thus cannot keep up with a rate at which the files are generated and/or populated (e.g., a storage system may generate a significant amount of log data stored in log files over time). The disclosed parsing technique is capable of more efficiently processing the files utilizing less memory and time. In particular, the files are parsed by threads that use node objects within a pool of memory (e.g., a sync pool) to store data being parsed and processed by the threads in parallel. When a thread finishes using a node object, the node object is cleared and returned to the pool for subsequent use by the thread or a different thread, which is memory efficient.
Techniques are provided for combining data block and checksum block I/O into a single I/O operation. Many storage systems utilize checksums to verify the integrity of data blocks stored within storage devices managed by a storage stack. However, when a storage system reads a data block from a storage device, a corresponding checksum must also be read to verify integrity of the data in the data block. This results in increased latency because two read operations are being processed through the storage stack and are being executed upon the storage device. To reduce this latency and improve I/O operations per second, a single combined I/O operation corresponding to a contiguous range of blocks including the data block and the checksum block is processed through the storage stack instead of two separate I/O operations. Additionally, I/O operation may be combined into a single request that is executed upon the storage device.
An object storage system with enhanced processes for managing replicas of data objects. Storage nodes of the object storage system receive replicas of objects to store. In response, a storage node identifies one of multiple virtual object space and its corresponding placement group to associate with the replica of the object. The storage node identifies a storage device associated with the virtual object space, stores the replica on the storage device, and updates system metadata to associate the placement group with the replicated object. In response to a restoration event for a storage device of the system, a storage node identifies associated virtual object spaces and placement groups for the effected storage device. Using the identified virtual object spaces and placement groups, the storage node identifies the objects needing restoration and restores replicas of the objects to a different storage device of the system.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
6.
Migrating data using volume clones in a distibuted storage system
Systems and methods are provided for data migration including cloning a multiple-logical unit number (LUN) volume into a plurality of cloned single-LUN volumes on a first storage node of a computing system; creating a plurality of new volumes on a second storage node of the computing system based at least in part on the plurality of cloned single-LUN volumes; selectively copying snapshot data from the plurality of cloned single-LUN volumes to the plurality of new volumes; establishing snapshot mirror relationships between the plurality of cloned single-LUN volumes and the plurality of new volumes; synchronizing the plurality of cloned single-LUN volumes and the plurality of new volumes; and performing a migration of data logical interface failovers (LIFs) from the plurality of cloned single-LUN volumes to the plurality of new volumes.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Dynamic distribution of compression and decompression job requests to hardware accelerators is disclosed. A set of requests is evaluated to determine a number of compression jobs and a number of decompression jobs in the set of requests. A first set of hardware accelerator engines is allocated to perform compression jobs, and a second set of hardware accelerator engines is allocated to perform decompression jobs. Compression jobs are assigned to the first set of hardware accelerator engines based, at least in part, on a compressibility score of the corresponding job and a workload of the selected hardware accelerator engine. Decompression jobs are assigned to the second set of hardware accelerator engines based, at least in part, on a decompression weight of the corresponding job and a workload of the selected hardware accelerator engine.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
A global policy-driven framework for managing, revising, and implementing data coherency policies in a distributed data storage system. An edge node of the distributed data storage system receives a request from an application to store a data object in the system. The edge node, in response to receiving the request to store the data object, generates a coherency policy request, which is then submitted to the primary node of the system. The primary node of the system generates a coherency policy response, which is returned to the edge node. The edge node then stores the data object in the system in accordance with the coherency policy response.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
10.
Adaptive Request Grouping Based On Node Pool Impact
The disclosure describes a node management service that groups pods based on an impact of the available instance pool. The node management service identifies a request group associated with a scale-up request to scale up a cluster of compute nodes to host pods in the request group. The node management service iteratively determines to add pods to the request group until an impact of a next pod on a pool of available nodes exceeds a threshold. The node management service sends a request to a distributor to distribute the pods in the request group to one or more nodes obtained from the pool of available nodes.
G06F 9/50 - Allocation of resources, e.g. of the central processing unit [CPU]
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
11.
MANAGING SHUTDOWN AND RESET OF A NETWORK INTERFACE CARD (NIC)
Managing shutdown and reset of a network interface card (NIC) in response to an error condition is disclosed. An indication to initiate a network interface card (NIC) reset and reconnection sequence is received. A notification of a link down condition is transmitted. Pending connections are disconnected. Queue pairs corresponding to the interconnect channels are destroyed. Links corresponding to the NIC are disconnected. Packets are cleared from queues corresponding to the NIC. Send and receive queues are reset. Queue pairs corresponding to the NIC are recreated. Queue pairs are connected to corresponding links. Data transfer resumes over the links.
The disclosure describes a system for managing a namespace move between nodes of a data storage environment. During a namespace move associated with a data storage system, the system receives requests at a first node (e.g., a source node) in the data storage system to perform input/output (I/O) operations associated with a namespace subject to the namespace move and stores the requests in a queue at the first node until completion of the namespace move. Upon completion of the namespace move, the system forwards the requests from the first node to a second node (e.g., a destination node) in the data storage system and performs the I/O operations at the second node.
Techniques for efficiently and durably implementing erasure coding. Data objects are processed to extract metadata, which is used to identify fragments of the data object and the stripe to which the fragments belong. The techniques described herein evaluate failure domains at the drive-level rather than at the node-level, thereby greatly expanding the number of failure domains for fragment storage. To support object availability and durability, each drive-level failure may only hold one fragment from any given stripe. Further, the storage drives of each storage node are restricted such that the total number of fragments from any given stripe stored in the storage node does not exceed a limit. With this arrangement, not only can erasure coding can be implemented while using fewer computing resources when compared with node-level failure domains, but data objects can also be reconstructed without compromising data integrity under configurable levels of tolerance to unavailable fragments.
The disclosure describes a node management service that determines to deploy a primary workload to a new instance in a compute cluster. The node management service projects an allocatable capacity for the new instance based on historical capacity data. The allocatable capacity is the amount of compute resources available for running the primary workload in the new instance after deployment of a supporting workload to the new instance. The node management service associates the primary workload with the new instance upon determining that the projected allocatable capacity is sufficient for running the primary workload.
Techniques are provided for a get key bot that securely provides access to passphrases. A set key workflow is executed to generate a get key executable binary that is implemented as the get key bot. The get key executable binary is encapsulated with encrypted information that includes a verified user identifier, a passphrase, and/or a bot expiry time. Upon receiving a request for the passphrase from a requestor, the get key executable binary is invoked. The encrypted information is decrypted and compared to a logged-in user identifier and current time for verification. In response to successful verification, the passphrase is provided to the requestor. Otherwise, the requestor is denied access to the passphrase.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup of application data associated with an application executing in a virtual machine managed by a hypervisor is performed. Backup of the application data includes retrieving a Logical Unit Number (LUN) identification (ID) used by the application to store the application data in a storage volume. Backup of the application data also includes performing a virtual storage resolution for the LUN ID to determine whether the application data is stored in the storage volume identified by the LUN ID based on a first virtual mapping or a physical mapping.
Backup also includes storing in metadata for the backup the LUN ID and whether the LUN ID is based on the first virtual mapping or the physical mapping. Backup includes creating a backup of the application data stored in the storage volume. Application data can subsequently be restored based on the application data that is backed up.
The disclosure describes a node management service that integrates right-sizing and node scaling operations. The node management service modifies a request parameter for a workload deployed in a compute cluster. The node management service determines an updated set of compute nodes for nodes affected by the updated request parameter. The node management service obtains, from a compute provider, the updated set of compute nodes for the compute cluster. The node management service provides the modified request parameter to a control plane after obtaining the updated set of compute nodes.
In one example, a computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a non-disruptive planned failover (PFO) to change a role for the secondary storage site and change a role for the primary storage site, initiating, with the primary storage site and/or secondary storage site, a PFO out of synchronization (OOS) event that is sent to a mediator agent with no indication of a role for serving IO, and starting a configuration independent unplanned failover if a disaster or site failure occurs during the PFO.
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
19.
SYSTEMS AND METHODS FOR NON-DISRUPTIVE PLANNED FAILOVER WITHIN A CROSS-SITE STORAGE SYSTEM HAVING BIDIRECTIONAL SYNCHRONOUS REPLICATION
In one example, the present storage solution provides an order of operations of a computer-implemented method that includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO). The method includes initiating a non-disruptive planned failover to change a role for the secondary storage site and change a role for the primary storage site while maintaining in sync status of the bi-directional synchronous replication between the one or more members of the CG1 of the primary storage site and the one or more members of the CG2 of a secondary storage site, and while maintaining zero data loss protection.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.
Various embodiments of the present technology generally relate to systems and methods for providing a replication engine for preserving discontiguous and fragmented compressed data extents (CDEs). In an aspect, a replication engine may determine a replication request to replicate one or more data blocks from a source storage system to a destination storage system. Based on the replication request, the replication engine may determine a first CDE containing the one or more data blocks. The replication engine may also determine a transfer map associated with the replication request. Based on the transfer map, the replication engine may determine a replication state associated with the one or more data blocks and initiate replication of the first CDE from the source storage system to the destination storage system based on the replication state associated with the one or more data blocks.
Techniques are provided for incremental backup to an object store. A request may be received from an application to perform a backup from a volume hosted by a node to a backup target within the object store. A set of changed files within the volume since a prior backup of the volume was performed to the backup target is identified, along with metadata associated with the set of changed files. The metadata is utilized to identify changed data blocks comprising data of the set of changed files that was modified since the prior backup. The changed data blocks are backed up to the object store.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to establish a remote direct memory access (RDMA) connection with a client computer system and remotely access a crash dump file stored at the client computer system, including providing a file offset of interest and size of data to be read from the client computer system and translating the file offset of interest and size of data into a plurality of RDMA messages.
A system is described including one or more processing resources and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the one or more processing resources cause the one or more processing resources to receive a request from a remote computer system to initiate a remote direct memory access (RDMA) connection, establish the RDMA connection with the remote computer system and provide access to a crash dump file via the RDMA connection.
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Systems and methods for implementation and use of remote clone volumes within a distributed storage system are provided. In one of various contemplated examples, all nodes of multiple nodes of a cluster representing a distributed storage system are able to access an entirety of a global physical volume block number (PVBN) space of a storage pod. A remote clone volume of a parent volume of a source node may be created for use by a destination node by creating a dummy volume on the destination node. The dummy volume may then be converted into the remote clone volume by copying metadata associated with a backing snapshot of the parent volume to the dummy volume. After completing creation of the remote clone volume, the backing snapshot may then be locked to protect the backing snapshot from deletion.
Systems, methods, and software are disclosed herein for identifying duplicate blocks of a storage system and deduplicating the storage system. In one example, a method of operating a computing device includes scanning first metadata of blocks of a container file of a virtual volume to generate a first log file including records of virtual volume block numbers (VVBNs) and fingerprints of the blocks; scanning second metadata of blocks of an active file system of the virtual volume to generate a second log file including records of VVBNs and file block numbers (FBNs) of the blocks; generating tuples based on merging the records of the first log file and the second records of the second log file according to the VVBNs; identifying duplications among the blocks based on the tuples; and deduplicating the blocks based on the duplications in the active file system identified based on the tuples.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
The disclosure describes system, devices, and methods for fan speed control. In an example implementation, a method for operating a computer-implemented service is provided. The method includes obtaining sensor data from one or more sensors in a data storage environment. The sensor data includes temperature data associated with storage devices in the data storage environment. The method also includes providing an input (e.g., the sensor data) to a machine learning model trained to predict fan control settings of a fan in the data storage environment, determining the fan control setting based on an output from the machine learning model, and controlling the fan based on the fan control setting.
A method, system and computer program product, the method comprising: determining properties of a set of containers that are deployed over a computer infrastructure, wherein the computer infrastructure is provisioned via an infrastructure management service; determining properties of one or more headroom containers, wherein the one or more headroom containers are not deployed over the computer infrastructure; simulating the container orchestrator using the properties of the set of container and the properties of the headroom containers, for obtaining an expected deployment of the set of containers together with the one or more head room containers; based on the expected deployment, determining whether the computer infrastructure is sufficient for deploying the set of containers together with the one or more headroom containers; and subject to the computer infrastructure being insufficient, issuing a request to the infrastructure management service to allocate additional computer infrastructure.
Various embodiments of the present technology generally relate to systems and methods for providing a data preparation engine for curating secure and compliant data collections from distributed storage systems. In an aspect, a data preparation engine receives a query from a client device and determines files from one or more distributed sources based on the query. The data preparation engine determines sensitive data within the files and anonymizes the sensitive data while preserving context and integrity of the underlying information. The data preparation engine generates a data collection including the files with anonymized sensitive data. The data collection may then be deployed to downstream applications or workflows, such as used to generate curated data sets for training of artificial intelligence applications. Once deployed, the data preparation engine may continuously monitor the distributed sources for changes to data within the files and automatically update data collections in real-time.
Systems and methods are described for performing an instant recovery of data associated with a locked snapshot. In various examples, the amount of time for performing a recovery of data associated with a locked snapshot is performed by making use of volume cloning functionality instead of making an actual copy of the data to be recovered. In one embodiment, the resulting volume clone representing the recovery volume is cleared of all data protection information (e.g., WORM flags and/or lock metafiles) that was previously used to protect the content from being changed when stored on the data protection volume so as allow the recovery volume to be used in read-write mode.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/176 - Support for shared access to filesFile sharing support
A liveness monitoring architecture to deterministically ascertain the functional status of nodes in a high-availability (HA) environment to more efficiently recover from a split-brain condition is disclosed.
The disclosure describes artificial intelligence (AI) data platform that utilizes snapshots obtained from a storage node to update a vector database. The AI data platform compares snapshots to generate differential snapshots that identify changed data in storage volumes. The AI data platform uses the differential snapshots to update vector embeddings in a vector database for retrieval-augmented generation (RAG) workflows.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
37 - Construction and mining; installation and repair services
41 - Education, entertainment, sporting and cultural services
42 - Scientific, technological and industrial services, research and design
Goods & Services
Installation, maintenance, and repair of computer hardware for storing, managing, replicating, transferring, securing, retrieving, and restoring data and files; installation, maintenance, and repair of computer hardware for sharing, updating, partitioning, and accessing files over a computer network; installation, maintenance, and repair of computer hardware for optimizing the accessibility, delivery, backup, security, restoration, and replication of data; installation, maintenance, and repair of computer hardware for managing, monitoring, and securing networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; installation, maintenance, and repair of computer hardware using artificial intelligence for the purpose of managing business data; consulting services related to installation, repair, and maintenance of computer hardware and artificial intelligence (AI) in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Educational services, namely, arranging and conducting training courses, lectures, classes, workshops, seminars and conferences in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; providing educational testing to determine professional skills in the fields of computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks Providing online and cloud-based non-downloadable software for monitoring, securing, optimizing, analyzing, and managing electronic and cloud data storage, computer servers, databases, and other storage devices and systems across computer networks; technical support services, namely, monitoring and troubleshooting for problems with computers, computer software and hardware and computer storage networks, and providing back-up computer programs and facilities; software as a Service (SAAS) services featuring software for management, monitoring, modeling, troubleshooting, optimization, reporting and analysis of network storage infrastructure and data; cloud computing featuring software for use in controlling, monitoring, and managing cloud infrastructure, virtual services, and networking configuration; design, deployment and management of software and hardware using artificial intelligence for the purpose of managing business data; technical consulting in the field of artificial intelligence (AI) software customization; technical consulting in the fields of artificial intelligence, computers, computer software and hardware, computer storage networks, computer storage devices, data and computer file storage, management, replication, security, retrieval, restoration and distribution, and management, monitoring and security of computer networks, computer disc drives, electronic data storage systems, databases and other storage devices across computer networks; artificial Intelligence as a Service (AIAAS) services featuring software using artificial intelligence (AI) for use in database management; providing online and cloud-based non-downloadable software for deploying, implementing, monitoring, securing, optimizing, analyzing, storing, managing, and troubleshooting artificial intelligence (AI) platforms and internal processes; providing temporary use of online non-downloadable software for use in connection with data management, data storage, data protection, security, and compliance; providing temporary use of online non-downloadable software for managing, accessing, modifying, updating, organizing, delivering, synchronizing, processing, transmitting, storing, and restoring data
35.
SYSTEM AND METHOD FOR EFFICIENT BLOCK LEVEL GRANULAR REPLICATION
A system and method for efficiently restoring one or more data containers is provided. A common persistent consistency point image (PCPI) is identified between a source and a destination storage systems prior to the destination storage system performing a rollback operation to the commonly identified PCPI. Differential data is then transmitted from the source storage system in a line efficient manner to the destination storage system.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
36.
SERVERLESS TIEBREAKER FOR SHARED-NOTHING ARCHITECTURE
Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and/or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and/or assisting in the failover and failback processes.
H04L 67/1095 - Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
37.
Methods and Systems for Raid Protection in Zoned Solid-State Drives
Methods and systems for a storage environment are provided. One method includes splitting storage of a plurality of zoned solid-state drives (ZNS SSDs) into a plurality of physical zones (PZones) across a plurality of independent media units of each ZNS SSD, the PZones visible to a first tier RAID (redundant array of independent disks) layer; generating a plurality of RAID zones (RZones), each RZone having a plurality of PZones; presenting one or more RZones to a second tier RAID layer by the first tier RAID layer for processing read and write requests using the plurality of ZNS SSDs; and utilizing, by the first tier RAID layer, a parity PZone at each ZNS SSD for storing parity information corresponding to data written in one or more PZone corresponding to a RZone presented to the second tier RAID layer and storing the parity information in a single parity ZNS SSD.
Systems and methods for providing a file system with object versioning support are provided. Rather than adding object records for each version of an object to a chapter database, in one example, the chapter database may be limited to a single object record for a given object including: (i) a name of the object; (ii) an object file handle containing information regarding a file containing data of a current version of multiple versions of the object; and (iii) a version table file handle containing information regarding a file containing a version table. In this manner, enumeration of objects associated with a given chapter may be performed more efficiently and prior versions of objects may be maintained separately within the version table without causing disproportionate growth of object records and without increasing the search depth with objects that are not referenced by the search at issue.
Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.
Techniques are provided for compressing weights of models during training of the models. A model is trained for execution on a target device. As part of training, weights of the model are compressed utilizing palettes to represent weight values using bits. A coding procedure, such as Huffman coding, is used to remove or modify the bit representations of infrequently utilized palettes. The model may be iteratively trained to compress the weights of the model in order to reduce the amount of storage consumed by the model without unduly sacrificing quality of the model. Reducing the size of the model provides the ability to deploy the model on devices that would otherwise lack storage and compute resources for storing and running an uncompressed version of the model.
The disclosure describes system, devices, and methods for dual-stage vector search. In an example implementation, a method for operating a computer-implemented service is provided. The method includes receiving a context request for content with which to augment a prompt, generating a base vector based on input data in the context request and quantizing the base vector to produce a quantized vector. The method also includes searching a vector database to identify content items based at least on the quantized vector and obtaining the content items and generating base vectors for the content items. The method further includes selecting a subset of the content items based on at least on the base vector generated for the input data and the base vectors for the content items.
The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.
The disclosure describes systems, devices, and methods for managing access to storage devices in a shared-everything data storage environment in which any controller can access each storage device of a storage aggregate. In an implementation, a method for managing the layout of the storage aggregate is provided, which may be performed by a controller. The controller receives a request to add a storage device to the data storage environment, processes the request to identify metadata associated with the storage device, including characteristics of the storage device, processes characteristics of the storage device and characteristics of redundancy groups in the storage environment to select a redundancy group for the drive, and adds the storage device to the redundancy group.
The disclosure describes systems, devices, and methods for tracking operations of controllers in a data storage environment on a per-controller basis. In an implementation, a method for re-performing an incomplete operation is provided. In the method, a controller reads, from a parity drive in the data storage environment, a parity bitmap associated with the controller. The parity bitmap includes sections each corresponding to a different controller in the data storage environment, and each section includes status indicators at specific locations indicative of a status of parity data stored at corresponding locations of a parity region of the parity drive. For each incomplete status indicator, the controller re-computes parity data based on source data associated with the status indicator, stores the parity data at a location of the parity region corresponding to a location of the status indicator in the parity bitmap, and updates the status indicator from incomplete to complete.
The disclosure describes systems, devices, and methods for re-computing lost data in data storage environments. In an example embodiment, a method for rebuilding a failed storage device by multiple controllers in a data storage environment is provided. In the method, each of the controllers determines a failed state of a storage device in the data storage environment. Upon replacement of the failed storage device with a replacement storage device, each controller identifies corresponding storage allocation areas of the storage device, then rebuilds corresponding portions of the failed storage device at portions of the replacement storage device.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
G06F 11/10 - Adding special bits or symbols to the coded information, e.g. parity check, casting out nines or elevens
A data management system can include a disk unit and a set of controllers. The disk unit can contain, at least in part, a set of storage media, a first persistent memory, and a second persistent memory. The set of storage media can be configured to implement a storage space. The set of controllers can be configured to write to the storage space and to implement a set of nodes including a first node and a second node. The first node can be configured to generate and write first node journal data to the first persistent memory. The second node can be configured to obtain a failure indication for the first node, obtain the first node journal data from the second persistent memory, and generate and provide a reply to a backend using the first node journal data.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 11/07 - Responding to the occurrence of a fault, e.g. fault tolerance
Systems, methods, and machine-readable media are disclosed for isolating and reporting a volume placement error for a request to place a volume on a storage platform. A volume placement service requests information from a database using an optimized database query to determine an optimal location to place a new volume. The database returns no results. The volume placement service deconstructs the optimized database query to extract a plurality of queries. The volume placement service iterates over the plurality queries, combining queries in each iteration, to determine a cause for the database to return no results. The volume placement service determines based on the results of each iterative database request a cause the database to return an empty result. The volume placement service provides an indication of the cause for returning an empty result.
Techniques are provided for determining a physical size of a snapshot backed up to an object store. Snapshot data of the snapshot may be backed up into objects that are stored from a node to the object store, such as a cloud computing environment. A tracking object is created to identify which objects within the object store comprise the snapshot data of the snapshot. In order to determine the physical size of the snapshot, the tracking object and/or tracking objects of other snapshots such as a prior snapshot are evaluated to identify a set of objects comprising snapshot data unique to the snapshot and not shared with the prior snapshot. The physical sizes of the set of objects are combined with a metadata size of metadata of the snapshot to determine the physical size of the snapshot.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for implementing a snapshot copy operation between endpoints. One or more snapshots (e.g., snapshots of an on-premise volume) is stored within a source endpoint, such as a source bucket of an object store. A post operation is executed to copy objects comprising snapshot data of a snapshot from the source endpoint to a destination endpoint. A get operation and a tracking object such as a cookie is used to track progress of copying the objects from the source endpoint to the destination endpoint. The tracking object is used to restart the copying of the objects from a point where the copying left off (e.g., in the event there is a failure) without having to restart from the beginning.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
50.
Prevention Of Residual Data Writes After Non-Graceful Node Failure In A Cluster
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.
G06F 11/18 - Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits, e.g. by quadding or by majority decision circuits
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
H04L 65/00 - Network arrangements, protocols or services for supporting real-time applications in data packet communication
H04L 67/00 - Network arrangements or protocols for supporting network services or applications
A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to collect telemetry data of a distributed storage system associated with a client device, monitor a first set of the IOPS values, select a first IOPS value in the first set of the IOPS values as a highest IOPS value, determine whether the first IOPS value is unequal to a current Max-IOPS parameter value and adjust the Max-IOPS parameter value to be equal to the first IOPS value upon a determination that the first IOPS value is unequal to the current Max-IOPS parameter value.
Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.
Improved write allocation in data storage systems is described. A data storage system controller determines a contiguity score for an allocation area on drives of the data storage system. The contiguity score for the allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the allocation area. A contiguity score is then determined for a secondary allocation area within the allocation area. The contiguity score for the secondary allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the smaller allocation area. The physical storage blocks mapped to the secondary allocation area are a subset of the physical storage blocks mapped to the primary allocation area. Where the contiguity score for the secondary allocation area meets or exceeds the contiguity score of the primary allocation area, the secondary allocation area is selected for use.
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer includes a storage device connector configured to couple a dual port interface selected among a first interface protocol and a second interface protocol, and a protocol unit configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue shared among the ports. A single port storage drive coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
G06F 3/06 - Digital input from, or digital output to, record carriers
55.
COALESCING MULTIPLE SMALL WRITES TO LARGE FILES OR MULTIPLE WRITES TO A NUMBER OF SMALL FILES TO GENERATE LARGER COMPRESSIBLE CHUNKS FOR INLINE COMPRESSION
Systems and methods for coalescing writes to facilitate generation of larger compression groups for use during inline compression are provided. According to one embodiment, inline compression performed by a storage system is improved by temporarily staging writes to in-memory data structures (e.g., inline storage efficiency (ISE) index nodes (inodes)) and performing coalescing in a deferred manner to generate larger compression groups for use during performance of inline compression. In one example, all files may be treated in the same manner, for example, by staging writes within a staging area and then processing the staged data by an inline compression workflow. In another example, the staging processing for small and large file may be different. For instance, the data blocks associated with small files may be staged separately from data blocks associated with large files and/or data blocks of multiple small files may be staged within the same ISE inode.
G06F 3/06 - Digital input from, or digital output to, record carriers
56.
SYSTEMS AND METHODS TO HANDLE DEPENDENT DATA, CONFLICTING DATA, OR METADATA OPERATIONS ON A DUAL COPY CROSS-SITE STORAGE SYSTEM WITH SIMULATANEOUS READ-WRITE ABILITY ON EACH COPY
The present storage solution provides an order of operations of a computer-implemented method that includes implementing a primary-First principle with a first data Op received by the primary storage site being executed on the primary storage site and then replicated to the secondary storage site and a second data Op received by the secondary storage site being first replicated to the primary storage site. The method further includes acquiring overlap write manager (OWM) lock locally on the primary storage site for the first data Op if there are no conflicting ops that are already inflight working on an overlapping range, sending the first data Op to a file system of the primary storage site to modify the file system as per primary-first principle, and suspending any new Ops from the primary storage site that have an overlapping range that overlaps with a range of the first data Op.
The present storage solution provides an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing persistent failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time to maintain dependent write order consistency for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing secondary side write Op handling to maintain dependent write order consistency for a symmetric distributed storage system; and an order of operations of a computer-implemented method for performing secondary side read Op handling to maintain dependent write order consistency for a symmetric distributed storage system in accordance with some embodiments.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
A computer-implemented method includes receiving, with the primary storage site, a clone request for a copy of data, invoking, based on the clone request an asynchronous drain with hold (DWH) process to drain any inflight operations (ops) on the primary storage site and hold any new ops received on the primary storage site, sending a replication message from the primary storage site to the secondary storage site to invoke an asynchronous DWH process on the secondary storage site to drain any inflight ops on the secondary storage site and hold any new ops received on the secondary storage site, and waiting for a completion notification from both the DWH process of the primary storage site and the DWH process of the secondary storage site.
Techniques are provided for creating file clones of multipart files. Creating clones of files is an integral part of providing backup, restore, and other storage services. However, conventional file cloning techniques are unable to create clones of multipart files that are composed of multiple parts stored across different volumes and/or nodes in a constant time. The disclosed techniques are capable of cloning multipart files by creating a clone parent file into which catalog entries from a source multiple file are moved. A destination multipart file is initially created as an empty clone of the source multipart file. Block sharing of the catalog entries from the clone parent file to the source and destination multipart files is performed, and cloning of the source multipart file is declared complete in a constant time such as within a few seconds or less.
G06F 16/16 - File or folder operations, e.g. details of user interfaces specifically adapted to file systems
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer (110) includes a storage device connector (112) configured to couple a dual port interface (141, 142) selected among a first interface protocol (141) and a second interface protocol (142), and a protocol unit (120, 725) configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit (120, 726) configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue (121) shared among the ports. A single port storage drive (130) coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
Various mechanisms and workflows are described that can utilize power and/or carbon footprint-based metrics to manage storage unit usage and/or configuration, which can provide a more efficient and environmentally friendly computing environment. In some example configurations, storage system management mechanisms collect power consumption for storage units (e.g., individual drives, storage shelfs, nodes, clusters) and can utilize the power consumption information with other storage unit characteristics to generate power and carbon footprint metrics.
SYSTEMS AND METHODS TO REDUCE APPLICATION INPUT/OUTPUT RESUMPTION TIME DUE TO A FAILURE OF A STORAGE SITE OR A NETWORK PARTITION WITHIN A CROSS-SITE STORAGE SYSTEM
A computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access. The method includes detecting a disruption in a data replication session from one or more members of the CG1 to one or more members of the CG2 due to a disaster event, initiating a consensus establishment request to be sent to a mediator agent of the primary storage site, rejecting with the mediator agent the consensus establishment request, and initiating a role flip process for primary and secondary roles in serving I/O Operations in response to the rejection of the consensus establishment request to reduce an application input/output (I/O) resumption time due to the disaster event.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its related functions. In an aspect, a waypoint prediction engine may determine navigation data associated with a client device and a source model. Then a decompression-side of the waypoint prediction engine may generate a predicted waypoint based on the source model and the navigation data for the client device as the client device travels along a navigation route. The decompression-side may receive, from a compression-side of the waypoint prediction engine, a correction factor for the predicted waypoint. Responsive to receiving the correction factor, the decompression-side of the waypoint prediction engine may store the correction factor as associated with the source model, where the correction factor and the source model allow for recreation of the navigation route of the client device.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its functions. For example, a waypoint prediction engine may determine navigation data associated with a client device and a source model. The waypoint prediction engine may determine a current waypoint of the client device as the client device travels along a navigation route and generate, by a compression-side of the waypoint prediction engine, a predicted waypoint based on the source model and the navigation data. The compression-side may determine an accuracy of the predicted waypoint and generate a correction factor based on the accuracy of the predicted waypoint. The compression-side may transmit the correction factor to a decompression-side of the waypoint prediction engine, which may, in turn store the correction factor such that the correction factor and the source model allow for recreation of the navigation route of the client device.
Techniques are provided for upgrading an external distributed storage layer that provides storage services to containerized applications hosted within a container hosting platform. An operator within the container hosting platform is custom configured to orchestrate, from within the container hosting platform, the upgrade for the external distributed storage layer. Because the external distributed storage layer and the container hosting platform are separate computing environment that utilize different namespaces, semantics, operating states, and/or application programming interfaces, a cluster controller within the container hosting platform is custom configured to reformat/translate commands between the external distributed storage layer and the container hosting platform for performing the upgrade. Because the external distributed storage layer upgrade may be part of an overall upgrade that upgrades the containerized applications hosted within the container hosting platform, the operator and cluster controller provide a single upgrade orchestration point for perform both upgrades in an orchestrated manner.
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Systems and methods for reducing the provisioned storage capacity of a storage device or aggregate of storage devices are provided. According to one embodiment, the size of the aggregate may be reduced by shrinking the file system of the storage appliance and removing a selected storage device from the aggregate. When an identified shrink region is less than the entire addressable space of the selected storage device, the file system is shrunk by relocating data from the shrink region of the selected storage device to one or more regions outside of the shrink region, mirroring data of the selected storage device from outside of the shrink region to a smaller storage device added to the aggregate, and then removing the selected storage device after the mirrors are in sync, thereby reducing the provisioned storage capacity by the difference in size between the selected storage device and the smaller storage device.
Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
G06F 11/20 - Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 11/16 - Error detection or correction of the data by redundancy in hardware
69.
ARTIFICIAL INTELLIGENCE BASED APPLICATION ERROR DETECTION AND RESOLUTION
Techniques are provided for artificial intelligence (AI) based application error detection and resolution. Extensive amounts of time and resources are consumed by service providers when attempting to resolve application errors experienced by customers. Unfortunately, a service provider may spend tedious amounts of manual effort to evaluate and solve an error that is already known or already solved. The techniques provided herein reduce the amount of time and resources involved in detecting and resolving errors associated with applications. In particular, an error mapping is generated for a current troubleshooting case to resolve for an application. The error mapping is compared to error mappings of previously resolved troubleshooting cases. If a match is found, then a troubleshooting action associated with a previously resolved troubleshooting case is suggested or executed. Otherwise, a service ticket is created for solving the current troubleshooting cases.
Techniques are provided for implementing a persistent key-value store for caching client data, journaling, and/or crash recovery. The persistent key-value store may be hosted as a primary cache that provides read and write access to key-value record pairs stored within the persistent key-value store. The key-value record pairs are stored within multiple chains in the persistent key-value store. Journaling is provided for the persistent key-value store such that incoming key-value record pairs are stored within active chains, and data within frozen chains is written in a distributed manner across distributed storage of a distributed cluster of nodes. If there is a failure within the distributed cluster of nodes, then the persistent key-value store may be reconstructed and used for crash recovery.
Techniques are provided for block allocation for persistent memory during aggregate transition. In a high availability pair including first and second nodes, the first node makes a determination that control of a first aggregate is to transition from the first node to the second node. A portion of available free storage space is allocated from a first persistent memory of the first node as allocated pages within the first persistent memory. Metadata information for the allocated pages is updated with an identifier of the first aggregate to create updated metadata information reserving the allocated pages for the first aggregate. The updated metadata information is mirrored to the second node, so that the second node also reserves those pages. Control of the first aggregate is transitioned to the second node. As a result, the nodes do not attempt allocating the same free pages to different aggregates during a transition.
A data vault system for quickly acquiring snapshots of primary storage of a data storage service and providing snapshots to the service for recovery. The data vault system is hosted on an isolated network with no communicative visibility from the storage service. The system is configured to minimize vulnerability to attackers by storing both data snapshots and data vault system configuration settings on the isolated network. Further, the snapshots are taken of primary storage, allowing for greatly improved performance compared to snapshots taken of backup data. The ports that facilitate communication between the data storage service and the data vault system can only be enabled from within the isolated network side, as the system is not visible from the data storage service. The system enables and disables ports before and after communication to the data storage service, minimizing vulnerability while the vault system both obtains and provides snapshots.
G06F 21/56 - Computer malware detection or handling, e.g. anti-virus arrangements
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 21/78 - Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer to assure secure storage of data
73.
Isolated snapshot storage for fast ransomware recovery
A data vault system for quickly acquiring snapshots of primary storage of a data storage service and providing snapshots to the service for recovery. The data vault system is hosted on an isolated network with no communicative visibility from the storage service. The system is configured to minimize vulnerability to attackers by storing both data snapshots and data vault system configuration settings on the isolated network. Further, the snapshots are taken of primary storage, allowing for greatly improved performance compared to snapshots taken of backup data. The ports that facilitate communication between the data storage service and the data vault system can only be enabled from within the isolated network side, as the system is not visible from the data storage service. The system enables and disables ports before and after communication to the data storage service, minimizing vulnerability while the vault system both obtains and provides snapshots.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
Techniques are provided for migrating a volume utilizing an object copy work queue and an object copy driver module. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
The disclosure describes a system for developing a forensic projection for data lost in a cyberattack. After identifying a cyberattack causing a loss of data in the data volume, the system identifies a snapshot of the portion of the data volume affected by the cyberattack. The system estimates, based on the snapshot, an amount of lost data caused by the cyberattack. The system then determines based at least on the amount of lost data, a data loss metric.
Techniques are provided for data management across a persistent memory tier and a file system tier. A block within a persistent memory tier of a node is determined to have up-to-date data compared to a corresponding block within a file system tier of the node. The corresponding block may be marked as a dirty block within the file system tier. Location information of a location of the block within the persistent memory tier is encoded into a container associated with the corresponding block. In response to receiving a read operation, the location information is obtained from the container. The up-to-date data is retrieved from the block within the persistent memory tier using the location information for processing the read operation.
The technology disclosed herein enables movement of a lower-performance pod to a lower-performance computing node from a higher-performance computing node. In a particular example, a method includes determining a lower-performance pod is executing on a higher-performance node without at least one higher-performance pod. The method also includes requesting instantiation of a dummy pod from a control plane of a cluster including the higher-performance node. The dummy pod identifies as lower performance to the control plane. In the control plane, the method includes adding a lower-performance node to the cluster, instantiating the dummy pod on the lower-performance node, and moving the lower-performance pod to the lower-performance node in response to determining a lower-performance node is available to host the lower-performance pod.
A cloud asset manager can securely provide multi-tenant access to remote assets while preserving isolation across tenants. The remote asset manager defines various roles for legitimate users of the remote asset manager. The roles are associated with credentials that provide access to the remote assets and/or information about the remote assets maintained by a service provider. And the users map to roles based on attempted actions that access the service provider. Thus, a user's requested action is attempted with credentials associated with a role that maps to the requested action.
Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. Thus, the distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. In this way, the control plane controllers can facilitate communication and performance of commands between the applications of the container orchestration platform and the worker nodes of the distributed storage architecture.
Systems and methods for creation of bucket-level snapshots and snapshot ownership determination are provided. In one example, a storage system maintains a bucket containing multiple objects each having one or more object versions. A snapshot of the bucket may be efficiently created to protect object versions in the bucket at a specific point in time by simply adding an entry, containing information regarding a snapshot identifier (ID) and a snapshot creation time indicator, to a snapshot metafile. Object-modifying operations may be hooked to internally modify them while making it appear to the client the operation has been successfully completed. For example, before deletion of a particular object, an “Is-Object-Protected” check may be performed based on time indicators of the one or more object versions and respective snapshot creation time indicators. When the particular object is protected, it may be subsequently hidden from the client but maintained as an internal version.
Systems and methods for performing an instant and immediately consistent snapshot restore from a client perspective are provided. In one example, a storage system, may restore a previous version of one or more objects to a bucket based on a snapshot of the bucket by performing a background restore process. During the background restore process, the restoration of the previous version of the one or more objects is made to appear instant to a client. For example, during the background restore process, object accesses by the client associated with a read-only operation may be redirected to content of the snapshot. Additionally or alternatively, during the background restore process, prior to acting on a request from the client involving an object-modifying operation relating to a particular object of the one or more objects, the previous version of the particular object may be restored on-demand.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/14 - Details of searching files based on file metadata
Techniques are provided for an object file system for an object store. Data, maintained by a computing device, is stored into slots of an object. The data within the slots of the object is represented as a data structure comprising a plurality of nodes comprising cloud block numbers used to identify the object and particular slots of the object. A mapping metafile is maintained to map block numbers used to store the data by the computing device to cloud block numbers of nodes representing portion of the data stored within slots of the object. The object is stored into the object store, and the mapping metafile and the data structure are used to provide access through the object file system to portions of data within the object.
G06F 16/14 - Details of searching files based on file metadata
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
The technology disclosed herein enables a higher-level process to perform storage volume management with knowledge of a physical storage backend underlying a storage volume. In a particular example, a method includes mounting a storage volume to a computing node of the computing nodes. The storage volume is stored in a storage pool of a plurality of underlying storage pools. The method further includes determining an identifier for the storage pool, receiving a request to duplicate the storage volume, and determining a second identifier for a second storage pool of the plurality of underlying storage pools to which the storage volume will be duplicated. When the second identifier matches the identifier, creating a clone of the storage volume rather than copying the storage volume to the second storage pool.
Systems and methods for supporting granular snapshots are provided. In one example, a storage system may limit a scope of an operation relating to a snapshot of a bucket by applying a snapshot filter associated with the snapshot in which the snapshot filter specifies one or more criteria for determining a subset of multiple objects of a bucket to which the snapshot applies. In one embodiment, the snapshot filer may represent a prefix specified as part of the operation and application of the snapshot filter may involve filtering the multiple objects based on the prefix. The operation may involve creation of a snapshot, enumeration of objects protected by the snapshot, deletion of the snapshot, or restoration of the snapshot. The association of the snapshot filter with the snapshot may be accomplished by persisting the snapshot filter to a snapshot metafile within a snapshot entry corresponding to the snapshot.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
Techniques are provided for directory snapshot and data management. Conventional snapshot functionality creates snapshots at a volume level. Volume level snapshots are inadequate for scale-out storage architectures because a single volume snapshot of a shared storage resource may not satisfy different data protection requirements of clients using the shared storage resource. The disclosed techniques are capable of creating snapshots at a directory level. The directory level snapshots are created and maintained using an inode identity map to track active inode numbers of directory files that have diverged. Snapshot generation numbers are used to determine whether a file is part of a directory for which snapshotting is enabled. A version map used to track versions of a file modified across different directory snapshots and an active file system. A delayed free metafile is used to determine whether file block numbers of a directory can be freed.
Systems and methods are disclosed for implementing a system to generate a knowledge graph of trust relationships between roles in a cloud environment, and to identify misconfigurations that may lead to privilege escalation. In certain embodiments, a method may comprise implementing a graph-based role permission inspection system for identity and access management (IAM) roles in a cloud environment, including generating a graph representation of trust relationships between roles, where a first role having a first set of privileges can endorse a second role having a second set of privileges. The method may further include determining whether the second set of privileges includes a permission not available in the first set of privileges, and generating an indicator that the first role violates a policy when the second set of privileges includes the permission not available in the first set of privileges.
H04L 41/16 - Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
Systems and methods are disclosed for implementing graph-based role inspection for roles in a cloud environment based on a graph neural network (GNN). In certain embodiments, a method may comprise performing graph-based role similarity inspection using a GNN, the graph-based role similarity inspection configured to identify roles, in a graph representation of relationships between identity and access management (IAM) roles of a cloud environment, that are most similar to a target role. The method may include determining a graph structure of the graph representation, identifying the target role, performing a similarly calculation between the target role and other roles in the graph structure to determine similarity scores for the other roles; identifying a similar role having a same security vulnerability as the target role based on the similarity calculation, and correcting the security vulnerability in the similar role based on the identification.
Systems and methods are disclosed for implementing a process for graph database storage optimization, applicable to delta-based cloud asset tracking. In certain embodiments, a method may comprise implementing a delta-based graph storage optimization system for asset tracking in a cloud environment, including storing a graph database representing a configuration of a cloud environment, obtaining configuration settings representing a current state of the cloud environment from a cloud platform, and identifying a delta based on changes between the configuration from the graph database and the configuration settings from the cloud platform. The method may further comprise creating an asset property node based on the delta, and adding the asset property node to the graph database without creating a new graph based on the configuration settings.
In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.
Techniques are provided for object store mirroring. Data within a storage tier of a node may be determined as being data to tier out to a primary object store based upon a property of the data. A first object is generated to comprise the data. A second object is generated to comprise the data. The first object is transmitted to the primary data store for storage in parallel with the second object being transmitted to a mirror object store for storage. Tiering of the data is designated as successful once acknowledgements are received from both the primary object that the first object was stored and the mirror object store that the second object was stored.
G06F 3/06 - Digital input from, or digital output to, record carriers
91.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON FILE SYSTEM OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot process for storage objects. A computer-implemented method comprises performing a baseline transfer from at least one storage object of a first storage node to at least one replicated storage object of a second storage node, starting the common snapshot process including stop processing of I/O operations, performing a snapshot create operation on the primary storage site for the at least one storage object of the first storage node, resuming processing of I/O operations, and assigning a new universal unique identifier (UUID) to the at least one storage object of the second storage node after resuming processing of I/O operations with the new UUID to identify when file system contents are different than the baseline transfer.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
G06F 16/178 - Techniques for file synchronisation in file systems
92.
SLICE FILE RECOVERY USING DEAD REPLICA SLICE FILES
Techniques are provided for repairing a primary slice file, affected by a storage device error, by using one or more dead replica slice files. The primary slice file is used by a node of a distributed storage architecture as an indirection layer between storage containers (e.g., a volume or LUN) and physical storage where data is physically stored. To improve resiliency of the distributed storage architecture, changes to the primary slice file are replicated to replica slice files hosted by other nodes. If a replica slice file falls out of sync with the primary slice file, then the replica slice file is considered dead (out of sync) and could potentially comprise stale data. If a storage device error affects blocks storing data of the primary slice file, then the techniques provided herein can repair the primary slice file using non-stale data from one or more dead replica slice files.
Data is replicated on a backup node, where the granularity of the replication can be less than a full volume. A data consistency group comprising a subset of data for a volume is defined for a primary node. A set of differences for the data consistency group is sent to a backup node. The backup node creates change logs in response to receiving the set of differences. In response to receiving a request to access a file having data in the data consistency group, the backup node creates a clone of the file. The backup node determines whether an update to a data block of the file exists in the change logs. In response to determining that the update to the data block exists in the change logs, the backup node updates a copy of the data block for the cloned file with data in the change logs.
G06F 16/27 - Replication, distribution or synchronisation of data between databases or within a distributed database systemDistributed database system architectures therefor
94.
VIRTUAL MACHINE BACKUP FROM COMPUTING ENVIRONMENT TO STORAGE ENVIRONMENT
Techniques are provided for backing up virtual machines from a computing environment to a storage environment. A virtual machine agent is utilized to generate a snapshot of the virtual machine. Metadata comprising a snapshot identifier of the snapshot and virtual disk information of virtual disks captured by snapshot is generated at the computing environment. The metadata is retrieved and used to create a metafile that is transferred to the storage environment within which snapshots of the virtual machine are to be stored. The snapshot is retrieved from the computing environment and is packaged into a snapshot package having a protocol format used by the storage environment. The snapshot package is transferred to the storage environment.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 9/455 - EmulationInterpretationSoftware simulation, e.g. virtualisation or emulation of application or operating system execution engines
95.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON CONSISTENCY POINT OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshotprocedure for storage objects. A computer-implemented method includes initiating a snap create handler operation for a storage object of a batch of storage objects having a plurality of replicated datasets with each replicated dataset having a synchronous replication relationship between at least one storage object of the first storage node and at least one replicated storage object of the second storage node, determining whether a consistency point is currently in progress or not, and providing a hint to accelerate a currently in progress consistency point when the consistency point is currently in progress.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 16/11 - File system administration, e.g. details of archiving or snapshots
G06F 16/178 - Techniques for file synchronisation in file systems
96.
TRANSITIONING VOLUMES BETWEEN STORAGE VIRTUAL MACHINES
A volume rehost tool migrates a storage volume from a source virtual server within a distributed storage system to a destination storage server within the distributed storage system. The volume rehost tool can prevent client access to data on the volume through the source virtual server until the volume has been migrated to the destination virtual server. The tool identifies a set of storage objects associated with the volume, removes configuration information for the set of storage objects, and removes a volume record associated with the source virtual server for the volume. The tool can then create a new volume record associated with the destination virtual server, apply the configuration information for the set of storage objects to the destination virtual server, and allow client access to the data on the volume through the destination virtual server.
Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.
VERIFICATION OF A PUBLISHED IMAGE HAVING A PREDEFINED PORTION THAT HAS BEEN ALTERED BY A CLOUD PROVIDER PRIOR TO BEING MADE AVAILABLE VIA A MARKETPLACE OF THE CLOUD PROVIDER
Systems and methods for verifying an executable portion of a published cloud image represents an unaltered version of an executable portion of a corresponding original cloud image are provided. In one embodiment, modification of a predefined portion of a cloud image by a cloud provider prior to its publication via a marketplace of the cloud provider is proactively addressed as part of (i) an automated signing process performed by a software publisher on the original cloud image prior to delivery to the cloud provider and (ii) a corresponding background verification process performed on the published cloud image on behalf of users by a management platform. The signing and verification processes are operable to exclude the predefined portion when creating their respective digests, thereby allowing the signed digest created prior to the modification to remain useful as part of a subsequent digest comparison performed by the verification process.
H04L 9/32 - Arrangements for secret or secure communicationsNetwork security protocols including means for verifying the identity or authority of a user of the system
Methods and systems for managing storage is provided. One method includes initializing, a storage service user interface (“SSUI”) within a cloud manager user interface (“CMUI”), the SSUI enables a storage service operation selected from a backup operation, a restore operation and a cloning operation associated with a storage object stored at a cloud volume presented by a cloud layer, the CMUI presented by a cloud provider; transmitting, by the SSUI, an authorization token to a Software As a Service (“SaaS”) layer for authenticating deployment of the SSUI; upon validating the authorization token, initializing a SSUI agent to interface with a deployed storage micro-service layer offered by the cloud layer to execute the storage service operation; transmitting, by the SSUI, an application programming interface (API) request for the SSUI agent for executing the storage service operation; and executing, by the deployed storage micro-service layer, the storage service operation.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
H04L 67/1097 - Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
H04L 67/51 - Discovery or management thereof, e.g. service location protocol [SLP] or web services
100.
REDUCING POTENTIAL DATA-LOSS SCENARIOS WHEN USING EPHEMERAL STORAGE AS BACKING STORAGE FOR JOURNALING BY A VIRTUAL STORAGE SYSTEM
Systems and methods for flushing an operation log journal to both ephemeral storage and persistent storage during a shutdown sequence of a virtual storage system to minimize data-loss scenarios are provided. According to one embodiment, the shutdown or reboot scenarios that result in loss of data are minimized by using persistent storage as a backup to ephemeral storage when the scenario results in rehosting of virtual storage system. For example, responsive to an event indicative of an imminent shutdown or reboot of the virtual storage system, vNVRAM memory may be flushed to both ephemeral storage and persistent storage (e.g., a boot disk). In this manner, when the virtual storage system is rehosted after an unplanned shutdown or reboot resulting from an unrecoverable host error (other than an unrecoverable hardware failure), the operation log journal may be recovered from persistent storage to facilitate vNVRAM replay and avoid data loss.
G06F 11/14 - Error detection or correction of the data by redundancy in operation, e.g. by using different operation sequences leading to the same result
G06F 3/06 - Digital input from, or digital output to, record carriers
G06F 9/455 - EmulationInterpretationSoftware simulation, e.g. virtualisation or emulation of application or operating system execution engines
