A system and method for efficiently restoring one or more data containers is provided. A common persistent consistency point image (PCPI) is identified between a source and a destination storage systems prior to the destination storage system performing a rollback operation to the commonly identified PCPI. Differential data is then transmitted from the source storage system in a line efficient manner to the destination storage system.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
2.
SERVERLESS TIEBREAKER FOR SHARED-NOTHING ARCHITECTURE
Systems and methods for a serverless tiebreaker for a shared-nothing architecture are provided. In some examples, a cloud-native service that supports serialization of writes (or write fencing), for example, via atomic operations with persistent locking and/or reservations, is used to support HA mediation instead of a separate server operating as a tiebreaker, thereby reducing costs and complexity as well as increasing availability and durability of the HA mediation functionality. For example, a fast, fully managed, serverless, key-value noSQL database service (e.g., the Amazon DynamoDB) may be used to perform one or more of maintaining the authoritative source of information regarding which node of an HA pair currently represents the primary node for serving data from a particular dataset, persisting HA metadata, and/or assisting in the failover and failback processes.
H04L 67/1095 - Réplication ou mise en miroir des données, p. ex. l’ordonnancement ou le transport pour la synchronisation des données entre les nœuds du réseau
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
3.
Methods and Systems for Raid Protection in Zoned Solid-State Drives
Methods and systems for a storage environment are provided. One method includes splitting storage of a plurality of zoned solid-state drives (ZNS SSDs) into a plurality of physical zones (PZones) across a plurality of independent media units of each ZNS SSD, the PZones visible to a first tier RAID (redundant array of independent disks) layer; generating a plurality of RAID zones (RZones), each RZone having a plurality of PZones; presenting one or more RZones to a second tier RAID layer by the first tier RAID layer for processing read and write requests using the plurality of ZNS SSDs; and utilizing, by the first tier RAID layer, a parity PZone at each ZNS SSD for storing parity information corresponding to data written in one or more PZone corresponding to a RZone presented to the second tier RAID layer and storing the parity information in a single parity ZNS SSD.
G06F 11/10 - Détection ou correction d'erreur par introduction de redondance dans la représentation des données, p. ex. en utilisant des codes de contrôle en ajoutant des chiffres binaires ou des symboles particuliers aux données exprimées suivant un code, p. ex. contrôle de parité, exclusion des 9 ou des 11
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
Systems and methods for providing a file system with object versioning support are provided. Rather than adding object records for each version of an object to a chapter database, in one example, the chapter database may be limited to a single object record for a given object including: (i) a name of the object; (ii) an object file handle containing information regarding a file containing data of a current version of multiple versions of the object; and (iii) a version table file handle containing information regarding a file containing a version table. In this manner, enumeration of objects associated with a given chapter may be performed more efficiently and prior versions of objects may be maintained separately within the version table without causing disproportionate growth of object records and without increasing the search depth with objects that are not referenced by the search at issue.
Techniques are provided for compressing weights of models during training of the models. A model is trained for execution on a target device. As part of training, weights of the model are compressed utilizing palettes to represent weight values using bits. A coding procedure, such as Huffman coding, is used to remove or modify the bit representations of infrequently utilized palettes. The model may be iteratively trained to compress the weights of the model in order to reduce the amount of storage consumed by the model without unduly sacrificing quality of the model. Reducing the size of the model provides the ability to deploy the model on devices that would otherwise lack storage and compute resources for storing and running an uncompressed version of the model.
Techniques are provided for providing a storage abstraction layer for a composite aggregate architecture. A storage abstraction layer is utilized as an indirection layer between a file system and a storage environment. The storage abstraction layer obtains characteristic of a plurality of storage providers that provide access to heterogeneous types of storage of the storage environment (e.g., solid state storage, high availability storage, object storage, hard disk drive storage, etc.). The storage abstraction layer generates storage bins to manage storage of each storage provider. The storage abstraction layer generates a storage aggregate from the heterogeneous types of storage as a single storage container. The storage aggregate is exposed to the file system as the single storage container that abstracts away from the file system the management and physical storage details of data of the storage aggregate.
The disclosure describes system, devices, and methods for dual-stage vector search. In an example implementation, a method for operating a computer-implemented service is provided. The method includes receiving a context request for content with which to augment a prompt, generating a base vector based on input data in the context request and quantizing the base vector to produce a quantized vector. The method also includes searching a vector database to identify content items based at least on the quantized vector and obtaining the content items and generating base vectors for the content items. The method further includes selecting a subset of the content items based on at least on the base vector generated for the input data and the base vectors for the content items.
G06F 16/38 - Recherche caractérisée par l’utilisation de métadonnées, p. ex. de métadonnées ne provenant pas du contenu ou de métadonnées générées manuellement
The disclosure describes systems, devices, and methods for managing data storage environments. In an example implementation, a method of operating a controller in a data storage environment is provided. In performing the method, the controller identifies a change to a layout of drives in the data storage environment, and in response to identifying the change, takes a lock on instances of layout metadata stored on the drives. The controller then updates the instances of the layout metadata to reflect the change to the layout and releases the lock.
The disclosure describes systems, devices, and methods for managing access to storage devices in a shared-everything data storage environment in which any controller can access each storage device of a storage aggregate. In an implementation, a method for managing the layout of the storage aggregate is provided, which may be performed by a controller. The controller receives a request to add a storage device to the data storage environment, processes the request to identify metadata associated with the storage device, including characteristics of the storage device, processes characteristics of the storage device and characteristics of redundancy groups in the storage environment to select a redundancy group for the drive, and adds the storage device to the redundancy group.
The disclosure describes systems, devices, and methods for tracking operations of controllers in a data storage environment on a per-controller basis. In an implementation, a method for re-performing an incomplete operation is provided. In the method, a controller reads, from a parity drive in the data storage environment, a parity bitmap associated with the controller. The parity bitmap includes sections each corresponding to a different controller in the data storage environment, and each section includes status indicators at specific locations indicative of a status of parity data stored at corresponding locations of a parity region of the parity drive. For each incomplete status indicator, the controller re-computes parity data based on source data associated with the status indicator, stores the parity data at a location of the parity region corresponding to a location of the status indicator in the parity bitmap, and updates the status indicator from incomplete to complete.
G06F 11/10 - Détection ou correction d'erreur par introduction de redondance dans la représentation des données, p. ex. en utilisant des codes de contrôle en ajoutant des chiffres binaires ou des symboles particuliers aux données exprimées suivant un code, p. ex. contrôle de parité, exclusion des 9 ou des 11
The disclosure describes systems, devices, and methods for re-computing lost data in data storage environments. In an example embodiment, a method for rebuilding a failed storage device by multiple controllers in a data storage environment is provided. In the method, each of the controllers determines a failed state of a storage device in the data storage environment. Upon replacement of the failed storage device with a replacement storage device, each controller identifies corresponding storage allocation areas of the storage device, then rebuilds corresponding portions of the failed storage device at portions of the replacement storage device.
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
G06F 11/07 - Réaction à l'apparition d'un défaut, p. ex. tolérance de certains défauts
G06F 11/10 - Détection ou correction d'erreur par introduction de redondance dans la représentation des données, p. ex. en utilisant des codes de contrôle en ajoutant des chiffres binaires ou des symboles particuliers aux données exprimées suivant un code, p. ex. contrôle de parité, exclusion des 9 ou des 11
A data management system can include a disk unit and a set of controllers. The disk unit can contain, at least in part, a set of storage media, a first persistent memory, and a second persistent memory. The set of storage media can be configured to implement a storage space. The set of controllers can be configured to write to the storage space and to implement a set of nodes including a first node and a second node. The first node can be configured to generate and write first node journal data to the first persistent memory. The second node can be configured to obtain a failure indication for the first node, obtain the first node journal data from the second persistent memory, and generate and provide a reply to a backend using the first node journal data.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 11/07 - Réaction à l'apparition d'un défaut, p. ex. tolérance de certains défauts
Systems, methods, and machine-readable media are disclosed for isolating and reporting a volume placement error for a request to place a volume on a storage platform. A volume placement service requests information from a database using an optimized database query to determine an optimal location to place a new volume. The database returns no results. The volume placement service deconstructs the optimized database query to extract a plurality of queries. The volume placement service iterates over the plurality queries, combining queries in each iteration, to determine a cause for the database to return no results. The volume placement service determines based on the results of each iterative database request a cause the database to return an empty result. The volume placement service provides an indication of the cause for returning an empty result.
Techniques are provided for determining a physical size of a snapshot backed up to an object store. Snapshot data of the snapshot may be backed up into objects that are stored from a node to the object store, such as a cloud computing environment. A tracking object is created to identify which objects within the object store comprise the snapshot data of the snapshot. In order to determine the physical size of the snapshot, the tracking object and/or tracking objects of other snapshots such as a prior snapshot are evaluated to identify a set of objects comprising snapshot data unique to the snapshot and not shared with the prior snapshot. The physical sizes of the set of objects are combined with a metadata size of metadata of the snapshot to determine the physical size of the snapshot.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
Techniques are provided for implementing a snapshot copy operation between endpoints. One or more snapshots (e.g., snapshots of an on-premise volume) is stored within a source endpoint, such as a source bucket of an object store. A post operation is executed to copy objects comprising snapshot data of a snapshot from the source endpoint to a destination endpoint. A get operation and a tracking object such as a cookie is used to track progress of copying the objects from the source endpoint to the destination endpoint. The tracking object is used to restart the copying of the objects from a point where the copying left off (e.g., in the event there is a failure) without having to restart from the beginning.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
16.
Prevention Of Residual Data Writes After Non-Graceful Node Failure In A Cluster
The technology disclosed herein enables a storage orchestrator controller to prevent residual data from being written to a storage volume when a node fails non-gracefully. In a particular example, a method includes determining a health status of nodes in the cluster and, in response to determining a node in the cluster failed, marking the node as dirty. After marking the node as dirty and in response to determining the node is ready, the method includes directing the node to erase data in one or more write buffers at the node. The one of more write buffers buffer data for writing to one or more storage volumes when the one or more storage volumes are mounted by the node. After the one or more write buffers are erased, the method includes marking the node as clean.
G06F 11/18 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage passif du défaut des circuits redondants, p. ex. par logique combinatoire des circuits redondants, par circuits à décision majoritaire
G06F 11/16 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
H04L 65/00 - Dispositions, protocoles ou services dans les réseaux de communication de paquets de données pour prendre en charge les applications en temps réel
H04L 67/00 - Dispositions ou protocoles de réseau pour la prise en charge de services ou d'applications réseau
A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to collect telemetry data of a distributed storage system associated with a client device, monitor a first set of the IOPS values, select a first IOPS value in the first set of the IOPS values as a highest IOPS value, determine whether the first IOPS value is unequal to a current Max-IOPS parameter value and adjust the Max-IOPS parameter value to be equal to the first IOPS value upon a determination that the first IOPS value is unequal to the current Max-IOPS parameter value.
Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.
G06F 16/907 - Recherche caractérisée par l’utilisation de métadonnées, p. ex. de métadonnées ne provenant pas du contenu ou de métadonnées générées manuellement
G06F 16/901 - IndexationStructures de données à cet effetStructures de stockage
G06F 16/9035 - Filtrage basé sur des données supplémentaires, p. ex. sur des profils d'utilisateurs ou de groupes
Improved write allocation in data storage systems is described. A data storage system controller determines a contiguity score for an allocation area on drives of the data storage system. The contiguity score for the allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the allocation area. A contiguity score is then determined for a secondary allocation area within the allocation area. The contiguity score for the secondary allocation area is determined based on an evaluation of the contiguity of physical storage blocks mapped to the smaller allocation area. The physical storage blocks mapped to the secondary allocation area are a subset of the physical storage blocks mapped to the primary allocation area. Where the contiguity score for the secondary allocation area meets or exceeds the contiguity score of the primary allocation area, the secondary allocation area is selected for use.
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer includes a storage device connector configured to couple a dual port interface selected among a first interface protocol and a second interface protocol, and a protocol unit configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue shared among the ports. A single port storage drive coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
21.
COALESCING MULTIPLE SMALL WRITES TO LARGE FILES OR MULTIPLE WRITES TO A NUMBER OF SMALL FILES TO GENERATE LARGER COMPRESSIBLE CHUNKS FOR INLINE COMPRESSION
Systems and methods for coalescing writes to facilitate generation of larger compression groups for use during inline compression are provided. According to one embodiment, inline compression performed by a storage system is improved by temporarily staging writes to in-memory data structures (e.g., inline storage efficiency (ISE) index nodes (inodes)) and performing coalescing in a deferred manner to generate larger compression groups for use during performance of inline compression. In one example, all files may be treated in the same manner, for example, by staging writes within a staging area and then processing the staged data by an inline compression workflow. In another example, the staging processing for small and large file may be different. For instance, the data blocks associated with small files may be staged separately from data blocks associated with large files and/or data blocks of multiple small files may be staged within the same ISE inode.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
22.
SYSTEMS AND METHODS TO HANDLE DEPENDENT DATA, CONFLICTING DATA, OR METADATA OPERATIONS ON A DUAL COPY CROSS-SITE STORAGE SYSTEM WITH SIMULATANEOUS READ-WRITE ABILITY ON EACH COPY
The present storage solution provides an order of operations of a computer-implemented method that includes implementing a primary-First principle with a first data Op received by the primary storage site being executed on the primary storage site and then replicated to the secondary storage site and a second data Op received by the secondary storage site being first replicated to the primary storage site. The method further includes acquiring overlap write manager (OWM) lock locally on the primary storage site for the first data Op if there are no conflicting ops that are already inflight working on an overlapping range, sending the first data Op to a file system of the primary storage site to modify the file system as per primary-first principle, and suspending any new Ops from the primary storage site that have an overlapping range that overlaps with a range of the first data Op.
The present storage solution provides an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing persistent failure handling with an improved application I/O resumption time for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing transient failure handling with an improved application I/O resumption time to maintain dependent write order consistency for a symmetric distributed storage system; an order of operations of a computer-implemented method for performing secondary side write Op handling to maintain dependent write order consistency for a symmetric distributed storage system; and an order of operations of a computer-implemented method for performing secondary side read Op handling to maintain dependent write order consistency for a symmetric distributed storage system in accordance with some embodiments.
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
A computer-implemented method includes receiving, with the primary storage site, a clone request for a copy of data, invoking, based on the clone request an asynchronous drain with hold (DWH) process to drain any inflight operations (ops) on the primary storage site and hold any new ops received on the primary storage site, sending a replication message from the primary storage site to the secondary storage site to invoke an asynchronous DWH process on the secondary storage site to drain any inflight ops on the secondary storage site and hold any new ops received on the secondary storage site, and waiting for a completion notification from both the DWH process of the primary storage site and the DWH process of the secondary storage site.
Techniques are provided for creating file clones of multipart files. Creating clones of files is an integral part of providing backup, restore, and other storage services. However, conventional file cloning techniques are unable to create clones of multipart files that are composed of multiple parts stored across different volumes and/or nodes in a constant time. The disclosed techniques are capable of cloning multipart files by creating a clone parent file into which catalog entries from a source multiple file are moved. A destination multipart file is initially created as an empty clone of the source multipart file. Block sharing of the catalog entries from the clone parent file to the source and destination multipart files is performed, and cloning of the source multipart file is declared complete in a constant time such as within a few seconds or less.
G06F 16/16 - Opérations sur les fichiers ou les dossiers, p. ex. détails des interfaces utilisateur spécialement adaptées aux systèmes de fichiers
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
Techniques, equipment, and systems for enhanced storage systems and storage drive interfacing are presented herein. In one example, a storage interposer (110) includes a storage device connector (112) configured to couple a dual port interface (141, 142) selected among a first interface protocol (141) and a second interface protocol (142), and a protocol unit (120, 725) configured to transfer storage transactions received over the dual port interface in a storage format. The storage interposer also includes a transaction unit (120, 726) configured to obtain the storage transactions in the storage format and process indications of which port among the dual port interface supplied each of the storage transactions against one or more criteria to order the storage transactions into a queue (121) shared among the ports. A single port storage drive (130) coupled to the storage interposer can be issued the storage transactions from the queue according to the order.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
27.
SYSTEMS AND METHODS TO REDUCE APPLICATION INPUT/OUTPUT RESUMPTION TIME DUE TO A FAILURE OF A STORAGE SITE OR A NETWORK PARTITION WITHIN A CROSS-SITE STORAGE SYSTEM
A computer-implemented method includes establishing bi-directional synchronous replication between one or more members of a first consistency group (CG1) of a primary storage site and one or more members of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access. The method includes detecting a disruption in a data replication session from one or more members of the CG1 to one or more members of the CG2 due to a disaster event, initiating a consensus establishment request to be sent to a mediator agent of the primary storage site, rejecting with the mediator agent the consensus establishment request, and initiating a role flip process for primary and secondary roles in serving I/O Operations in response to the rejection of the consensus establishment request to reduce an application input/output (I/O) resumption time due to the disaster event.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its related functions. In an aspect, a waypoint prediction engine may determine navigation data associated with a client device and a source model. Then a decompression-side of the waypoint prediction engine may generate a predicted waypoint based on the source model and the navigation data for the client device as the client device travels along a navigation route. The decompression-side may receive, from a compression-side of the waypoint prediction engine, a correction factor for the predicted waypoint. Responsive to receiving the correction factor, the decompression-side of the waypoint prediction engine may store the correction factor as associated with the source model, where the correction factor and the source model allow for recreation of the navigation route of the client device.
Various mechanisms and workflows are described that can utilize power and/or carbon footprint-based metrics to manage storage unit usage and/or configuration, which can provide a more efficient and environmentally friendly computing environment. In some example configurations, storage system management mechanisms collect power consumption for storage units (e.g., individual drives, storage shelfs, nodes, clusters) and can utilize the power consumption information with other storage unit characteristics to generate power and carbon footprint metrics.
Various embodiments of the present technology generally relate to systems and methods for providing a waypoint prediction engine and its functions. For example, a waypoint prediction engine may determine navigation data associated with a client device and a source model. The waypoint prediction engine may determine a current waypoint of the client device as the client device travels along a navigation route and generate, by a compression-side of the waypoint prediction engine, a predicted waypoint based on the source model and the navigation data. The compression-side may determine an accuracy of the predicted waypoint and generate a correction factor based on the accuracy of the predicted waypoint. The compression-side may transmit the correction factor to a decompression-side of the waypoint prediction engine, which may, in turn store the correction factor such that the correction factor and the source model allow for recreation of the navigation route of the client device.
Techniques are provided for upgrading an external distributed storage layer that provides storage services to containerized applications hosted within a container hosting platform. An operator within the container hosting platform is custom configured to orchestrate, from within the container hosting platform, the upgrade for the external distributed storage layer. Because the external distributed storage layer and the container hosting platform are separate computing environment that utilize different namespaces, semantics, operating states, and/or application programming interfaces, a cluster controller within the container hosting platform is custom configured to reformat/translate commands between the external distributed storage layer and the container hosting platform for performing the upgrade. Because the external distributed storage layer upgrade may be part of an overall upgrade that upgrades the containerized applications hosted within the container hosting platform, the operator and cluster controller provide a single upgrade orchestration point for perform both upgrades in an orchestrated manner.
Techniques are provided for performing a resync transfer to recover from a storage site failure. During normal operation of a first site hosting a first volume, data is replicated to a second volume hosted by a second site. If the first site fails, when clients are redirected to the second volume at the second site. When the first site recovers, data modifications made to the second volume are resynced back to the first volume. As part of synchronizing the first volume, a data warehouse is rebuilt at the first site in order to track the location of blocks present on the replication destination. Typically, the data modifications are transferred after the data warehouse is rebuilt, which results in significantly long resync times. The techniques provided herein decrease the resync time by either rebuilding the data warehouse in parallel with resyncing the data modifications or circumvent the need for rebuild.
Systems and methods for reducing the provisioned storage capacity of a storage device or aggregate of storage devices are provided. According to one embodiment, the size of the aggregate may be reduced by shrinking the file system of the storage appliance and removing a selected storage device from the aggregate. When an identified shrink region is less than the entire addressable space of the selected storage device, the file system is shrunk by relocating data from the shrink region of the selected storage device to one or more regions outside of the shrink region, mirroring data of the selected storage device from outside of the shrink region to a smaller storage device added to the aggregate, and then removing the selected storage device after the mirrors are in sync, thereby reducing the provisioned storage capacity by the difference in size between the selected storage device and the smaller storage device.
Techniques are provided for artificial intelligence (AI) based application error detection and resolution. Extensive amounts of time and resources are consumed by service providers when attempting to resolve application errors experienced by customers. Unfortunately, a service provider may spend tedious amounts of manual effort to evaluate and solve an error that is already known or already solved. The techniques provided herein reduce the amount of time and resources involved in detecting and resolving errors associated with applications. In particular, an error mapping is generated for a current troubleshooting case to resolve for an application. The error mapping is compared to error mappings of previously resolved troubleshooting cases. If a match is found, then a troubleshooting action associated with a previously resolved troubleshooting case is suggested or executed. Otherwise, a service ticket is created for solving the current troubleshooting cases.
Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 11/16 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel
Techniques are provided for implementing a persistent key-value store for caching client data, journaling, and/or crash recovery. The persistent key-value store may be hosted as a primary cache that provides read and write access to key-value record pairs stored within the persistent key-value store. The key-value record pairs are stored within multiple chains in the persistent key-value store. Journaling is provided for the persistent key-value store such that incoming key-value record pairs are stored within active chains, and data within frozen chains is written in a distributed manner across distributed storage of a distributed cluster of nodes. If there is a failure within the distributed cluster of nodes, then the persistent key-value store may be reconstructed and used for crash recovery.
G06F 12/0802 - Adressage d’un niveau de mémoire dans lequel l’accès aux données ou aux blocs de données désirés nécessite des moyens d’adressage associatif, p. ex. mémoires cache
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
37.
BLOCK ALLOCATION FOR PERSISTENT MEMORY DURING AGGREGATE TRANSITION
Techniques are provided for block allocation for persistent memory during aggregate transition. In a high availability pair including first and second nodes, the first node makes a determination that control of a first aggregate is to transition from the first node to the second node. A portion of available free storage space is allocated from a first persistent memory of the first node as allocated pages within the first persistent memory. Metadata information for the allocated pages is updated with an identifier of the first aggregate to create updated metadata information reserving the allocated pages for the first aggregate. The updated metadata information is mirrored to the second node, so that the second node also reserves those pages. Control of the first aggregate is transitioned to the second node. As a result, the nodes do not attempt allocating the same free pages to different aggregates during a transition.
A data vault system for quickly acquiring snapshots of primary storage of a data storage service and providing snapshots to the service for recovery. The data vault system is hosted on an isolated network with no communicative visibility from the storage service. The system is configured to minimize vulnerability to attackers by storing both data snapshots and data vault system configuration settings on the isolated network. Further, the snapshots are taken of primary storage, allowing for greatly improved performance compared to snapshots taken of backup data. The ports that facilitate communication between the data storage service and the data vault system can only be enabled from within the isolated network side, as the system is not visible from the data storage service. The system enables and disables ports before and after communication to the data storage service, minimizing vulnerability while the vault system both obtains and provides snapshots.
G06F 21/56 - Détection ou gestion de programmes malveillants, p. ex. dispositions anti-virus
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 21/78 - Protection de composants spécifiques internes ou périphériques, où la protection d'un composant mène à la protection de tout le calculateur pour assurer la sécurité du stockage de données
39.
Isolated Snapshot Storage For Fast Ransomware Recovery
A data vault system for quickly acquiring snapshots of primary storage of a data storage service and providing snapshots to the service for recovery. The data vault system is hosted on an isolated network with no communicative visibility from the storage service. The system is configured to minimize vulnerability to attackers by storing both data snapshots and data vault system configuration settings on the isolated network. Further, the snapshots are taken of primary storage, allowing for greatly improved performance compared to snapshots taken of backup data. The ports that facilitate communication between the data storage service and the data vault system can only be enabled from within the isolated network side, as the system is not visible from the data storage service. The system enables and disables ports before and after communication to the data storage service, minimizing vulnerability while the vault system both obtains and provides snapshots.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
Techniques are provided for migrating a volume utilizing an object copy work queue and an object copy driver module. Data of the volume is stored within objects stored across a storage tier and capacity tier of a source object store. As part of migrating the volume to a destination object store, the objects are migrated to the destination cluster. Directly copying the objects involves multiple read operations to the source object store and a write operation at the destination object store. The techniques provided herein improve the efficiency of the migration by initially sending metadata from the source object store to the destination object store for performing backend object copy operations to migrate the volume. This results in fewer operations and less network usage, thus improving the efficiency and cost of migrating the volume.
The disclosure describes a system for developing a forensic projection for data lost in a cyberattack. After identifying a cyberattack causing a loss of data in the data volume, the system identifies a snapshot of the portion of the data volume affected by the cyberattack. The system estimates, based on the snapshot, an amount of lost data caused by the cyberattack. The system then determines based at least on the amount of lost data, a data loss metric.
Techniques are provided for data management across a persistent memory tier and a file system tier. A block within a persistent memory tier of a node is determined to have up-to-date data compared to a corresponding block within a file system tier of the node. The corresponding block may be marked as a dirty block within the file system tier. Location information of a location of the block within the persistent memory tier is encoded into a container associated with the corresponding block. In response to receiving a read operation, the location information is obtained from the container. The up-to-date data is retrieved from the block within the persistent memory tier using the location information for processing the read operation.
The technology disclosed herein enables movement of a lower-performance pod to a lower-performance computing node from a higher-performance computing node. In a particular example, a method includes determining a lower-performance pod is executing on a higher-performance node without at least one higher-performance pod. The method also includes requesting instantiation of a dummy pod from a control plane of a cluster including the higher-performance node. The dummy pod identifies as lower performance to the control plane. In the control plane, the method includes adding a lower-performance node to the cluster, instantiating the dummy pod on the lower-performance node, and moving the lower-performance pod to the lower-performance node in response to determining a lower-performance node is available to host the lower-performance pod.
A cloud asset manager can securely provide multi-tenant access to remote assets while preserving isolation across tenants. The remote asset manager defines various roles for legitimate users of the remote asset manager. The roles are associated with credentials that provide access to the remote assets and/or information about the remote assets maintained by a service provider. And the users map to roles based on attempted actions that access the service provider. Thus, a user's requested action is attempted with credentials associated with a role that maps to the requested action.
Techniques are provided for implementing a distributed control plane to facilitate communication between a container orchestration platform and a distributed storage architecture. The distributed storage architecture hosts worker nodes that manage distributed storage that can be made accessible to applications within the container orchestration platform through the distributed control plane. The distributed control plane includes control plane controllers that are each paired with a single worker node of the distributed storage architecture. Thus, the distributed control plane is configured to selectively route commands to control plane controllers that are paired with worker nodes that are current owners of objects targeted by the commands. In this way, the control plane controllers can facilitate communication and performance of commands between the applications of the container orchestration platform and the worker nodes of the distributed storage architecture.
Systems and methods for creation of bucket-level snapshots and snapshot ownership determination are provided. In one example, a storage system maintains a bucket containing multiple objects each having one or more object versions. A snapshot of the bucket may be efficiently created to protect object versions in the bucket at a specific point in time by simply adding an entry, containing information regarding a snapshot identifier (ID) and a snapshot creation time indicator, to a snapshot metafile. Object-modifying operations may be hooked to internally modify them while making it appear to the client the operation has been successfully completed. For example, before deletion of a particular object, an “Is-Object-Protected” check may be performed based on time indicators of the one or more object versions and respective snapshot creation time indicators. When the particular object is protected, it may be subsequently hidden from the client but maintained as an internal version.
Systems and methods for performing an instant and immediately consistent snapshot restore from a client perspective are provided. In one example, a storage system, may restore a previous version of one or more objects to a bucket based on a snapshot of the bucket by performing a background restore process. During the background restore process, the restoration of the previous version of the one or more objects is made to appear instant to a client. For example, during the background restore process, object accesses by the client associated with a read-only operation may be redirected to content of the snapshot. Additionally or alternatively, during the background restore process, prior to acting on a request from the client involving an object-modifying operation relating to a particular object of the one or more objects, the previous version of the particular object may be restored on-demand.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/14 - Détails de la recherche de fichiers basée sur les métadonnées des fichiers
Techniques are provided for an object file system for an object store. Data, maintained by a computing device, is stored into slots of an object. The data within the slots of the object is represented as a data structure comprising a plurality of nodes comprising cloud block numbers used to identify the object and particular slots of the object. A mapping metafile is maintained to map block numbers used to store the data by the computing device to cloud block numbers of nodes representing portion of the data stored within slots of the object. The object is stored into the object store, and the mapping metafile and the data structure are used to provide access through the object file system to portions of data within the object.
G06F 16/14 - Détails de la recherche de fichiers basée sur les métadonnées des fichiers
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
The technology disclosed herein enables a higher-level process to perform storage volume management with knowledge of a physical storage backend underlying a storage volume. In a particular example, a method includes mounting a storage volume to a computing node of the computing nodes. The storage volume is stored in a storage pool of a plurality of underlying storage pools. The method further includes determining an identifier for the storage pool, receiving a request to duplicate the storage volume, and determining a second identifier for a second storage pool of the plurality of underlying storage pools to which the storage volume will be duplicated. When the second identifier matches the identifier, creating a clone of the storage volume rather than copying the storage volume to the second storage pool.
Systems and methods are disclosed for implementing a system to generate a knowledge graph of trust relationships between roles in a cloud environment, and to identify misconfigurations that may lead to privilege escalation. In certain embodiments, a method may comprise implementing a graph-based role permission inspection system for identity and access management (IAM) roles in a cloud environment, including generating a graph representation of trust relationships between roles, where a first role having a first set of privileges can endorse a second role having a second set of privileges. The method may further include determining whether the second set of privileges includes a permission not available in the first set of privileges, and generating an indicator that the first role violates a policy when the second set of privileges includes the permission not available in the first set of privileges.
H04L 41/16 - Dispositions pour la maintenance, l’administration ou la gestion des réseaux de commutation de données, p. ex. des réseaux de commutation de paquets en utilisant l'apprentissage automatique ou l'intelligence artificielle
Systems and methods are disclosed for implementing graph-based role inspection for roles in a cloud environment based on a graph neural network (GNN). In certain embodiments, a method may comprise performing graph-based role similarity inspection using a GNN, the graph-based role similarity inspection configured to identify roles, in a graph representation of relationships between identity and access management (IAM) roles of a cloud environment, that are most similar to a target role. The method may include determining a graph structure of the graph representation, identifying the target role, performing a similarly calculation between the target role and other roles in the graph structure to determine similarity scores for the other roles; identifying a similar role having a same security vulnerability as the target role based on the similarity calculation, and correcting the security vulnerability in the similar role based on the identification.
Systems and methods are disclosed for implementing a process for graph database storage optimization, applicable to delta-based cloud asset tracking. In certain embodiments, a method may comprise implementing a delta-based graph storage optimization system for asset tracking in a cloud environment, including storing a graph database representing a configuration of a cloud environment, obtaining configuration settings representing a current state of the cloud environment from a cloud platform, and identifying a delta based on changes between the configuration from the graph database and the configuration settings from the cloud platform. The method may further comprise creating an asset property node based on the delta, and adding the asset property node to the graph database without creating a new graph based on the configuration settings.
Systems and methods for supporting granular snapshots are provided. In one example, a storage system may limit a scope of an operation relating to a snapshot of a bucket by applying a snapshot filter associated with the snapshot in which the snapshot filter specifies one or more criteria for determining a subset of multiple objects of a bucket to which the snapshot applies. In one embodiment, the snapshot filer may represent a prefix specified as part of the operation and application of the snapshot filter may involve filtering the multiple objects based on the prefix. The operation may involve creation of a snapshot, enumeration of objects protected by the snapshot, deletion of the snapshot, or restoration of the snapshot. The association of the snapshot filter with the snapshot may be accomplished by persisting the snapshot filter to a snapshot metafile within a snapshot entry corresponding to the snapshot.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
Techniques are provided for directory snapshot and data management. Conventional snapshot functionality creates snapshots at a volume level. Volume level snapshots are inadequate for scale-out storage architectures because a single volume snapshot of a shared storage resource may not satisfy different data protection requirements of clients using the shared storage resource. The disclosed techniques are capable of creating snapshots at a directory level. The directory level snapshots are created and maintained using an inode identity map to track active inode numbers of directory files that have diverged. Snapshot generation numbers are used to determine whether a file is part of a directory for which snapshotting is enabled. A version map used to track versions of a file modified across different directory snapshots and an active file system. A delayed free metafile is used to determine whether file block numbers of a directory can be freed.
Techniques are provided for object store mirroring. Data within a storage tier of a node may be determined as being data to tier out to a primary object store based upon a property of the data. A first object is generated to comprise the data. A second object is generated to comprise the data. The first object is transmitted to the primary data store for storage in parallel with the second object being transmitted to a mirror object store for storage. Tiering of the data is designated as successful once acknowledgements are received from both the primary object that the first object was stored and the mirror object store that the second object was stored.
In one embodiment, a method comprises maintaining state information regarding a data synchronous replication status for a storage object of a primary storage cluster and a replicated storage object of a secondary storage cluster. The method includes temporarily disallowing input/output (I/O) operations for the storage object when the storage object of the primary storage cluster has a failure, which causes an internal state as out of sync for the storage object while maintaining an external state as in sync for external entities. The method performs persistent inflight tracking and reconciliation of I/O operations with a first Op log of the primary storage cluster and a second Op log of the secondary storage cluster and performs a resynchronization between the storage object and the replicated storage object based on the persistent inflight tracking and reconciliation of I/O operations.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
57.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON FILE SYSTEM OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot process for storage objects. A computer-implemented method comprises performing a baseline transfer from at least one storage object of a first storage node to at least one replicated storage object of a second storage node, starting the common snapshot process including stop processing of I/O operations, performing a snapshot create operation on the primary storage site for the at least one storage object of the first storage node, resuming processing of I/O operations, and assigning a new universal unique identifier (UUID) to the at least one storage object of the second storage node after resuming processing of I/O operations with the new UUID to identify when file system contents are different than the baseline transfer.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
G06F 16/178 - Techniques de synchronisation des fichiers dans les systèmes de fichiers
58.
SLICE FILE RECOVERY USING DEAD REPLICA SLICE FILES
Techniques are provided for repairing a primary slice file, affected by a storage device error, by using one or more dead replica slice files. The primary slice file is used by a node of a distributed storage architecture as an indirection layer between storage containers (e.g., a volume or LUN) and physical storage where data is physically stored. To improve resiliency of the distributed storage architecture, changes to the primary slice file are replicated to replica slice files hosted by other nodes. If a replica slice file falls out of sync with the primary slice file, then the replica slice file is considered dead (out of sync) and could potentially comprise stale data. If a storage device error affects blocks storing data of the primary slice file, then the techniques provided herein can repair the primary slice file using non-stale data from one or more dead replica slice files.
Data is replicated on a backup node, where the granularity of the replication can be less than a full volume. A data consistency group comprising a subset of data for a volume is defined for a primary node. A set of differences for the data consistency group is sent to a backup node. The backup node creates change logs in response to receiving the set of differences. In response to receiving a request to access a file having data in the data consistency group, the backup node creates a clone of the file. The backup node determines whether an update to a data block of the file exists in the change logs. In response to determining that the update to the data block exists in the change logs, the backup node updates a copy of the data block for the cloned file with data in the change logs.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
60.
VIRTUAL MACHINE BACKUP FROM COMPUTING ENVIRONMENT TO STORAGE ENVIRONMENT
Techniques are provided for backing up virtual machines from a computing environment to a storage environment. A virtual machine agent is utilized to generate a snapshot of the virtual machine. Metadata comprising a snapshot identifier of the snapshot and virtual disk information of virtual disks captured by snapshot is generated at the computing environment. The metadata is retrieved and used to create a metafile that is transferred to the storage environment within which snapshots of the virtual machine are to be stored. The snapshot is retrieved from the computing environment and is packaged into a snapshot package having a protocol format used by the storage environment. The snapshot package is transferred to the storage environment.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
61.
METHODS AND SYSTEMS TO REDUCE LATENCY OF INPUT/OUTPUT (I/O) OPERATIONS BASED ON CONSISTENCY POINT OPTIMIZATIONS DURING CREATION OF COMMON SNAPSHOTS FOR SYNCHRONOUS REPLICATED DATASETS OF A PRIMARY COPY OF DATA AT A PRIMARY STORAGE SYSTEM TO A MIRROR COPY OF THE DATA AT A CROSS-SITE SECONDARY STORAGE SYSTEM
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshotprocedure for storage objects. A computer-implemented method includes initiating a snap create handler operation for a storage object of a batch of storage objects having a plurality of replicated datasets with each replicated dataset having a synchronous replication relationship between at least one storage object of the first storage node and at least one replicated storage object of the second storage node, determining whether a consistency point is currently in progress or not, and providing a hint to accelerate a currently in progress consistency point when the consistency point is currently in progress.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 16/11 - Administration des systèmes de fichiers, p. ex. détails de l’archivage ou d’instantanés
G06F 16/178 - Techniques de synchronisation des fichiers dans les systèmes de fichiers
62.
TRANSITIONING VOLUMES BETWEEN STORAGE VIRTUAL MACHINES
A volume rehost tool migrates a storage volume from a source virtual server within a distributed storage system to a destination storage server within the distributed storage system. The volume rehost tool can prevent client access to data on the volume through the source virtual server until the volume has been migrated to the destination virtual server. The tool identifies a set of storage objects associated with the volume, removes configuration information for the set of storage objects, and removes a volume record associated with the source virtual server for the volume. The tool can then create a new volume record associated with the destination virtual server, apply the configuration information for the set of storage objects to the destination virtual server, and allow client access to the data on the volume through the destination virtual server.
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
Methods and systems for managing storage is provided. One method includes initializing, a storage service user interface (“SSUI”) within a cloud manager user interface (“CMUI”), the SSUI enables a storage service operation selected from a backup operation, a restore operation and a cloning operation associated with a storage object stored at a cloud volume presented by a cloud layer, the CMUI presented by a cloud provider; transmitting, by the SSUI, an authorization token to a Software As a Service (“SaaS”) layer for authenticating deployment of the SSUI; upon validating the authorization token, initializing a SSUI agent to interface with a deployed storage micro-service layer offered by the cloud layer to execute the storage service operation; transmitting, by the SSUI, an application programming interface (API) request for the SSUI agent for executing the storage service operation; and executing, by the deployed storage micro-service layer, the storage service operation.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 21/44 - Authentification de programme ou de dispositif
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
H04L 67/51 - Découverte ou gestion de ceux-ci, p. ex. protocole de localisation de service [SLP] ou services du Web
64.
NON-DISRUPTIVE FILE MOVEMENT WITHIN A DISTRIBUTED STORAGE SYSTEM
Approaches for providing a non-disruptive file move are disclosed. A request to move a target file from the first constituent to the second constituent is received. The file has an associated file handle. The target file in the first constituent is converted to a multipart file in the first constituent with a file location for the new file in the first constituent. A new file is created in the second constituent. Contents of the target file are moved to a new file on the second constituent while maintaining access via the associated file handle via access to the multipart file. The target file is deleted from the first constituent.
VERIFICATION OF A PUBLISHED IMAGE HAVING A PREDEFINED PORTION THAT HAS BEEN ALTERED BY A CLOUD PROVIDER PRIOR TO BEING MADE AVAILABLE VIA A MARKETPLACE OF THE CLOUD PROVIDER
Systems and methods for verifying an executable portion of a published cloud image represents an unaltered version of an executable portion of a corresponding original cloud image are provided. In one embodiment, modification of a predefined portion of a cloud image by a cloud provider prior to its publication via a marketplace of the cloud provider is proactively addressed as part of (i) an automated signing process performed by a software publisher on the original cloud image prior to delivery to the cloud provider and (ii) a corresponding background verification process performed on the published cloud image on behalf of users by a management platform. The signing and verification processes are operable to exclude the predefined portion when creating their respective digests, thereby allowing the signed digest created prior to the modification to remain useful as part of a subsequent digest comparison performed by the verification process.
H04L 9/32 - Dispositions pour les communications secrètes ou protégéesProtocoles réseaux de sécurité comprenant des moyens pour vérifier l'identité ou l'autorisation d'un utilisateur du système
G06F 8/71 - Gestion de versions Gestion de configuration
H04L 9/30 - Clé publique, c.-à-d. l'algorithme de chiffrement étant impossible à inverser par ordinateur et les clés de chiffrement des utilisateurs n'exigeant pas le secret
66.
REDUCING POTENTIAL DATA-LOSS SCENARIOS WHEN USING EPHEMERAL STORAGE AS BACKING STORAGE FOR JOURNALING BY A VIRTUAL STORAGE SYSTEM
Systems and methods for flushing an operation log journal to both ephemeral storage and persistent storage during a shutdown sequence of a virtual storage system to minimize data-loss scenarios are provided. According to one embodiment, the shutdown or reboot scenarios that result in loss of data are minimized by using persistent storage as a backup to ephemeral storage when the scenario results in rehosting of virtual storage system. For example, responsive to an event indicative of an imminent shutdown or reboot of the virtual storage system, vNVRAM memory may be flushed to both ephemeral storage and persistent storage (e.g., a boot disk). In this manner, when the virtual storage system is rehosted after an unplanned shutdown or reboot resulting from an unrecoverable host error (other than an unrecoverable hardware failure), the operation log journal may be recovered from persistent storage to facilitate vNVRAM replay and avoid data loss.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
Techniques are provided for journal replay optimization. A distributed storage architecture can implement a journal within memory for logging write operations into log records. Latency of executing the write operations is improved because the write operations can be responded back to clients as complete once logged within the journal without having to store the data to higher latency disk storage. If there is a failure, then a replay process is performed to replay the write operations logged within the journal in order to bring a file system up-to-date. The time to complete the replay of the write operations is significantly reduced by caching metadata (e.g., indirect blocks, checksums, buftree identifiers, file block numbers, and consistency point counts) directly into log records. Replay can quickly access this metadata for replaying the write operations because the metadata does not need to be retrieved from the higher latency disk storage into memory.
Systems and methods for making instance-type recommendations are provided. In various examples, an instance type recommendation system (internal or external to a cloud) provides users (cloud customers) with instance type recommendations and may automatically adjust their instance type groups (ITGs). The instance type recommendations may take into consideration other users with similar requirements and/or be based on frequency of co-occurrence of an instance type of the user at issue with one or more other instance types used by other users as reflected by their respective current ITGs. For example, a multi-layer perceptron (MLP) neural network may be trained by breaking instance types down into respective attributes and causing the MLP to encode the attributes as features and the training may make use of a triplet loss function that minimizes a distance between an anchor and a positive input while maximizing a distance between the anchor and a negative input.
Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.
Systems and methods for using object storage as a primary storage tier for a file system of a virtual storage system are provided. According to one embodiment, ephemeral storage of a host on which a virtual storage system is running in the cloud is used as an external cache in which all or some subset of file system metadata may be stored, for example, to boost read IOPS and reduce latency of the read path. In various examples, external cache policies may be defined to trigger adding to, updating, and/or re-warming the external cache to accelerate access to file system metadata, thereby making a virtual storage solution that maintains all file system data and file system metadata within object storage a suitable and cost-effective platform for certain types of workloads. In some example, the working set of a workload may also be kept in the external cache to further accelerate read performance of the virtual storage system.
Methods and systems involve sequentially storing compressed data segments received at a storage system in disk blocks such that at least one of the disk blocks includes compressed data from at least two of the compressed data segments.
Techniques are provided for performing a storage operation targeting objects stored across multiple storage tiers of a cloud storage environment. A volume may be backed up as objects stored across the multiple storage tiers of the cloud storage environment, such as a standard storage tier directly accessible to the storage operation, an archival storage tier not directly accessible to the storage operation, etc. The storage operation may target the objects, such as where the storage operation is a directory restore operation to restore a directory of the volume. The storage operation can be successfully implemented such as to restore the directory even though objects of the storage operation are stored across the multiple storage tiers of the cloud storage environment.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
73.
KEY VALUE STORE WITH DISTRIBUTED SHARED ACCESS BY CLIENTS
Techniques are provided for hosting a key value store. A persistent storage backend is used to centrally host a key value store as disaggregated storage shared with a plurality of clients over a network fabric. A network storage appliance is connected to the plurality of clients over the network fabric, and is configured with a key value store interface. The key value store interface is configured to receive a key value command from a client. The key value store interface parses the key value command to identify a translation layer binding for a key value store targeted by the key value command. The key value store interface translates the key value command into a key value operation using the translation layer binding, and executes the key value operation upon the key value store.
Systems, methods, and software are disclosed herein for phased-in restoration of an application hosted on a cloud orchestration platform in various implementations. In an implementation, a computing apparatus receives a configuration for a multiphase restoration process for restoring resources of an application to a destination platform, the restoration occurring in phases. To implement the multiphase restoration process, the computing apparatus captures a backup of application data of the application, then restores a phase including selected resources of the application to the destination platform based on the backup and according to the configuration. The computing apparatus validates the selected resources at the destination platform, then restores a next phase to the destination platform based on the backup and according to the configuration.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
75.
MANAGEMENT OF DATA STAGING FOR WRITING TO OBJECT BASED STORAGE
A method, computing device, and non-transitory machine-readable medium for managing incoming/outgoing flow through a staging area. A request to write data to an object based storage is received. A determination is made that the data is to be first staged within a transfer data structure. The transfer data structure is in a first storage tier and the object based storage is in a second storage tier. A determination is made that an amount of storage space used in the transfer data structure exceeds a start throttle threshold. In response to the determination that the amount of storage space exceeds the start throttle threshold, the request is sent into a queue selected from a set of throttling queues, wherein requests in the queue are handled in a first in, first out (FIFO) order. The request in the queue is processed based on the FIFO order and a dynamic throttle rate.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
G06F 12/00 - Accès à, adressage ou affectation dans des systèmes ou des architectures de mémoires
G06F 9/455 - ÉmulationInterprétationSimulation de logiciel, p. ex. virtualisation ou émulation des moteurs d’exécution d’applications ou de systèmes d’exploitation
The disclosure describes a node management service that proactively scales up compute nodes in a compute cluster. The node management service interfaces with an orchestration service, a compute provider and a compute cluster running instances of an object. The node management service receives meta data from an orchestration service indicating the desired number of instances of an object. Based on the desired number of instances, the node management service obtains, from the compute provider, new compute nodes for the compute cluster to accommodate the desired number of instances.
Disclosed herein are methods and systems for the operation of a resource management service. The resource management service deploys reclaimable compute instances from a resource pool and continuously generates predicted remaining lifespans for the deployed reclaimable compute instances. The predicted remaining lifespan is monitored to determine if the predicted remaining lifespan is below a threshold value. In response to the predicted remaining lifespan for a reclaimable compute instance falling below a threshold value, the resource management service instructs the reclaimable compute instance to create an application state snapshot of an application running thereon. A subsequent compute instance is deployed from the resource pool, on which the application can be restored to a previous state using the application state snapshot.
G06F 12/00 - Accès à, adressage ou affectation dans des systèmes ou des architectures de mémoires
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
79.
ANOMALY DETECTION BASED ON STORAGE PROTOCOL CONNECTIONS
Systems and methods for anomaly detection within a storage system based on a number of storage protocol connections are provided. In one example, alerting thresholds based on storage protocol connections are made more relevant and customer-specific by taking into consideration, among other things, one or more of the type/class/model of storage system, the configuration of the storage system, the type of the workload making use of the storage system, and various windows of time. Based on the customer-specific nature of the alerts generated by the proposed alerting system, potential performance issues being experienced by the storage system may be more efficiently identified and remediated, for example, by customer service/support, the customer, or possibly by automated storage system (on-box) functionality.
A method, computing device, and non-transitory machine-readable medium for implementing a client-controllable bypass mechanism for directly writing to object based storage. In one or more embodiments, a write request for writing data to a volume is received from a client in which the volume represents both first storage within a first storage tier and second storage within a second storage tier. A determination is made that a bypass write mode is enabled for the volume in which the bypass write mode allows directly assigning the data to the second storage tier. A set of objects is built for the data in which the data is stored in a transfer data structure in the first storage tier. The set of objects, which includes the data in the transfer data structure, is sent to the second storage tier.
Disclosed herein are systems, methods, and software for the operation of a ransomware detection system. The ransomware detection system generates a decoy file based on characteristics of an existing file in a file system. The decoy file is effectively indistinguishable from the existing file from the perspective of the ransomware but contains simulated data rather than authentic data. The ransomware detection system identifies a location in the file system and deploys the decoy file to the location. The decoy is then monitored to detect changes by comparing a ground truth for the decoy file to the current state of the decoy file. The decoy file is checked for changes at a rate associated with the identified location. Where a change is detected, an alert is sent to a ransomware mitigation process, which initiates ransomware mitigation.
The disclosure describes a data protection service that generates semantic descriptions of protected data volumes. The data protection service queries a monitoring service with the generated semantic descriptions. The monitoring service responds to the queries with indications of whether and data items on the dark web match the semantic descriptions. When a query receives a positive response from the monitoring service, the data protection service iteratively refines the semantic description and queries the monitoring service with the refined semantic descriptions until a breach is detected. Once a breach is detected, the data protection service initiates a mitigation action.
The disclosure describes a data protection service that generates semantic descriptions of protected data volumes. The data protection service queries a monitoring service with the generated semantic descriptions. The monitoring service responds to the queries with indications of whether and data items on the dark web match the semantic descriptions. When a query receives a positive response from the monitoring service, the data protection service iteratively refines the semantic description and queries the monitoring service with the refined semantic descriptions until a breach is detected. Once a breach is detected, the data protection service initiates a mitigation action.
In one embodiment, a computer implemented method includes comprises storing objects in a first bucket and files in a second bucket of a first storage cluster of the distributed storage system, initiating an audit job on the first storage cluster, synchronously replicating audit configuration data and mirroring audit data (e.g., audit files, logs) from the first storage cluster to the second storage cluster, performing a switchover process from the first storage cluster to the second storage cluster, and initiating an audit job on the second storage cluster based on the audit configuration during the switchover process. The first storage cluster initially handles input/output operations for a software application before the switchover process.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
Techniques are provided for rebuilding a destination file system as part of a resync operation. A file system of a first node may be replicated to a second node as a destination file system. The file system may include clone files that were created as clones of parent files. The clone files may have been created in a storage efficient manner where a clone file refers to data of a parent file, instead physically storing duplicate instances of the parent data of the parent file. The techniques provided herein are capable of replicating the clone files to the second node while preserve the storage efficiency of the clone files during a resync operation that rebuilds the destination file system after recovering from a failure of the second node.
G06F 11/14 - Détection ou correction d'erreur dans les données par redondance dans les opérations, p. ex. en utilisant différentes séquences d'opérations aboutissant au même résultat
G06F 11/20 - Détection ou correction d'erreur dans une donnée par redondance dans le matériel en utilisant un masquage actif du défaut, p. ex. en déconnectant les éléments défaillants ou en insérant des éléments de rechange
86.
Resource Distribution Engine(s) For Allocating And Securing Reclaimable Resources Within A Cloud Environment
Various embodiments of the present technology generally relate to systems and methods for providing a resource distribution engine. In an example, a method includes identifying, by an instance management service, a request from a cluster orchestrator to acquire computing resources associated with deployment of an application. The method may also include determining, by the instance management service, markets containing reclaimable resources suitable for the computing resources, and performing, by a resource distribution engine of the instance management service, a resource distribution process to determine a resource distribution of the reclaimable resources within the markets. The resource distribution of the reclaimable resources may include a distribution of instance type of the reclaimable resources across the markets. The method may also include securing, by the instance management service, the reclaimable resources identified by the resource distribution as provisioned reclaimable resources.
Techniques are provided for replicating clone files from a source file system to a destination file system in a computing system. A file system of a first storage node may be replicated to a second storage node such as for data protection and backup purposes. The file system may include clone files that were created as clones of parent files. The clone files may have been created in a storage efficient manner where a clone file refers to data of a parent file, instead physically storing duplicate instances of the parent data of the parent file. The techniques provided herein are capable of replicating the clone files to the second node while preserve the storage efficiency of the clone files.
One or more techniques and/or computing devices are provided for managing an arbitrary set of storage items using a granset. For example, a storage controller may host a plurality of storage items and/or logical unit numbers (LUNs). A subset of the storage items are grouped into a consistency group. A granset is created for tracking, managing, and/or providing access to the storage items within the consistency group. For example, the granset comprises application programming interfaces (APIs) and/or properties used to provide certain levels of access to the storage items (e.g., read access, write access, no access), redirect operations to access either data of an active file system or to a snapshot, fence certain operations (e.g., rename and delete operations), and/or other properties that apply to each storage item within the consistency group. Thus, the granset provides a persistent on-disk layout used to manage an arbitrary set of storage items.
Techniques are provided for a get key bot that securely provides access to passphrases. A set key workflow is executed to generate a get key executable binary that is implemented as the get key bot. The get key executable binary is encapsulated with encrypted information that includes a verified user identifier, a passphrase, and/or a bot expiry time. Upon receiving a request for the passphrase from a requestor, the get key executable binary is invoked. The encrypted information is decrypted and compared to a logged-in user identifier and current time for verification. In response to successful verification, the passphrase is provided to the requestor. Otherwise, the requestor is denied access to the passphrase.
Methods and systems use a central validation module of a microservice-based system to interface an external identity provider (IDP) among a plurality of external IDPs to receive authentication and/or authorization information for an external request from a client. After the authentication and/or authorization information has been received for the external request, an internal request containing the authentication and/or authorization information is generated by the central validation module, which is transmitted to at least one microservice of the microservice-based system to provide services in response to the external request. The internal request is validated with the central validation module by each of the at least one microservice, which executes at least one operation to provide the services. A response that is based at least partly on results of the services provided by the at least one microservice is returned for the external request to the client.
G06F 15/16 - Associations de plusieurs calculateurs numériques comportant chacun au moins une unité arithmétique, une unité programme et un registre, p. ex. pour le traitement simultané de plusieurs programmes
H04L 67/1097 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau pour le stockage distribué de données dans des réseaux, p. ex. dispositions de transport pour le système de fichiers réseau [NFS], réseaux de stockage [SAN] ou stockage en réseau [NAS]
91.
Managing Object Lock Settings During Cross-Grid Replication Within A Distributed Storage System
Various embodiments of the present technology generally relate to systems and methods for providing managing object lock settings during cross-grid replication within distributed storage systems. In an example, ingestion of an object into a first grid of a distributed storage system may be detected. Responsive to detecting ingestion of the object, object lock settings for the object may be determined. Once the object lock settings are determined, the object lock settings may be validated against destination object lock settings. If the destination object lock settings are validated, cross-grid replication of the object may be initiated. During cross-grid replication, the object lock header may be provided in a replication payload transmitted from the first grid to a second grid. When the object is replicated, the destination object lock settings may be determined for the object, which may include the object lock settings as identified in the object lock header.
Systems and methods for reducing the provisioned storage capacity of a disk or aggregate of disks of a storage appliance while the storage appliance continues to serve clients are provided. According to one embodiment, the size of the aggregate may be reduced by shrinking the file system of the storage appliance and removing a selected disk from the aggregate. When an identified shrink region includes the entire addressable PVBN space of the selected disk, the file system may be shrunk by relocating valid data from the selected disk elsewhere within the aggregate. After the valid data is relocated, the selected disk may be removed from the aggregate, thereby reducing the provisioned storage capacity of the aggregate by the size of the selected disk.
G06F 3/06 - Entrée numérique à partir de, ou sortie numérique vers des supports d'enregistrement
93.
NETWORK ADDRESS MIGRATION BETWEEN DIFFERENT NETWORKS BY UPDATING NETWORK CONFIGURATION USING A DESTINATION COMPUTE INSTANCE IN A CLOUD ENVIRONMENT TO REDUCE DISRUPTIONS
Network address migration using a destination compute instance to update network configuration information in a cloud environment is disclosed. A network interface either using a private address within a subnet corresponding to the created network interface or using a floating address outside of the subnet corresponding to the created network interface is created. A first node of the HA pair with a service provider interface. The first node is an active data server of the HA pair, and the second node is a backup node of the HA pair. Requests are serviced the first node using the created network interface. Upon failure of the first node, the second node performs a failover, wherein if the first node was utilizing a floating address, the second node registers the second node with the service provider interface by adding an address of the second node to the route table.
H04L 61/2539 - Traduction d'adresses de protocole Internet [IP] en masquant les adressesTraduction d'adresses de protocole Internet [IP] en gardant les adresses anonymes
H04L 67/10 - Protocoles dans lesquels une application est distribuée parmi les nœuds du réseau
NETWORK ADDRESS MIGRATION BETWEEN DIFFERENT NETWORKS BY UPDATING FLOATING LOGICAL NETWORK INTERFACES USING A DESTINATION COMPUTE INSTANCE IN A CLOUD ENVIRONMENT TO REDUCE DISRUPTIONS
Network address migration using a destination compute instance to update network configuration information in a cloud environment is disclosed. A network interface either using a private address within a subnet corresponding to the created network interface or using a floating address outside of the subnet corresponding to the created network interface is created. A first node of the HA pair with a service provider interface. The first node is an active data server of the HA pair and the second node is a backup node of the HA pair. Requests are serviced the first node using the created network interface. Upon failure of the first node, the second node performs a failover, wherein if the first node was utilizing a floating address, the second node registers the second node with the service provider interface by adding an address of the second node to the route table.
Systems and methods for performing an online migration of a storage solution deployment from one disk type to another are provided. In various examples described herein, after a storage solution has been deployed that utilizes disks of a first disk type (e.g., a Google Cloud Platform (GCP) Persistent Disk (PD)), the storage solution deployment may be modified to make use of a second disk type (e.g., a GCP Hyperdisk (HD)) without incurring downtime. In one embodiment, the online migration involves adding new disks of the second disk type to an aggregate of which the original disks of the first disk type are a part, mirroring writes to both the original disks and the new disks, and copying valid data from a given original disk to a corresponding new disk during a background scan of each of the set of original disks.
Various embodiments of the present technology generally relate to systems and methods for providing cross-grid replication within distributed storage systems. In an example, a method includes identifying an object for ingest into a first storage grid containing a first distributed storage system and replicating the object to one or more nodes within the first storage grid. The method may also include determining a cross-grid replication status of the object to a second storage grid containing a second distributed storage system and performing a cross-grid replication of the object to the second storage grid based on the cross-grid replication status of the object.
G06F 16/27 - Réplication, distribution ou synchronisation de données entre bases de données ou dans un système de bases de données distribuéesArchitectures de systèmes de bases de données distribuées à cet effet
Method and systems for co-locating journaling and data storage based on write requests are provided. In one example, a first logical storage unit for storing write operation records is provided by a cluster of multiple nodes representing a distributed storage system. The first logical storage unit is divided into a volume partition and a journal partition that includes a first log and a second log. A client write request including metadata and data is received by a first node of the cluster. The metadata is recorded in a first location in an active log of the first log and the second log and the data is recorded in a second location in the active log during a single input/output (I/O) operation performed by the first node. A reply is sent by the first node to the client after the metadata and the data are recorded in the journal partition.
Techniques are provided for key-value store and file system integration to optimize key value store operations. A key-value store is integrated within a file system of a node. A log structured merge tree of the key-value store may be populated with a key corresponding to a content hash of a value data item stored separate from the key. A random distribution search may be performed upon a sorted log of the log structured merge tree to identify the key for accessing the value data item. A starting location for the random distribution search is derived from key information, a log size of the sorted log, and/or a keyspace size of a keyspace associated with the key.
G06F 16/22 - IndexationStructures de données à cet effetStructures de stockage
G06F 16/21 - Conception, administration ou maintenance des bases de données
G06F 16/215 - Amélioration de la qualité des donnéesNettoyage des données, p. ex. déduplication, suppression des entrées non valides ou correction des erreurs typographiques
Techniques are provided for performing adaptive sampling for data summarization. An insight service may provide monitoring, troubleshooting, optimization, security, and/or other functionality for a computing environment. The insight service may intake millions to billions of events on a monthly basis from the computing environment, which are stored within a database. The insight service may provide data summarization for the events, which may include access patterns (e.g., file access patterns), anomalies, and ransomware detection. Dynamically querying and generating the data summarization may be impractical due to the sheer amount of events. Accordingly, adaptive sampling is provided for merely sampling certain events based upon various thresholds and criteria being met so that an evaluation output can be dynamically and efficiently generated within an acceptable time as the data summarization.
Systems, methods, and software are disclosed herein for detecting encrypted data in various implementations. In an implementation, a computing apparatus determines byte frequency distribution values associated with a compute workload. The computing apparatus executes a machine learning model trained to differentiate between encrypted portions and non-encrypted portions of the compute workload based on the byte frequency distribution values. The computing apparatus monitors an encrypted share of the compute workload represented by the encrypted portions and, in response to the encrypted share meeting or exceeding a threshold, initiating a mitigative action.
