AWS ParallelCluster
AWS ParallelCluster can be used with batch schedulers, such as AWS Batch
and Slurm. Learn more about this technology here.
Quickly build HPC compute environments on AWS
If you need an AWS-compatible cluster management tool, ParallelCluster is the right choice. Based on the open-source CfnCluster project, AWS ParallelCluster allows you to quickly build an HPC compute environment on AWS. It automatically configures the necessary computing resources and the shared file system.
AWS ParallelCluster services
Automated cluster
Define a config (YAML) and let ParallelCluster build compute, storage, network automatically.
Flexible scheduling
Using Slurm or AWS Batch, so you can submit jobs, manage queues, auto-scale compute resources depending on demand.
Support for HPC workloads
You’re not limited to small jobs; you can spin up large EC2 instances, GPU-enabled or specialized hardware as needed.
Shared storage and data pipelines
Ability to mount shared filesystems across nodes (EFS / EBS / parallel FS), useful for large-scale simulations, data analysis, ML training.
In their official documentation, they explain that AWS ParallelCluster can be used "with batch schedulers, such as AWS Batch and Slurm. AWS ParallelCluster facilitates quick start proof of concept deployments and production deployments."
It also allows the user to create higher-level workflows, such as a genomics portal that automates a complete DNA sequencing workflow, in addition to AWS ParallelCluster.
How AWS ParallelCluster works
- First, it's time to design the cluster based on the demands of your application or project, you can choose pre-established design options.
- Once the cluster is configured, you launch it by executing the "create" command
- When the cluster reaches "create complete" status, you connect via an SSH client.
- Finally, send your work to the organizer.
Our implementation process
1
Configuration
The necessary permissions, networks, and resources are defined in AWS. The environment is ready.
2
Customization
The cluster is adapted to the type of scientific or engineering workloads.
3
Integrations
We connect file systems, containers, and HPC frameworks.
4
Monitoring
Metrics, alerts, and autoscaling rules are enabled. The cluster remains optimized.
