It can call a single shell script or execute a single Python function that you define (either within FireWorks, or in an external package).Ī Firework contains the JSON spec that includes all the information needed to bootstrap your job. Workflows in FireWorks are made up of three main components:Ī Firetask is an atomic computing job. Running on a heterogeneous set of worker computers is simple because essentially the same code is used internally by FireWorks for running on simple workstations or a large supercomputing center, submitting to a traditional or web-based queue system, or packing together many jobs into a single queue submission. On the opposite end, administrators can configure worker computers without worrying about where workflows are coming from or what they look like (although you can assign jobs to certain resources if desired). This keeps the workflow specifications lightweight, tidy, and easy to learn and use (if you’ve ever seen lengthy XML-based specifications in other workflow software, you’ll notice the difference in FireWorks right away). End users can add new workflows to the LaunchPad without worrying about the details of how and where the workflows will be run (unless they really want to tailor the details of job execution). The components are largely decoupled, which makes FireWorks easier to use. The basic infrastructure looks like this: The FireWorker can be as simple as the same workstation used to host the LaunchPad, or complicated like a national supercomputing center with a queueing system. The FireWorkers request workflows from the LaunchPad, execute them, and send back information. One or more workers (“FireWorkers”) that run your jobs. The workflows can be a straightforward series of scripts or dynamically adapt depending on the results obtained. You can add workflows (a DAG of “FireWorks”) to the LaunchPad, query for the state of your workflows, or rerun workflows. There are essentially just two components of a FireWorks installation:Ī server (“LaunchPad”) that manages workflows. Support for several queueing systems such as PBS/Torque, Sun Grid Engine, SLURM, and IBM LoadLeveler. Package many small jobs into a single large job (e.g., automatically run 100 serial workflows in parallel over 100 cores) Remotely track the status of output files during execution. Support for dynamic workflows - workflows that modify themselves or create new ones based on what happens during execution.Īutomatic duplicate handling at the sub-workflow level - skip duplicated portions between two workflows while still running unique sectionsīuilt-in tasks for creating templated inputs, running scripts, and copying files to remote machines Multiple execution modes - directly on a multicore machines or through a queue, on a single machine or multiple machines. Some (but not all) of its features include:Ī clean and flexible Python API, a powerful command-line interface, and a built-in web service for monitoring workflows.Ī database backend (MongoDB) lets you add, remove, and search the status of workflows.ĭetect failed jobs (both soft and hard failures), and rerun them as needed. FireWorks is intended to be a friendly workflow software that is easy to get started with, but flexible enough to handle complicated use cases.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |