AXIS GridLink

AXIS GridLink is a grid solution managing the processor farms used in the distributed processing of AXIS batch calculation runs. It offers a comprehensive set of tools that help you create and manage an efficient and secure production, development or testing environment.

AXIS GridLink Distributed Processing

This Standard Distributed Processing feature is a single user solution for reducing AXIS runtimes when complex processes on large volumes of data are involved. Helpers used in Standard Distributed Processing mode are automatically started by the master copy of AXIS on the same machine as the master utilizing multiple processor (CPU) cores of the computer. Standard Distributed Processing works very well for a one-batch-at-a-time scenario if a user is in full control of a multi-processor multi-core computer. The standard AXIS license allows you to run an AXIS batch job using the Standard Distributed Processing option with up to 15 Helpers.

AXIS GridLink is designed to provide a secure, robust production environment that can scale over multiple dedicated computers up to hundreds of processor cores running multiple distributed jobs in parallel or in succession. Consider AXIS GridLink as a solution when:

  • Many jobs in different datasets need to be run
  • End-user workstation need to be free to develop models
  • Required processing resources exceed 15 CPU cores (eg. for stochastic processing)
  • Multiple users need to share the processor farm
  • Multiple versions are in use
  • Access control to data partitions is required
  • Teams of users need secure production/testing/development environment

With AXIS GridLink you can:

  • Control your farm without logging on to the farm servers
  • Initiate AXIS batch jobs from any AXIS workstation
  • When the job is submitted to the farm it is not running on the user's workstation anymore so it becomes free to do other work - even run AXIS and work in a different dataset
  • Automatically start the appropriate Master and Helpers for the current job on the farm, and control the maximum number of helpers
  • Monitor the batch job running on the farm and the queue of jobs submitted by multiple users in multiple AXIS versions, from your local workstation or from home (using VPN)
  • Move pending jobs up and down in the queue, or cancel them if required
  • Perform multiple non-distributed jobs (eg. DataLink macros) in parallel with distributed ones
  • Review the history of runs with detailed documentation
  • Install new versions of AXIS on all computers in the farm with just one button click
  • Authorize users to perform runs on the processor farm and specify what actions are allowed for each user
  • Create groups with different access permissions for extra security in a production environment
  • Perform dataset level batch jobs as well as system batch jobs such as system formula
  • Schedule and execute safe backups of your data that will not interfere with active jobs

Who can use AXIS GridLink?

AXIS GridLink processor farm administrators can define a list of users who can connect and perform jobs on the GridLink farm. Each user can be granted unique permissions (Basic, Regular and Advanced) defining what administrative actions they can control (eg. monitor jobs, cancel jobs, etc.). AXIS users can monitor the remote job status screen and control the job from any AXIS machine without logging on to the farm server. Users can close the monitor and even turn off the local machine, reconnecting to the AXIS Gridlink Controller later and relaunching the monitor. More than one user can monitor and control the job at the same time.

For additional production environment security, AXIS GridLink implements a feature called "Access Groups" that allows the administrator to set up a farm in such a way that jobs submitted by different users will be ran under specific windows accounts using the accounts permissions and security settings.

How does AXIS GridLink work?

AXIS GridLink Scenario 1

The following diagrams illustrate how AXIS Gridlink can help manage the use of the processor farm and optimize your access to its computing resources.

Scenario 1.1


Scenario 1.2


Scenario 1.3


Scenario 1.4


Scenario 1.5


Scenario 1.6


AXIS GridLink Scenario 2

The following diagrams illustrate how AXIS GridLink provides the ability to balance processing resources between multiple users and jobs and achieve full utilization of the computing power of your farm:

Scenario 2.1


Scenario 2.2


AXIS GridLink is efficient

There are many scenarios where processor farms can be shared between multiple users. Administrators may allow the processor farm to be shared by two or more departments, or organize it to perform development work regularly and high priority production runs at the end of each reporting period. If you have a batch job to process on your farm which does not use distributed processing, you will be able to run this at the same time as it processes another distributed batch job.

The most flexible setups, however, can be achieved with multiple AXIS GridLink job queues on the farm. With these queues, multiple masters split the pool of helpers on a cooperative basis for both distributable and non-distributable jobs. The system automatically determines whether the job needs helpers or is a non-distributable job, balancing resources accordingly. The priorities of the jobs currently running are managed so that farm processing power is always 100% utilized:

  • If you have two job queues and 32 processor cores available on the farm then you can run two distributed jobs in parallel with 15 Helpers on each if they have the same resource priority, or one job using only three helpers and another using the remaining 27 if the first job had Low resource priority and the second one, high
  • If there is only one job running on the farm from one of the queues and there are no more jobs in other queues, the job will use all 31 Helpers regardless of its resource priority (except if it is "No Helpers")
  • As soon as a new job appears in another queue, the processor cores will be split according to the jobs respective resource priorities
  • AXIS GridLink has active backup functionality that allows you to schedule and execute backups of your data without interfering with calculation jobs currently running on the farm
  • Another way to backup your datasets and other related files is to perform the backup at the system level batch - such as system macro or system formula - these types of batch jobs are now supported on the GridLink too

AXIS GridLink Feature List

Configuration and Deployment

  • Support 512 CPU cores per job on the farm
  • Configure and administer GridLink processor farm remotely via AXIS GridLink Utility
  • Update GridLink Controller service remotely on all machines
  • Deploy and check AXIS versions on all machines remotely with one button click
  • Configure centralized drive mapping, AXIS data and executable paths, etc.
  • Customize ratios for resource priority levels
  • Suspend/resume processor farm
  • Remotely check system information of the servers
  • Take helper servers offline for maintenance

Submit Jobs from a Workstation to a GridLink Processor Farm

  • After submission the workstation is free to perform other tasks
  • Submit one or multiple jobs in one or multiple datasets
  • Submit jobs to GridLink via command line
  • Submit jobs to GridLink without opening the datasets (useful when connection is slow)
  • Submit jobs with start-up options (wait for a specified time or wait until a file is available)

Monitor Job Progress

  • Connect to GridLink and see real-time job progress and disconnect at any time
  • Monitor user jobs from local or remote connections

Manage Job Queue

  • Review current job queue, jobs history and jobs statistics (user Id, timestamps, status, etc.)
  • View last saved logs of finished jobs right from the job queue screen
  • View log of currently running job without interrupting the calculations
  • Change pending jobs sequence of execution
  • Change pending and running job resource priority
  • Suspend/resume pending jobs
  • Terminate running jobs
  • Specify aliases for job queues
  • Print job queue reports for documentation

Configure Multiple Queues

  • Split jobs over multiple departments:
    • Maintain/manage multiple job queues
    • Execute multiple distributed jobs in parallel in different queues
    • Move remote jobs available job queues

AXIS GridLink Performance and Disaster Recovery

  • Supports unlimited AXIS versions
  • Starts/stops master and helper copy of AXIS on all farm machines
  • Executes jobs in a queue according to user-defined sequence
  • Executes one distributed job and/or multiple non-distributed jobs in one queue in parallel
  • Executes multiple distributed jobs in parallel (multi-queue configuration)
  • Load-balances helpers between jobs in queues according to user-specified resource priority
  • Automatic data cache for remote datasets
  • Automatically restarts failed helpers
    • Terminates crashed helper copies (if any) when the job is done
    • Cleans-up AXIS temporary files between jobs
  • Automatically restart service after failure
  • Disaster Recovery

Diagnosis

  • Log the controller activities on the farm
  • Option to log Windows event when a job finishes/fails/crashes/hangs
  • Help on troubleshooting printer problems on the farm
  • Save the batch status when a job is hung
  • Generate an error report on the controller crash
  • Detect problems on the farm servers and generate a report

Security

  • Maintain authorized users list per job queue
  • Assign different levels of permissions to different users to specify which actions are allowed for each user
  • Access Groups feature to secure access to various disk partitions per user
  • AXIS security key verification at the time of the job submission (no need to stay connected after this and keep RKey running)
  • Ability to limit maximum resource priority per user
  • Authorize user groups instead of individual users

Backup/Restore/Copy

  • Built-in Active Backup functionality that will not interfere with running jobs
  • Incremental backup for improved performance
  • Ability to schedule backup jobs to safely move specified data to a shadow location for tape backup etc.
  • Backup jobs status documentation
  • Backup/Restore/Copy individual datasets from your AXIS client copy via GridLink - much faster for large datasets as it is done remotely

Reporting

  • Provide farm usage report
  • Provide report on job statistics and server usage/specifications

Easy Integration

  • Integrated with enterprise grid products, such as DataSynapse GridServer, Platform EGO and Microsoft HPC server
  • Burstable integration mode to guarantee the availability for the minimum processing power and allow the access to additional resources
  • Ability to work with cloud products from various vendors
GridLink Support